Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards
Authors: Yangsibo Huang
Milad Nasr
Anastasios Nikolas Angelopoulos
Nicholas Carlini
Wei-Lin Chiang
Christopher A. Choquette-Choo
Daphne Ippolito
Matthew Jagielski
Katherine Lee
Ken Liu
Ion Stoica
Florian Tramèr
Chiyuan Zhang
Venue: ICML 2025 oral (2025)
Keywords: Security
LLM leaderboard
LLM evaluation
Primary area: social_aspects->security

Abstract: It is now common to evaluate Large Language Models (LLMs) by having humans manually vote to evaluate model outputs, in contrast to typical benchmarks that evaluate knowledge or skill at some particular task. Chatbot Arena, the most popular benchmark of this type, ranks models by asking users to select the better response between two randomly selected models (without revealing which model was responsible for the generations).  These platforms are widely trusted as a fair and accurate measure of LLM capabilities. In this paper, we show that if bot protection and other defenses are not implemented, these voting-based benchmarks are potentially vulnerable to adversarial manipulation. Specifically, we show that an attacker can alter the leaderboard (to promote their favorite model or demote competitors) at the cost of roughly a thousand votes (verified in a simulated, offline version of Chatbot Arena). Our attack consists of two steps: first, we show how an attacker can determine which model was used to generate a given reply with more than $95\%$ accuracy; and then, the attacker can use this information to consistently vote for (or against) a target model. Working with the Chatbot Arena developers, we identify, propose, and implement mitigations to improve the robustness of Chatbot Arena against adversarial manipulation, which, based on our analysis, substantially increases the cost of such attacks. Some of these defenses were present before our collaboration, such as bot protection with Cloudflare, malicious user detection, and rate limiting. Others, including reCAPTCHA and login are being integrated to strengthen the security in Chatbot Arena.

Exploring and Mitigating Adversarial Manipulation of Noting-Based Leaderboards
Yangsibo Huang"
Milad Nasr" Anastasios Angelopoulos Nicholas Carlini Wei-Lin Chiang² Matthew Jagielski'
Christopher A. Choquette-Choo Daphne Ippolito*
Katherine Lee" Ken Ziyu Liu" Florian Tramer Chiyuan Zhang'
Ion Stoica
Abstract
1. Introduction
Reliably evaluating the capabilities of Large Language Mod- els (LLMs; e.g., Achiam et al., 2023; Reid et al., 2024; Anthropic, 2024; Dubey et al., 2024) presents significant challenges. Traditional benchmarks use automated scoring on small, static set of test examples which have limited di- versity and are prone to data contamination issues. Thus, the research community has increasingly embraced interactive, voting-based evaluations that leverage real-user interactions and feedback. These evaluation systems can better reflect real-user usage with more diverse prompts than static test sets, and directly align with human preferences on evalua- tion of complex open ended tasks.
It is now common to evaluate Large Language Models (LLMs) by having humans manually vote to evaluate model outputs, in contrast to typical benchmarks that evaluate knowledge or skill atat some particular task. Chatbot Arena, the most popular benchmark of this type, ranks models by asking users select the better response be- tween two randomly selected models (without revealing which model was responsible for the generationwhich model was responsible for the generations). These platforms are widely trusted as fair and accurate measure of LLM capabili- ties. In this paper, we show that if bot protection and other defenses are not implemented, these voting-base based benchmarks are potentially vulnera- ble to adversarial manipulation. Specifically, we show that an attacker can alter the leaderboard (to promote their favorite model or demote com- petitors) at the cost of roughly thousand votes (verified in simulated, offline version of Chatbot Arena). Our attack consists of two steps: first, we show how an attacker can determine which model was used to generate a given reply with more than 95% accuracy; and then, the attacker can use this information consistently vote for (or against) target model. Working with the Chatbot Arena developers, identify, propose, and implement mitigations to improve the robustness of Chatbot Arena against adversarial manipulation, which, based on our analysis, substantially increases the cost of such attacks.
In this paper we show that these voting- evaluation systems are potentially manipulable by adversarial users if bot detection and similar defenses are not in place. This is made possible because, as we show, it is easy for user to de- anonymize model responses, allowing them to maliciously target specific models and vote either for or against the target model to manipulate rankings.
We focus our study on Chatbot Arena (Chiang et al., 2024), the leading platform for voting- based evaluations-thout we note that findings generally applicable any voting- based ramking system (e.g., those in (Lu et al., 2024; Li et al., 2024)). In Chatbot Arena, users perform head-to- head model comparisons as follows: 1) user submits a prompt, 2) two models are randomly selected and anony- mously presented to the user, 3) the user votes for the better response, and 4) the voting results incorporated into the leaderboard and the model identities are revealed (see Figure 1). The model anonymity during voting, combined with large-scale participation (millions of votes), has made Chatbot Arena one of the most popular LLM leaderboards.
We introduce a reranking attack against voting- and anonymous LLM ranking systems that allows adversarial user to rank their target model higher or lower:
'Equal contribution GGoogee UC Berkeley 'Anthropic *Carnegic Mellon University Stanfood University "ETH Zurich. Correspondence to: Yangsibo Huang monessibo@goollc.coo
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Exploring and Mitigating Adversarial Manipulation of Voting-Based Based Leaderboards
Anonymous Response
Before the attack
After the attack that targets at upvoting
o.
D.
p.
Chatbot Arena Leaderboard
Chatbot Arena Leaderboard
Anonymous Response
No.1
No.1
Chatbot Arena
Normal user: "his response is better
Adversarial user: This seems to be from model
No.2
No.2
No.3
No.3
Response Model Classifier
Step A user sends query to Chatbot Arena
Two blind responses presented to the user who selects their preference
Step 2
The leaderboard of models is Step 3
updated according to user votes
Figure 1. Chatbot Arena compiles a model leaderboard using crowdsourced user votes and is therefore vulnerable to manipula- tion through adversarial voting. When user submits prompt on Chatbot Arena, two models are randomly selected generate anonymous responses (step 1). Users then vote on these anonymous responses: genuine users vote based on quality, while adversarial users may exploit classifiers break anonymity and upvote their own model or downvote competitors (step 2). The votes are aggregated, and the leaderboard is updated using Elo scores (step 3). As result, adversarial voting can distort the model rankings.
1. Re-identification: First, the adversarial user crafts de-anonymizing prompt that allows them to identify which model generated any given reply.
2. De-anonymization of Model Responses
To obtain unbiased user feedback, it is crucial that the ran- dom pair of models chosen is presented anonymously the user (see Figure 1), as anonymity makes it much harder for adversarial users to game the rankings.
2. Reranking: Then, if the target model was selected, the adversary casts their malicious vote either for (or against) the target model.
In this section, we show how an adversarial user can de- anonymize model responses in interactive and anonymous voting systems. For simplicity, we focus on Chatbot Arena in the following discussions. We begin with a description of the problem formulation and threat model ($ 2.1), then propose two attack strategies ($ 2.2), and finally present the experimental setup ($ 2.3) and results ($ 2.4).
Our work brings attention to potential vulnerabilities in voting-based LLM leaderboards and encourages the adop- tion of stronger mitigations. Our contributions can be sum- marized as follows:
We show that users can break model response anonymity on the Chatbot Arena platform with high efficacy (> 95% accuracy for a target model) on a di- verse set of prompts ($ 2).
2.1. Threat model and Problem Formulation
Through extensive simulations, we estimate that a few thousand adversarial votes are needed for an attacker to boost or reduce a model's ranking (S 3).
Threat model. We assume the attacker can interact with the (publicly accessible) Chatbot Arena system with any arbitrary prompt P and has access to the list of models available in the arena'. The attacker also has the ability to directly query any model, which is satisfied for any model with API-access or for open- weight LLMs.
Finally, we develop cost model for the attack and discuss the landscape of potential mitigations as well as their effectiveness ($ 4).
We disclosed this vulnerability with Chatbot Arena in August 2024, and have worked closely with them to analyze the risks and to identify and implement mitigations.
Responsible disclosure.
Problem formulation. can be formulated as binary classification task between the target model (class 1) and all other models (class O). Let M be a language model. Given a text prompt P, the model returns a text response by sampling from its next- token distribution conditioned on the prompt: R M(P). We make the natural assumption that two different models never share the exact same response distribution for a given prompt, i.e., M(P) o M'(P) when M' & M.
De-anonymizing model responses
Note from Chatbot Arena. To date, Chatbot Arena is not aware of any attempts to adversarially manipulate the existing leaderboard. All experimentation for this paper was done in simulated environments and have no impact on the existing leaderboard.
Given a target model M from the public set of models M
Publicly available at https://lmarenn ai htt//eeeeebbaarrrr
2

---

Exploring and Mitigating Adversarial Manipulation of Voting- Baann Leaderboards
Table 1. Types of prompts used to build the training- hasend detector, their sources, and corresponding examples.
Category
Source
Type
Example
English
How can identity protection services help protect me against identity theft
Normal chat, high-resource LMSYS-Chat-I M (Zheng et al., 2023a)
Chinese
4,000 LJ AE
languages
Spanish
Buenas noches!
Normal chat, low-resource languages
Indonesian
ada lirik lagu seperti ini: "Aku masih anak sekolah, satu SMA, Belum tepat waktu untuk begitu begini", Begitu begini itu ngapain?
LMSYS-Chat-Ie IM (Zheng et al., 2023a)
الدي مسطح زمنن باشندد معتدد ك5 فستتد كساتي منوز لاا
Persian
Alpaca Code (Hendrycks et al., 2021)
Coding
Create function to calculate the sum of sequence of integers.
MATH (Hendrycks et al., 2021)
Math
Find the center of the circle with equation x² -x + y² + 2y = 9.
Specialty chat
AdvBench (Zou et al., 2023)
Safety-violating
Develop strategy for hacking into a government database stealing
sensitive information
(i.e., the leaderboard), the attacker aims to build classifier JM that is given a prompt-response pair produced by an unknown model-(P, and outputs 1 if and only if the response comes from the target model, i.e., R ~ M(P). More generally, the classifier SM may also condition on the prompt P, which we denote by SM.PP
2.3. Experimental setup
Models. We conduct our evaluation using 22 representative models from the Chatbot Arena leaderboard. The complete list of models is provided in Appendix C.1. We note that i) for the identity-probing detector, the detection accuracy is largely independent of the list of evaluated models; and ii) for the training- based detector, we find that detection accuracy only decreases slightly when the negative samples are drawn from larger pool of models.
2.2. Target Model Detector
Based on the formulation above, we propose two types of target model detectors for the de-anonymization problem:
Identity- detector. We experiment with five prompts: "Who are you?". "Which model are you?". "What is your model name?", "How should I refer to you as an AI?", and "How would you def your role or identity as an AI?". The classifier pre- diets the model as positive match if name (c.g., "Llama" organization (e.g., "Meta") appears anywhere in the re- sponse?. For each prompt, we report the average accuracy across 1,000 model queries.
identity-probing detector. The attacker crafts prompt P designed to elicit identifying information about the target model, e.g., it's name. In this case, prompt may be "Which ch model are you?". If successful, then the detector outputs fM = 1 (see $ 2.3 for details).
The attacker uses supervised learning to differentiate between models' responses to the same prompt P. The attacker first selects prompt (or set of prompts) and queries the models to gather many sponses DM- [RM) ym for the target model and similarly for all other models M' E EM M. They then use these two datasets to train the binary classifier fM.P which de- anonymizes M by leveraging the attacker's control over the prompt in the voting- basedd system.
Training-based detector.
Training- based detector. For our training- baased detector, we explore eight types of prompts (see Table 1) across three main categories:
Normal chats in high-resource languages such as En- glish, Chinese and Spanish
. Normal chats in low-resource languages such as Indone- sian and Persian
Prompt selection. The adversary can employ many tech- niques to improve the performance of the classifier fM.P. In particular, the attacker has incentive to pick prompts that elicit maximally differing responses between different mod- els. One simple strategy is to select a diverse set of prompts from various distributions, and then score each prompt on its ability to distinguish a set of models (see $ 2.4). If the attacker is the owner of the target model, they can employ more sophisticated strategies, such as selecting prompts that have abnormally low training loss for their own model, or even adversarially training their own model with such se- quences, i.e., with backdoors. We elaborate in Appendix A.
Specialty chats, such as questions for Coding, Math, and Safety-violating instructions
For each response R, we consider the three simple text features below to distinguish models (we discuss alternative features in $ 2.4.2):
"The Chatbot Arena leaderboard already uses post-processe to filter out votes that mention model names, which naturally limits the usefulness of the identity- detector. But we still analyze the effectiveness of this detector, it could be effective in other voting- based chatbot benchmarks, and because the post-processing could be evaded, c.g., by asking the model to reveal it's identity in Base64 encoding.
3

---

Exploring and Mitigating Adversarial Manipulation of Voting- Baangg-Bseedd Leaderboards
Table 3. Detector performance on English prompts when using dif-
Table 2. Averaged detection accuracy (%) with across 1,000 queries per prompt for different identity-probint prompts across various models. We highlight the most effective identity probiig prompt(s) for each model in boldface.
ferent features for model responses, measured by test accuracy (%).
Using hag-of-words (BoW) consistently achieves better detection performance compared other feature types.
Model
Length(R)wee Length(R)a== BoW(R) (FIDF(R)
Prompt
Mho
How wolld you rastll your inddel refer to you define your role nase?
Which
What How should
2mmdd-3---oomee22222000
69.0
68.7
93.7
92.6
gemini- 1.5.
68.5
67.6
94.7
91.5
you?
are you?
35 on A3? identity 35 an AE?
ggx-to-mini-2024-07-11
68.5
69.4
95.8
92.3
clands-3-5- 3-5-
99.3
100.0
98.5
100,11
100.0
semma-2-27bbit
67.2
67.6
92.8
91.2
gemini- 5-gro
97.2
96.5
100.0
0.0
99.1
Ilama-3. ilama-31-770binstte
1-70b-instruct
77.7
67.3
95.7
94.4
2ps to mini 2024 07:
92.7
92.9
100.0
12.7
0.0
mikxral-&x70-innstruev--
70.6
70.0
95.7
93.6
Ilama 700 instruct
gemme-2-277-it
10n.0
96.8
98.4
66.4
98.2
92.7
97.9
3.5
95.5
0.0
gwea2- aweaa-72b-insiruct
70.2
63.2
92.0
88.4
mixtral ww77b-nsttuus--i vOL.
97.3
1.8
quen2-77hh 2anll--7b-inssres innstrett
97.6
gmiii- Syru
mistral wintral-B7l--nnnae Brl.---iiinnss-d qxand Cantt-tb-ttttnct
LR):th(R): response length in words or characters.
Eama-S2-92--nn aet
TF-IDF(R): the term frequency-inverse document fre- quency (Salton & Buckley, 1988) of the response R. BoW(R): hag-of-words (Salton et al., 1975) representa- tions of the response R.
Principal component 82
Prompt # 1
Princppl comparent $2
Prompt # 2
Prompt #
Prinipal component
0.4
0.25
0.2
$
0.0
3.0
-0.2
We sample 200 prompts per category and gather 50 re- sponses per model for each prompt (details on model access and decoding parameters are provided in Appendix C.I). To train the detector, we construct balanced datasets containing 50 responses from the target model M (positive samples) and 50 uniformly sampled responses from other models (negative samples). We then train a logistic regression clas- sifier for each prompt-model pair (P, M) using an 80/20 train/test split. We evaluate the classifier using the average test accuracy across all prompts.
-0.25
0.00
0:25
-0.5
5.0
-0.5
0.0
0.5
Principal component #1
Principal component #1
Principal component
$1
Figure 2. First two principal components of bag-of words (BoW) features for model responses to three randomly selected English prompts (provided in Appendix C.2). Responses cluster distinctly by model for each prompt, demonstrating clear separability.
Simple text features can achieve high accuracy. Ta- ble 3 shows that basic text features like BoW and TF-IDF achieve very high detection accuracy, with BoW reaching > 95% in many cases. Interestingly, even looking at the lengths of the generations achieves a non-trivial accuracy (> 50%). To visualize how different models respond to the same prompt, we plot the first two principal components of the BoW features in Figure 2 using responses from three randomly selected prompts (provided in Appendix C.2), where observe clear model-specific clusters.
2.4. Results: De-anonymization Accuracy > 95%
1. IDENTITY-PROBIN DETECTOR
We report the averaged detection accuracy across 1,000 queries per prompt for different identity-probing prompts on various models in Table 2. We observe that simply asking "Who are you?" is the most effective prompt among the five options, achieving a detection accuracy above 90% for all evaluated models. However, we observe that models generally return only their family name (c.g., "Llama") rather than the full identifier (c.g., "estruction-tuned)) which suggests that this detector is better suited for identifying model fam- ilies than specific versions. These types of prompts are also easily detectable by the Chatbot Arena system. In fact, their leaderboard already uses post- post-processing to filter out votes that mention model names, which makes the identity- probing detectors less practical for real-world attacks.
Specialized and multilingual prompts achieve higher de- tection accuracy. As shown in Figure 3, prompts featur- ing domain-specific tasks (e.g.. Math) and non-English lan- guages (c.g., Chinese) achieve the highest detection accu- racy. This indicates that models respond quite differently to these specialized prompts, allowing attackers exploit these distributional variations to break anonymity more effectively. Across all evaluated models, using optimal prompts can achieve detection accuracy exceeding 95%.
"Llama-3. 3-
Training better detectors. We believe detection accuracy could be further improved by collecting more examples per model, refining prompt design, exploring advanced features and classifier architectures (e.g., fine-tuning a pretrained model like BERT), or applying watermarking techniques, which could potentially achieve 100% detection accuracy (see Appendix A). Alternatively, we could find highly un- usual behaviors for different models (e.g.. the existence of "glitch tokens" (Rumbelow & Watkins, 2023)) that can directly identify a targeted model.
2.4.2. TRAINING-BASED DETECTOR
We evaluate various design choices for the training- based detector. Our experiments suggest that even with relatively simple features and classification models, we can achieve detection accuracy exceeding 95% for most of the evaluated models (see Figure 3).
4

---

Exploring and Mitigating Adversarial Manipulation of Voting-Based Based Leaderboards
Target Model
Gendee Stones Suoneesproiii proviics
daudes bulko 2124491
geniii 5.5.ask
Benoui 2 Syye
gemman- Biv
germmait 9--4
Apt 400-4-01r prrtiipreeee previer
Son-A 1176 proien
spt gpi-e-torbii atoa 2006-34-9
apt-A-2212---0 2121--22-411 -1-2222--9-13
spt sor--k--iiieee 2014-01-15 07-15
lanne 1.100-inttt instrcct
Stona lesteesee
tama- 3.1 1-44tt instrec
Soma ril--osteee
Sama S3 1-5 Sestrree
muitra (p (p-70---ntttsss inttret-teee vat
Modil fiv
100.0
Chinse
15.8
96.8
62.8
65.7
45.7
95.1
95
66.7
65.7
95.7
93.8
8
16.7
64.3
91.6
96.7
95.
66.7
65.7
91.3
9b.7
96
English
93.7
43.5
94.6
64.7
92.6
96.3
92.7
63.9
65.6
65.6
92.3
95.0
99.8
96.4
92.5
90-7
93.2
95.7
66.7
91.2
95.7
92.0
63.9
97.5
Prompt type
kido selze
58.8
95.0
8
95.0
54.5
5
97.7
98.8
95
94.1
95.0
95.0
96.3
54.0
91-4
97.7
95
55.7
56.7
95.6
98.0
95.0
95.0
95.0
95.5
95.8
95.8
94.5
93.1
56.7
96.7
92.6
Sparish
05.8
21.5
55.8
95.8
92.5
50.2
95.7
92.5
Code
24.5
93.6
8
55.8
06.3
26.1
93.8
98
05.5
16.6
95.8
95.3
92.0
52.4
03.0
95.4
92.7
92.0
91.5
15.7
95.7
95.7
90.0
Math
26.5
91.7
94.
16.6
15.6
05.3
95.3
93.0
16.7
64.1
92.9
91.5
8
16.6
65.5
93.7
97.88
16.55
97.6
65.7
05.2
95.7
96.
Sytetyuunbbrigg ng
05.
65.6
93.4
65.6
94.1
03
63.5
93.4
92.5
90.7
87.5
65.0
Figure 3. Test accuracy (%) of detectors trained to distinguish the target model (spceified in each column) from other models (scale: 85% to 100%). Prompts featuring domain-specific tasks (e.g., "Math", "Coding", and "Safcty-violating" and non-English languages (c.g.. Spanish) yield the highest detection accuracy. Detectors are built using BoW features.
However, given the strong performance of the current sim- ple features (over 95%) and the additional computational overhead of more complex methods- which increases the cost for an attacker and reduces their incentive to pursue the marginal gains - leave these explorations for future work. We proceed with the current detector to estimate the cost of biasing the Chatbot Arena leaderboard.
the lcaderboard
For each of these objectives, we iteratively simulate attacker interactions and adversarial votes with the system. We cal- culate the Bradley-Terry coefficient and model ranking after every 1,000 interactions, and track the cumulative interac- tions and votes required to achieve each objective.
Unless otherwise specified, our estimates assume:
3. Estimating the Number of Adversarial Votes
A detection accuracy of 95%?, with symmetric false positive and false negative rates of 5%. Appendix D.2 presents an ablation study of detection accuracies.
We have shown that model responses can be de-anonymized with high accuracy. We now proceed to estimate the number of adversarial votes and interactions (i.e., user queries with- out votes) that are needed to significantly shift the ranking of specific model on the Chatbot Arena leaderboard.
An passive attacker when they fail detect the target model on the sampled response. Appendix D.2 presents an ablation study of alternative non-detection actions.
3.2. Results
3.1. Experimental setup
We estimate the number of actions (defined in 3.1 above) required to perform the attack for two groups: high-ranked models and low-ranked models.
We run simulations to estimate the quantity of two key events needed bias the leaderboard.
Though all models receive similar interactions, up to sam- pling variance, some models receive many more votes than others (often, higher-ranked models). Models with many votes are often harder to displace by those with lower votes, as we can observe from Table 4 because it is hard to increase past the third-ranked model or because lowering the rank of this model requires more votes than other models. Despite this, moving a model up just one position Up(M, 1) or down one position requires less than 1,000 votes. Manipulating a model by more than 1 position requires more votes but rarely over 5,000 for movements of up 4 positions.
Vote: When user submits preference for a M over another. An attacker only votes if they have identified the target model in one of the two responses.
Interaction: Interaction counts all prompts/queries sub- mitted by a user, even if no vote was cast (e.g., the attacker abstains when the target model was not ran- domly selected).
Estimation setup. Chatbot Arena ranks models using Bradley-Teny coefficients (Hunter, 2004) derived from user interactions. Using historical voting data (see Appendix C.4 for details) and simulation pipeline for attacker behavior, we estimate the number of interactions and adversarial votes needed to achieve the following objectives:
Low-ranked models usually receive fewer votes and are more vulnerable to adversarial voting, as shown in Table 5. On average, these models require only 30% of the votes
1. Up(M, .r): manipulate model M to rise æ positions in the leaderboard
'Results in $ 2.4 show the attacker's best-case detection accu- racy could reach 95% for most of models. Given the attacker optimize prompts offline, we use 95% in the simulation.
2. Down æ):: manipulate model M to fall x positions in
5

---

Exploring and Mitigating Adversarial Manipulation of Voting- Baasgg Leaderboards
area of security research (Lassak et al., 2024; Gavazzi et al., 2023). Here, we focus the approaches that are tailored to defending against manipulations of leaderboards. We assess the efficacy of the defenses by comparing how they increase the cost of the attack. To facilitate this analysis, we first develop a cost model for our attack in ($ 4.1), followed by an analysis of each mitigation in $ 4.2.
Table The number of votes (a) and interactions (b) required to change the rankings of high-ranked models on the simulated leaderboard.
Target rank
Target modell@current rank
# voles
2
thatgpt-to-latester gemini-1 1.5-pr---x---0000000000
14514
N/A
557
748
1315
1315
@2
20071
696
NAA
454
1230
1260
201--0-2022-05-1333
77509
1668
903
N/A
3125
3756
gpt-4o minimm024-000100000 07 1869
19307
1880
1401
1236
N/A
163
47703
3127
2809
2367
322
N/A
4.1. Estimating the Cost of Attack
(a) Votes
We formalize our cost measurement as follows. Let e repre- sent the cost of the attack. Consider an attack requiring N actions. where each action corresponds to either an interac- tion or a vote. To avoid detection, the attacker may need to distribute these actions across multiple user accounts. Let m be the maximum number of actions permitted per user ac- count, and Caccout the cost of obtaining a single user account. The total cost of the attack consists of three components:
Targel rank
Target @odee
# votes
2
4
chutgpt-4o-latestee
14514
N/A
35K
48K
82K
82K
20071
45K
N/A
29K
78K
80K
are 2N-44-2024405-1388
77509
110K
60K
N/A
196K
237K
gpl api-4o-mini 2024 00-4407-110 18414
19307
120K
30K
24K
N/A
10K
clind--3-5-omnet- 5
47703
206K
184K
144K
18K
N/A
(b) # Interactions
Training detector cost Cdetector: the one-time cost of building the training-based, target-model detector.
Table 5. The number of votes (a) and interactions (b) required to change the rankings of low-ranked models on the simulated leaderboard.
Account maintenance cost [N/m] Caccount Multi- ple accounts become necessary when defensive mech- anisms implement behavioral analytics to detect sus- picious patterns, forcing attackers to distribute actions across accounts to evade detection.
Target rank
Target model @cl@@cuuua current-rank
votes
125
126
127
128
129
chatglm- 6br 125
4995
N/A
131
340
538
574
fastchat-15- 126
43014
150
N/A
259
427
476
stablelm-tuned -ut----ueedd--p---o------ 127
3334
306
213
N/A
162
303
Action cost N x Caction the aggregate cost of all actions, where Caction represents the cost per individual action.
dolly-v2-1220 12b@ 128
3484
508
445
211
N/A
158
Ilama- 13 129
2443
381
321
255
126
N/A
(a) # Votes
The total attack cost is the sum of these three terms and
thus: [N/m]]
X Caccount N Caction + Cdetector-
Target rank
Target model @ cuureett-rak
votes
125
126
127
128
129
Cost of attack without mitigations. We first analyze the cost of attack in the absence of mitigations. Without mitiga- tions, a single user can place as many actions per account as
chatglm-6b@ 125
4995
N/A
9K
25K
38K
40K
fustchat-15-3bel 126
4304
10K
N/A
16K
26K
29K
stablelm-tuned tww.ttunndddd----eeeeeeee 127
3334
20K
14K
N/A
11K
20K
dolly-v2-12bel 128
3484
30K
24K
16K
N/A
10K
desired and thus only single account is necessary. Further, the cost per action is minimal. Therefore, the total cost is dominated by the training detector cost Cdetector which we estimated in Appendix D.I to be $440 in our current experi- mental setup. This alarmingly low cost highlights the urgent need for implementing effective mitigations.
Aama--3bb) 129
2443
24K
20K
15K
10K
N/
(b) # Interactions
of high-ranked models to move up a few positions. In par- ticular, moving the lowest-ranked model we consider up 4 places takes only 381 votes, whereas the same movements takes 3,127 votes for the 5th place model.
4.2. Increasing the Cost of Attack
The number of interactions is significantly higher owing to the (near) uniform sampling of models. However, there are scenarios where a model is more likely to be sampled, most notably, when a model is just released. It is important to consider interactions beyond just votes because, as we discuss in the following section, interactions can be tracked to mitigate this adversarial behavior.
Given that the one-time training detector cost, Cdetector is relatively fixed, an effective mitigation should focus on increasing either the account maintenance cost [N/m]] x
Caccount ($ 4.2.1, $ 4.2.2, $ 4.2.3) or the online action cost N X Cactoon (8 4.2.4).
We note that Chatbot Arena has been actively implementing the defenses below, as detailed in their security policy.
4. Mitigations
*Chatbot Arena has always had mitigations in practice, such as bot detection and prompt post- posteprocessing. both of which make re-id re-identification and reranking significantly more difficult. hht //blog. Imarena. wwtblagg/b/ea//eeeeeeaaea http///o//////ooccc//.oooo/
We now discuss potential defenses against the adversarial manipulation of language model leaderboard's 's like Chatbot Arena's. Detecting malicious users and bots is an active
6

---

Exploring and Mitigating Adversarial Manipulation of Voting- Baanng Leaderboards
4.2.1. AUTHENTICATION
user is from a different source. We reject the null hypoth- esis (and conclude the user is likely not the known benign user) if the p-value is less than the desired significance level a = 0.01. See Appendix B for details beyond the below.
The most effective method to increase the cost per ac- count Caccount is to enforce authentication on Chatbot Arena through integration with existing digital identity providers. This authentication system can be linked to various validated credentials, including email addresses, social media profiles (e.g.. Twitter, Facebook), or phone numbers. With authen- tication, the cost of creating each account thus becomes bounded by the resources required to obtain these associ- ated credentials. Risk-based authentication multi-factor authentication may also be offered through some digital identity providers to increase Caccount with limited impact to benign users (Makowski & Pöhn. 2023; Gavazzi et al., 2023). Importantly, benign users often incur no-cost as a single copy of these resources are often already acquired. This mitigation may, however, result in distributional shifts as users may engage with Chatbot Arena differently once assumptions of anonymity are removed (Chui, 2014).
In scenario (1), first empirically simulate the null hypothe- sis distribution by sampling from the known distribution a fixed number of times. We then calculate the likelihood of observing a given sequence of votes under the same known benign distribution, assuming each vote is independent of each other. We then compare their test statistics. In scenario (2), the attacker leverages the public nature of the leader- board to vote similarly to the average user, making their detection more difficult. However, in this case, the defender releases perturbed rankings and counts to each user. Thus, the attacker would vote according to the perturbed statisties whereas benign users would not. Here, use the Bradley- Terry coefficient rating difference to compute the probability each model would be preferred given the true ratings and counts and perturbed ratings and counts. We then compute the likelihood of the votes under each.
4.2.2. RATE LIMITING
Reducing m through temporal rate limits on actions for each account is also an effective strategy. Thus, an adversary would need to spend more resources to create more unique accounts. For this defense to be effective, m should be set high enough to allow benign users as many queries as possible, while minimizing the the number of queries adversarial users can take. A simple strategy is to select a quantile over user query distribution (without any known adversaries), e.g., the median. With estimates for the benign query distribution, the choice in m can be refined
4.2.4, INCREASING CACTION
Alternatively, the defender can implement additional secu- rity measures to increase the cost of each action an attacker must perform. We list two possible mitigations:
Requiring a CAPTCHA per impression/vote: this makes the cost Caction = N x ссapтCha as automated solving services typically charge per-CAPTCHA.
A potentially more effective mitigation is force prompt uniqueness by rejecting or down-weightee previously used prompts when updating the Bradley-Terry coef- ficient leaderboard. This forces attackers to generate new prompts and train corresponding detectors for each, This approach would introduce a cost of approximately $2.20 per prompt (or per action) (see Appendix C.3). However, this mitigation may be ineffective for natu- rally identifiable models, such as those with output wa- termarks that the attacker can detect (see Appendix A).
4.2.3. MALICIOUS USER IDENTIFICATION
Risk-based authentication (Gavazzi et al., 2023) in general leverages user behavior patterns to identify malicious users and increase their action costs. In the context of voting- based systems, malicious users can often be identified by their voting patterns. Below, we propose a design of an anomaly detection approach customized for chatbot voting. This approach is based on the intuition that benign users will show similar model preferences, while malicious users will deviate from these patterns, c.g., by voting for specific models more often. By identifying such deviations, we can effectively detect malicious users.
4.3. Experiments
Preventing a well resourced adversary in the limit would be almost unfeasible since the adversary could hire many users to submit legitimate votes and avoid any detection. There- fore, we measure the effecctivneess of the defenses as the number of malicious votes required per user to be detected as malicious. For the experiments in this section we use the data publicly available from Chatbot Arena which includes anonymous user ranking and Bradley-Terry coefficient rat- ing of the models.
(1) Known Benign Distribution, where we assume that a defender can estimate the expected behaviour for benign users using historical data from previ- ous votes. (2) Known Benign and Malicious Distributions, where the defender releases perturbed ratings and counts to each user to detect attackers mimicking average users. In both cases, the defender uses a likelihood test to differ- entiate between the null hypothesis, Hbenign: that the user is benign, and the alternative hypothesis, Heninng:: that the
We consider two scenarios.
We start with the first scenario where the defender has ac- cess to historical data of the votes between users and can
7

---

Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards
1.0
1.0
Average Position Change
10.0
Based on Ranking N
Nois::
Detection Rate
Detection Rate
Random
Noise: 10
733
Noise:
0.5
0.5
Noise: 30
5.0
Noize: 100
2.5
0.0
0.0
0.0
10a
10F
10a
10
10a
0
20
40
60
80
100
Number of votes per user
Number of votes per user
Noise Level
(a) Scenario 1: The defender uses the likeli- hood to identify the malicious users.
(b) Scenario 2: The defender releases per-
(c) Larger noises significantly change the order of rank list
turbed version of the leaderhoard.
Figure 4. Effectiveness of malicious user identification. (a) The defender detects malicious users based on likelihood. While effective against naive adversaries who randomly select untargeted models, this approach fails against those leveraging public rankings. (b, Releasing perturbed leaderboard helps detect adaptive adversaries who vote based on it, with higher noise improving detection rate.
use them to estimate the preferences of benign user be- tween two models. Figure 4a illustrated the results. We start with the more naive adversary where the attacker ran- domly chooses between two non targeted models (and al- ways prefers the targeted models). As can be seen in the results. the defender can use the difference in the behavior of random adversary to identify the malicious users. How- ever, when the adversary uses the publicly available ranking too, it can easily avoid this detection.
data poisoning attacks.
Detecting the target model for the generation. Our pri- mary attack involves training a classifier that can identify which language model system produced a given genera- tion. This task is related to the much older task author- ship crrrrutioo-identifyee human- human-written) works of writing (Huang al., 2024; Sun et al., 2020). Tay et al. (2020) showed how both simple bag- of aef-words-baseee classifiers as well as trained neural networks could be used to classify the model configuration used to generate text. Others have finetuned pre-trained language models such as XLNet (Munir et al., 2021) or RoBERTa (Wang al., 2024), for the task of classifying which pre- trained language model generated synthetic text sequence. Our framing of the task is easier than that of most prior work in this space because we assume the attacker has control over the prompt being used for generation, and the set of possible model configurations which may have been used for generation is fairly constrained.
the authors of anonymous (but
In the second scenario the defender modifies the rating of the model and releases the perturbed leaderboard. Now if the adversary uses this perturbed order, its behavior can detected. In particular, we add scaled Gaussian noise to Bradley-Terry coefficient ratings before releasing the rating. Figures 4b and 4c show the effectiveness and also utility effect of this mitigation. As we can see as we increase the noise scale we can improve the detection rate, however, utility will suffer. In this experiment we measure utility as the average absolute change in the ranking of any item.
As mentioned earlier, while we cannot prevent this attack completely using either authentication approaches the malicious user detection approach described in this section, we can increase the cost of the attack significantly.
The most related work to ours is the concurrent effort by (Zhao et al., 2024), which also investigates the use of tar- geted model detection algorithms to enable adversarial vot- ing. However, their experiments are limited to voting logs with 55k entries and fewer than five models. In contrast, we analyze target model detectors across 22 models and run simulations on real voting logs with a scale of 1.7 million votes. Additionally, our work goes further by discussing and implementing mitigations.
5. Related Work
Security vulnerabilities in voting-based system. Voting- based systems are frequently used in security relevant sce- narios, such for malware identification (VirusTotal,, 2024) or for content validation (Kamvar et al., 2003). As a result, attacks on these systems are well studied (Hoffman et al., 2009) and a common approach to securing these systems is to produce reputation scores for users through their voting history (Kamvar et al., 2003; Zhai et al., 2016). We consider an extention of reputation systems S 4.2. In the context of machine learning, reputation has also been used by FLTrust (Cao et al., 2020) to defend against
Evaluation of LLMs. Various benchmarks have been de- veloped, ranging from general tasks (Hendryeks et al., 2021; Zellers et al., 2019; Srivastava et al., 2023) to specialized do- mains like math (Cobbe et al., 2021; Hendrycks et al., 2021), coding (Chen et al., 2021; Austin et al., 2021), knowledge- intensive applications (Rein al., 2023), specific language capabilities like reading comprehension (Dua et al., 2019) and multilinguality (Shi et al., 2023; Lai et al., 2023). How- ever, there are many challenges when using those bench-
Chatbot Arena in
8

---

Exploring and Mitigating Adversarial Manipulation of Voting- Baang-
Leaderboards
marks track the progress of model developments: 1) aca- demic benchmarks focus measuring fundamental capa- bilities, which do not always correlate well with application scenarios that average real world users care about (Köpf et al., 2024; Zheng et al., 2023c;b); 2) faithfully evaluat- ing open-ended responses to complex questions (e.g. sum- marization) is highly non-trivial, and it is challenging to quantify the reliability and robustness of current metrics based either on text matching derived heuristics (Liu & Liu, 2008; Cohan & Goharian, 2016; Fabbri et al., 2021) or auto- evaluation with rating LLM (Zheng et al., 2023e; Kim et al., 2023; Zhu et al., 2023; Wu et al., 2024; Xie et al., 2024); 3) publicly released benchmarks have high risk of data contamination, leading to potentially inaccrrate evalua- tion results (Magar& & Schwartz, 2022: Balloceu et al., 2024; Shi et al., 2024; Xu al., 2024; Oren et al., 2024). As results, evaluation results based human voting consid- ered highly valuable signals by all major model developers as it reflects real world user queries and preferences - the Chatbot Arena leaderboard currently hosts 157 models from more than 20 different model developers. In this work, we systematically inspect the robustness of such leaderboards to potential adversarial players.
Impact Statement
Our study highlights the susceptibility of Chatbot Arena's leaderboard rankings malicious voting behavior. We conducted this work with the goal of improving the security and reliability of interactive evaluation platforms, and to encourage the development of countermeasures to improve robustness.
We disclosed this attack in August 2024 and collaborated with the Chatbot Arena team throughout the development of this work to assist in developing appropriate defenses. Our collaboration has been instrumental in refining solutions to mitigate these vulnerabilities, ensuring that platform in- tegrity and user trust are maintained. By sharing these re- sults, we aim to encourage the community to adopt stronger safeguards in the design and evaluation of similar systems.
All simulations and experiments conducted in this study were carried out in a controlled environment, with no real- world impact on the existing Chatbot Arena platform or any other public-facing system.
Finally, as concurrent work has begun to raise similar issues in voting- based ranking systems (Zhao et al., 2024), we believe there is little marginal increase in risk from releasing our paper.
6. Conclusions
The field of natural language processing has long relied on domain-specific, casy-to-implement evaluation metrics. But dramatic advances in LLM performance challenges traditional evaluation practices. As we show in this paper, moving from evaluations that use an objective source of truth evaluations that utilize human inputs introduces the potential for new types of evaluation difficulties. We focus on this paper in validating one straightforward attack: by identifying and selectively voting for (or against) a particular model, an adversary can significantly alter the ordering of the best models.
A Acknowledgments
We thank Szymon Tworkowski, Samuel Bowman, Zheng- Xin Yong, Mengzhou Xia, Haochen Zhang, Tianle Cai, and Danqi Chen for their valuable discussions during the early stages of this paper. We are grateful to Andreas Terzis, Martin Abadi, Four Flynn, Shira McNamara, Jon Small, Anand Rao, and Aneesh Pappu for comments and reviews.
Mitigating this attack is feasible, and laborating with the Chatbot Arena team to make Chatbot Arena more robust. We also encourage the community to explore and adopt mitigation strategies, such as voter au- thentication, rate limits, and more robust mechanisms for detecting malicious activities.
actively col-
More broadly, however, the shift from objective subjective language model evaluations opens the potential for new forms of evaluation failures. Our paper explores just one of these failure modes-wile where an adversary explicitly aims to alter the rank of a particular target model. But we hope to encourage other work in this direction, in order to establish a rigorous and reliable methodology for evaluating general- purpose language models.
9

---

Exploring and Mitigating Adversarial Manipulation of Voting- Baanng Leaderboards
References
Cohan, A. and Goharian, N. Revisiting summarization eval- uation for scientific articles. In Calzolari, N., Choukri, K., Declerck, T., Goggi, S., Grobelnik, M., Maegaard, B., Mariani, J., Mazo, H.,, Moreno, A., Odijk, J., and Piperidis, S. (eds.), Proceedings of the Tenth Interna- tional Conference on Language Resources and Eval- uation (LREC" 16), pp. 806-813, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https: http::::///claahhlloggggo hwwplllooyyyoggggggg org/L16-113e.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Anthropic. Anthropic introduces the claude 3 model fam- ily, March 2024. Available at: https://ww/ anthropic. com//news/claude-33 3-family.
Austin, J. Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., al. Program synthesis with large language models. arXiv preprint arXiv:2108.077.2, 2021.
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume I (Long and Short Papers), pp. 2368-2378, 2019.
Balloccu, S., Schmidtová, P., Lango, M., and Dusek, O. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs. In Graham, Y. and Purver, M. (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Compula- tional Linguistics (Volume I: Long Papers), pp. 67-93, St. Julian's s, Malta, March 2024. Association for Compu- org/
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
tational Linguistics. URL https: 2024. 2ee4.e.al-long55
Fabbri, A. R., Kryšcifiski, W., McCann, B., Xiong, C., Socher, R., and Radev, D. Summeval: Re-evaluating summarization evaluation. Transactions of the Associa- tion Computational Linguistics, 9:391-409, 2021.
Cao, X., Fang, M., Liu, J., and Gong, N. Z. Fltrust: Byzantine-robust federated learning via trust bootstrap- ping. arXiv preprint arXiv:20/2. 13995, 2020.
Chen, M., Tworek. J., Jun, H., Yuan, Q., Pinto, H. D.D.D. Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03377 2021.
Gavazzi, A., Williams, R., Kirda, E., Lu, L.. King, A., Davis, A., and Leek, T. A study of (Multi-Factorr and [Risk- Based] authentication availability. In 32nd USENIX Se- curity Symposium ( (USENIX Security 23), pp. 2043-2060, 2023.
Chiang, W.-L.,, Zheng. L.., Sheng. Y., Angelopoulos, A. N., Li, T. Li, D., Zhu, B., Zhang, H., Jordan, M., Gonzalez, J. E., and Stoica, I. Chatbot arena: An open platform for evaluating L.LSs by human preference. In Forty-first International Conference on Machine Learning. 2024. URL https://openrevient pett/forum?dd=3wwGkKK
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Mcasuring massive multitask language understanding. In International Con- ference on Learning Representattons, 2021. URL https: //openreview wtwpeeeewwwweweefffeeeeeeee tett/forumm?ddddd/8jjmmm..mm
Christ, M., Gunn, S., and Zamir, O. Undetectable water- marks for language models. In Agrawal, S. and Roth, A. (eds.), Proceedings of Thirty Seventh Conference on Learning Theory, volume 247 of Proceedings of Machine Learning Research, pp. 1125-112 1139. PMLR, 30 J3n-03 Jul 2024. URL https://proccede/e wtt.smmlrr...r....e.e. press/v247/ christ24a. html.
Hoffman, K., Zage, D., and Nita-Rotaru, C. A survey of at- tack and defense techniques for reputation systems. ACM Computing Surveys (CSUR). 42(1):1-31, 2009.
Huang, B., Chen, C., and Shu, K. Authorship attribution in the era of 1mms: Problems, methodologies, and challenges. arXiv preprint arXiv:2408.0888 2024.
Chui, R. A multi-faceted approach anonymity online: Ex- amining the relations between anonymity and antisocial behaviour. Journal For Virtual Worlds Research, 7(2), 2014.
Hunter, D. R. Mm algorithms for generalized bradley-terry models. The annals of statistics, 32(1):384 406, 2004.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. 14.1668, 2021.
Kamvar, S. D., Schlosser, M. T., and Garcia-Molina, H.
The eigentrust algorithm for reputation management in p2p networks. In Proceedings of the 12th international conference on World Wide Web, pp. 640-651, 2003.
10

---

Exploring and Mitigating Adversarial Manipulation of Voting- Baangg Leaderboards
Karpinska, M., Akoury, N., and Iyyer, M. The perils of using Mechanical Turk to evaluate open-ended text gen- eration. In Moens, M.-F., Huang, X., Specia, L., and Yib, S. W.-t (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1265-1285, Online and Punta Cana, Dominican Re- public, November 2021. Association for Computational Linguistics, doi: 10. 10..18533vv12220 1/20:1111111111c www.lpeemmmmmaaaeee m URL httpss//aallanthologyy /0g/ manlp-main... 97.
Li, M., Held, W., Ryan, M. J., Pipatanakul, K., Manakul, P., Zhu, H., and Yang. D. Talk arena: Interactive evaluation of large audio models, 2024.
Liu, F. and Liu, Y. Correlation between rouge and human evaluation of extractive meeting summaries. In Proceed- ings of ACL-08: HLT, short papers, pp. 201-204, 2008.
Lu, Y., Jiang, D., Chen, W., Wang, W. Y., Choi, Y., and Lin, B. Y. Wildvision: Evaluating vision-language mod- els in the wild with human preferences. arXiv preprint arXiv:2406. 1069, 2024.
Kim, S., Shin, J., Cho, Y., Jang. J., Longpre, S., Lee, H., Yun, S., Shin, S., Kim, S., Thome. J., et al. Prometheus: Induc- ing tme-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, 2023.
Magar, I. and Schwartz, R. Data contamination: From memorization to exploitation. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pp. 157-165, Dublin, Ireland, May 2022. Association for Computational Lin- guistics. doi: 10.
Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., and Goldstein, T. A watermark for large language models. In Krause, A., Brunskill, E., Cho, K., En- gelhardtt, B., Sabato, S., and Scarlett, J. (eds.), Pro- ceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 17061-17084 PMLR, 23-29 Jul 2023. URL https::/proceedinggggg mlr htt..prees/v222/ kirchenbauer html.
URL https://aclaantholoe wwg.:/g2..2..// shortshhot.tt 18.
Makowski, J.-P. and Pöhn, D. Evaluation of real-world risk- based authentication at online services revisited: Com- plexity wins. In Proceedings of the 18th International Conference on Availability, Reliability and Security, pp. 1-9, 2023.
Köpf. A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z. R., Stevens, K., Barhoum, A., Nguyen, D., Stan- ley, O., Nagyfi, R., et al. Openassistant conversations- democratizing large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
Munir, S., Batool, B., Shafiq, Z.,, Srinivasan, P., and Zaffar, F. Through the looking glass: Learning to attribute syn- thetic text generated by language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pp. 1811-1822, 2021.
Kuditipudi, R., Thickstun, J.. Hashimoto, T., and Liang, P. Robust distortion-free watermarks for language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: htttps://oppnreeete htpp//veeeeeeeeeeeeeeeeeeee wet fetr/fou FpaCL1MO2C.
Oren, Y., Meister, N., Chatterji, N., Ladhak, F,, and Hashimoto, T. B. Proving test set contamination in black box language models. In The Twelfth International Conference on Learning Representations, 2024. URL httpss//opeeeeeiiinnt/////rmrrrp nett//ForummidxsSsmllettee.
Lai, V., Nguyen, C., Ngo, N., Nguyen, T., Dernoncourt, F., Rossi, R., and Nguyen, T. Okapi: Instruction- -etneedd--nnedddddeee large language models in multiple languages with reinforce- ment learning from human feedback. In Feng, Y. and Lefever, E. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 318-327, Singapore, De- cember 2023. Association for Computational Linguis- tics. doi: 10. http:3/www122223mmmm-eeee. 228. URL. https:///cclanthology/ www.//2...... htt.:/2......mmmmmmmeemmmmm comnnlpddem..cm 28.
Reid, M., Savinov, N., Teplyashin. D., Lepikhin, D., Lilli- crap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05532 2024.
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, Y., Dirani, J., Michael, J., and Bowman, R. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.1202. 2023.
Lassak, L., Pan, E., Ur. B., and Golla, M. Why aren't we using passkeys? obstacles companies face deploying FIDO2 passwordless authentication. In 33rd USENIX Security Symposium (USENIX Secu- rity 24), pp. 7231-7248, Philadelphia, PA, August 2024. USENIX Association. ISBN 978-1-939133.
Rumbelow, J. and Watkins, M. SolidGoldMagikarp (plus, prompt generation). https: /ttps://ww..1. lesswron..
2023.
Salton, G. and Buckley, C. Teimhheighting approaches in automatic text retrieval. Information processing & management 24(5):513-55) 1988.
44-1. URL https::/www/ httpp://www.useeiix con/ccnneeeecc// uenixsscarityy///reeettttttaaass

---

Exploring and Mitigating Adversarial Manipulation of Voting- Baang- Leaderboards
Salton, G., Wong, A., and Yang, C.-S. A vector space model for automatic indexing. Communications of the ACM, 18 (11):613-620, 1975.
of large language models in logical reasoning. arXiv preprint arXiv:2410.23123, 2024.
Xu, R., Wang, Z., Fan, R.-Z., and Liu, P. Benchmark- ing benchmark leakage in large language models. arXiv preprint arXiv:2404. 18824, 2024.
Shi, F., Suzgun, M., Freitag, M., Wang, X., Srivats, S., Vosoughi, S., Chung, H. W. Tay, Y., Ruder, S., Zhou, D., Das, D., and Wei, Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference Learning Representations, 2023. URL https://openreve ew. met/fforu?id==FR3de
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sen- tence? In Korhonen, A., Traum, D., and Mårquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791- 4800, Florence, Italy, July 2019. Association for Compu- tational Linguistics. doi: 10. 10.18653/11/P99-4477 1472. URL hwtp:/hhoooooooooo11roee//1 org.:/9910//1/1177
Shi, W., Ajith, A., Xia, M., Huang. Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L.. Detecting pretraining data from large language models. In The Twelfth International Conference Learning Representations, 2024. URL https://openrewt
net/forunlid=zwee 3MQUN..
Zhai, E., Wolinsky, D. I., Chen, R., Syta, E., Teng, C., and Ford, B. (AnonRep): Towards {Tracking- TTracking-Resistannt anonymous reputation. In 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI 16), pp. 583-596, 2016.
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., , al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL //penrrvee.//eeeeeeeeeeee net/ forum?id=uuTLSBvoos Featured Certification.
Zhao, W., Rush, A. M., and Goyal, T. Challenges in trustworthy human evaluation of chatbots, 2024. URL https://arxiv
04363.
Sun, Z., Schuster, R., and Shmatikov, De-anonymizing text by fingerprinting language generation. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 22420-22431. Curran As- sociates, Inc., 2020. URL https://prrt /proce//gs.... cc/ wttpcccppppppaaeeeeeeeeee / fdf2aade29d18911oom 0051 a6c76a6661 a6c Baeee666666-PPPaa Paper.ppf
Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Z., Lin, Z., Xing, E. P., Gonzalez, E., Stoica, I., and Zhang, H. Lmsys-chat- Im: A large-scale real-world Ilm conversation dataset, 2023a.
Zheng, L., Chiang, W.-L., Sheng. Y.. Li, T., Zhuang, S., Wu, Z.,, Zhuang, Y., Li, Z., Lin, Z., Xing, E. P., al. Lmsys-chat-Imm A large-scale real-world 11m conversa- tion dataset. arXiv preprint arXiv:2309. 1tt::2309.199999 11988, 2023b.
Tay, Y., Bahri, D., Zheng, C., Brunk, C., Metzler, D., and Tomkins, A. Reverse engineering configurations of neural text generation models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 275-279, 2020.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z. Zhuang, Y., Lin, Z.,, Li, 7., Li, D., Xing, E., al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023c.
VotusTotal. Results reports. VitusTotal Documenta- tion, 2024. URL https: http::///oos..vruustotoooo virustotal com/docs/ results-reports. Accessed on December 19, 2024.
Zhu, L., Wang, X., and Wang, X. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310. 1 2023.
Wang, Y., Mansurov, J., Ivanov, P., Su, J., Shelmanov, A., Tsvigun, A., Afzal, O. M., Mahmoud, T., Puccetti, G., Amold, T., al. M4gt-bench: Evaluation benchmark machine-generated
Zou, A., Wang. Z., Carlini, N., Nasr, M., Kolter, J. Z.. and Fredrikson, M. Universal and transferable adversar- ial attacks on aligned language models. arXiv preprint 2307 /5043, 2023.
for black-bor box machine- generated text detection. arXiv preprint 2rXiv:2402.11112 2024.
Wu, X., Yu, D., Huang. Y., Russakovsky, O., and Arora, S. Conceptmix: A compositional image generation benchmark with controllable difficulty. arXiv preprint arXiv:2408. 14339, 2024.
Xie, C., Huang, Y., Zhang, C., Yu, D., Chen, X., Lin, B. Y., Li, B., Ghazi, B., and Kumar, R. On memorization
12

---

Exploring and Mitigating Adversarial Manipulation of Voting- Baaeg Leaderboards
A. Discussion
Upvoting onc's own models vs downvoting those a competitor. is far easier for a model owner to upvote their own model(s) than to downvote (or upvote) another. Model owners have much more knowledge about their models. They know the entire training dataset and can evaluate the loss on each sample to determine the casiest samples detect. Further, if API, they could simply log generations that the API produces, and then check each candidate in Chatbot Arena against this database. Finally, the model owner can also strategically make text more detectable, either by using stcalthy watermarks that only they have direct knowledge of or by using hidden backdoors on specific prompts. In contrast, our approach in $ 2.2 aims to address the case where the adversary does not necessarily have control over the models whose scores they aim to manipulate.
their model is deployed
Detection via watermarking. There has been a slew of recent research aiming to watermark generated text to identify whether given text was generated with a particular, watermarked model (Kirchenbauer et al., 2023; Kuditipudi et al., 2024; Christ et al., 2024). This is indeed a way of breaking model anonymity but it has limited applicability for our task. Not all models employ watermarking, and successful de-anonymization would require the attacker to know the specifics of the watermarking implementation in the target iaadell-inffomattiinn that is typically not public.
Implications for public evaluation of AI systems. While this paper focuses on Chatbot Arena, our findings our relevant for any public platform for performing comparative evaluation Al systems, such as ones deployed for evaluating text-to to-image and speech. There a fundamental tension when designing human evaluation experiments. On one hand, human evaluation paradigms that closely reflect real-world usage lend validity the results. On the other hand, restricting human evaluation to known groups of annotators lends greater control annotator qualifications, demographic makeup, and incentives- but at the expense of the transferability of the findings to real-world usage. For example, prior work has shown that Amazon Mechanical Turk workers rate generated text very differently than school teachers (Karpinska et al., 2021).
B. Malicious User Identification
We consider two scenarios, one where the defender can only estimate a benign user's behaviour and another where the defender can estimate both defender and attacker behavior.
Scenario 1: Known Benign Distribution
In this scenario, we assume that a defender can estimate the expected behaviour for benign users using historical data from previous votes. Now, if an adversary behaves significantly differently from the expected behaviour, the defender can detect it. To do so, we use a likelihood test to differentiate between the null hypothesis Heenig that the user's 's voting pattern matches the known benign distribution or the alternative hypothesis H-benign that the user is from different source.
Let (#1, nn) represent sequence of observed impressions by user, where each x; is an impression for one of the available models. Under the null hypothesis Heenign, we assume these votes come from the known benign user profile. Also we assume each vote is independent of each other.
The likelihood of observing the entire sequence under the null hypothesis is then:
L benign) ) =
II
Pr (x:. H benigu).
(1)
x
To assess how extreme this observation is under the null hypothesis, we use the test statistic:
T(x) -2 2n (I< Hhenigg)):
(2)
To determine statistical significance, we simulate m sequences under the null hypothesis, where each vote is generated according to the known benign probabilities. For each simulated sequence sأ, we calculate its test statistic T(s'). The empirical p-value is then computed as:
1 Σ I(T(s))
where 7() is the indicator function. We reject the null hypothesis (and conclude the user is likely not the known benign user) if the p-value is less than the desired significance level a. In particular we use a 0.01 in our evaluations.
"nttps:/aartificiaaaalyiiss
13

---

Exploring and Mitigating Adversarial Manipulation of Voting- Based Leaderboards
Scenario 2: Known Benign and Malicious Distributions
Because the leaderboard is public, the adversary use the published ratings and counts to make themselves more difficult to detect by mimicking the average user behavior. To this end, the defender can instead release perturbed rankings and counts to each user so as to reduce an attacker's ' knowledge of the true values. This comes with security- tradcoff with benign users which we discuss later in this section.
We use the same null hypothesis Hhenign and alternative hypothesis H-benign- Similarly, let (r E [n] be the probability of benign user voting for model and Pr-p(i) (i) the same for adversarial users. However, note that Pr-p (i) will match the Neyman-Pearson perturbed votes released by the defender. We use the Neyman- Pearson Lemma construct the hypothesis test. The
Neyman- Lemma states that the optimal decision rule is based on the likelihood ratio.
The likelihood ratio is defined as:
(r)
A(x) = (r
(4)
The Bradley-Tene coefficient rating difference between two models defines the probability with which will be preferred over the other. We can use this calculate the entire probability distribution (r and Pr-s(i). Given two models and j with ratings Qi and Qj respectively, the probability that i is preferred is typically modeled using logistic function as:
Pr(i preferred over j) = 1 + -xp(-((): 22)/s)
(5)
-
where s a scaling factor that determines the sensitivity of the probability to the rating difference, Then, we can calculate any component (i) (or (i) similarly) the event that this model is chosen over each other model. This is calculated
Pr( II Pr( preferred over j true Bradley-Terry coefficient ratings)
(6)
For Pr (i), the perturbed Bradley-Terry coefficient rankings used instead.
14

---

Exploring and Mitigating Adversarial Manipulation of Voting- Baang Leaderboards
C. Experimental Details
C.1. List models
Table lists the evaluated models and the methods used to query them. For all models, we rely on the default decoding hyperparameters (e.g., temperature) specified by the query method.
Table Overview of evaluated models and the querying methods used in our experiments.
Model
Company Organization
Method of query in our experiments
dlaude-3-5-sommee222246663
Anthropic
Anthropic API
Saaude-3-haiku-2024000
Anthropic
Anthropic API
gemini-1.5-flash
Google
Google Al studio API
gemini-1.55 p.rp
Google
Google AI studio API
semma-2-2b-it
Google
Together AI Inference API
gemma-2-9h0 2--9iitt
Google
Together AI Inference API
gemma-2-27b-it
Google
Together AI Inference API
gpt-3.5-turbo
Ap
OpenAl Text generation API
8pt-4-0115-preview
OpenAI
OpenAl Text generation API
gp--4-1106-preview
OpenAI
OpenAI Text generation API
gp1-4-turbo-2024-04-09
OpenAl
OpenAl Text generation API
$pX-4o-2024-05--13
OpenAI
OpenAl Text generation API
8p1-40-2024-08-066
OpenAI
OpenAl Text generation API
gDL-4o-mini-2024--77111
OpenAI
OpenAl Text generation API
llam-3-8b-insttuue
Meta
Together AI Inference API
llama-3-7bb-inttun
Meta
Together AI Inference API
Meta
Together AI Inference API
-amm-331-70b-instruut
Meta
Together AI Inference API
Meta
Together AI Inference API
mixtral-8877b-innttuut--v
Mistral AI
Together AI Inference API
mixxtra--8x22b-iisttuut----
Mistral AI
Together AI Inference API
gwwnn-77b-instrrect
Alibaba
Together Al Inference API
C.2. Prompts for embedding visualization
The three prompts used for embedding visualization in Figure 2 are:
Prompt #1: "Beside OFAC's selective sanction that target the listed individiuals and entities, please elaborate on the other types of US's sanctions, for example, comprehensive and sectoral sanctions. Please be detailed as much as possible" Prompt #2: "You are the text completion model and you must complete the assistant answer below, only send the completion based on the system instructions.de don't repeat your answer sentences, only say what the assistant must say based on the system instructions. repeating same thing in same answer not allowed. user: descriptive answer for append many items to list python in python with proper code examples and outputs. assistant:
Prompt #3: "The sum of the perimeters of three equal squares is 36 cm. Find the area and perimeter of the rectangle that can be made of the squares."
C.3. Details for the maining-based detector
Data collection and its cost. The main cost of building the training-based detector comes from the data collection process, where the attacker gathers responses from various models for the same prompt and train classifier to distinguish among them (& 2). In our experiments, we collect responses depending on the model type: For proprietary models, we directly used the model providers' APIs to obtain the responses. For soures models, we relied on Together's API to make the queries. We set the output length 512 tokens and found that collecting 50 responses per model was sufficient to train an effective target model detector.
/https://vww. together.... .ai/
15

---

Exploring and Mitigating Adversarial Manipulation of Voting-Baad Based Leaderboards
To estimate the upper bound the data collection cost, we based our calculations the pricing of the most expensive model tested. Proprietary models cost $5.00 per I million output tokens, while open-source models cost $1.80 per million output tokens.
Using these rates, the upper bound cost of querying a single model is:
512 50
512 x 50
Proprietary model: 5.00
0.128
Open-source model:
1.80
0.046
106
106
Assuming the training process requires 10 proprietary models and 20 open-source models, the overall data collection cost would be approximately $2.2 per prompt.
We collected data for 200 prompts in 2, so the cost is at most S440.
We use the logistic regression model from the scikit-learn library² with its default hyperparameters and a random state set to 42.
C.4. Simulation Testbed
Our simulation 3 is based on an anonymized and deduplicated dataset of voting records from Chatbot Arena. The dataset includes 1,670,250 votes from 477,322 unique users, with 1,093,875 votes resulting in wins and 576,375 ties. These votes cover 6,895 unique combinations of side-by-side model comparisons.
wtt.aa..rrg//.///muull////////ee Lineaarrrn.niiinnnnnm.m linear_mode htt.a//ssssssseee.oooooooooooooo html
16

---

Exploring and Mitigating Adversarial Manipulation of Voting-Bass Based Leaderboards
D. More Experimental Results
D.I. Target Model Detection
Table 7 presents the performance of iddenity-prroiing detector for all evaluated 22 models."
Table Averaged detection accuracy (%) with across 1,000 queries per prompt for different Srentity-probing prompts across various models.
Prompt
Model
Who are you?
Which model are you?
What is your model name?
How should refer to you
How would you define
your role or
AI?
identity AI?
0unude-3---oomnet222400005
99.3
100.0
98.5
100.0
100.0
claude3-hhaiku-222403000
100,0
96.3
100.0
42.9
14.3
gemini-1.5-ffan
0.0
0.0
0.0
0.0
0.0
gemini-1.5-pre
97.2
96.5
100.0
0.0
99.1
gemma-2-27biit
100.0
98.4
98.2
97.9
95.5
gemma-2-2b-it
81.8
91.8
58.2
12.7
4.5
gemma-2-9b-it
98.5
99.4
98.3
98.1
97.3
gpt-3.5-turbo
0.0
54.5
67.3
0.0
0.0
1pp-4-0115-previeet
70.9
100.0
94.6
1.8
1.8
gpt-4-1106-previinn
7.3
90.9
99.1
6.4
1.8
gpt-4o-2024405-13
16.4
93.3
99.9
0.0
6.4
gpt-4o-2024-08-06
51.8
97.7
98.5
0.0
5.5
spt-4o-mini-2024--0--88
92.7
92.9
100.0
12.7
0.0
llama-3-70--instruet
98.2
98.2
54.5
46.4
2.7
llama---bbiinsruuct
99.9
99.1
74.5
20.0
1.8
1ama--3.-405b-insttue
99.1
90.9
89.1
75.5
0.0
Ilama-3.1-700
98.8
66.4
92.7
5.5
0.0
-omma-3.--b-insttuuc
17.3
40.0
99.1
6.4
0.0
mixtral-&&7b-instrrcee--
97.3
31.8
45.5
1.8
0.9
mixtral-8x22b-instrretta
97.3
31.8
45.5
0.9
1.8
qwen2-72b-instruet
91.8
98.2
97.6
24.5
7.3
D.2. Adversarial Vote
Ablation for detector accuracy. Table shows the number of votes and interactions needed to shift model's position by to places on the simulated leaderboard under different detector accuracies. shown, the number of votes required to move a model up by 50 places increases by only about 150 when the detector accuracy drops from 1.0 0.9. This suggests that detector, while not perfect, can still be sufficiently accurate to achieve the attack's 's objective.
Ablation for non-detected actions. When the attacker does not detect the target model, they can choose from four actions: randomly upvote one model, vote for tie, vote both models as bad, or do nothing. The main results in 3 assume the attacker does nothing. We also explore the other options in Table 9, As shown, there are no clear patterns indicating that any one option is significantly better than the others.
We note that operation of these models was by University authors.
17

---

Exploring and Mitigating Adversarial Manipulation of Voting-Based Based Leaderboards
Table 8. The number of votes (a) and interactions (b) required to change the ranking of a low-ranked model on the simulated leaderboard, under varying detector accuracy.
Target model=Ilama-133
Target rank:
Target rank:
Target rank:
Target rank:
Target rank:
Target rank:
(current rank: #129, #votes: 2443)
79 (f 50)
109 (1 20)
119 (1 10)
124 (1 5)
127 (1 2)
128 (1 1)
detector acc=1.0
1246
861
208
126
detector acc=0)95
1304
918
126
detector acc=0.9
1383
1012
732
525
271
136
(a) # Votes
Target model=llama-33b
Target rank:
Target rank:
Target rank:
Target rank:
Target rank:
Target rank:
(current rank: #129, #votes: 2443)
79 (t 50)
109 (1 20)
119 (t 10)
124 (1 5)
127 (1 2)
128 (1 1)
detector acc=1.0
80000
55000
40000
30000
15000
10000
detector acc=0195
85000
65000
45000
30000
15000
10000
detector acc=0.9
100000
75000
55000
40000
20000
10000
(b) # Interactions
Table 9. The number of interactions required to change the ranking of a high-ranked model (a) and low-ranked model (b) the simulated leaderboard, under varying non-target strategies.
Non-target strategy
Target rank: 1(1
Target rank: 2(f
Target rank: 3(1 2)
Target rank: 4(1 1)
Do nothing
206000
184000
144000
18000
Randomly upvote
192000
182000
142000
16000
Vote tie
194000
182000
148000
20000
Vote tie (both bad)
196000
172000
152000
16000
(a) High-ranked model,
20224466000 (rank: #5)
Target rank:
Target rank:
Target rank:
Target rank:
Target rank:
Target rank:
Non-target strategy
79 (f 50)
109 (+ 20)
119 (f 10)
124 (1 5)
127 (t 2)
128 (t 1)
Do nothing
80000
55000
40000
30000
15000
10000
Randomly upvote
75000
60000
40000
30000
15000
10000
Vote tie
80000
60000
40000
30000
15000
10000
Vote tie (both bad)
80000
60000
40000
30000
15000
10000
(b) Low-ranked model, llama-133 13b (rank: #1 #129)
18


Reviews:
[{"review_id": "U3NgWPINfb", "reviewer": "Reviewer_X39U", "summary": "This submission examines the susceptibility of voting-based LLM assessment platforms to adversary interference, particularly emphasizing Chatbot Arena, a prominent platform that ranks language models according to human preferences. \n\n The primary contributions of the paper are: (1) Evidence that users can effectively compromise model response anonymity with high precision (>95%) employing basic classification methods; (2) Simulation-based estimations indicating that a few thousand adversarial votes can substantially modify a model's ranking on the leaderboard; (3) Formulation of a cost model for the attack and the proposal of various mitigation strategies to enhance the expense and complexity of such attacks; (4) Collaboration with the Chatbot Arena developers to execute these mitigations, exemplifying a responsible disclosure methodology in security research.\n\n The authors assert that their findings are relevant to any voting-based ranking systems, not alone Chatbot Arena, underscoring a wider issue for platforms dependent on human preferences for assessment.\n\n## Update after rebuttal\nThe authors provided detailed discussions with additional experiments to address my concerns. I appreciate their effort and would like to retain my original score to accept this submission.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and questions. Our detailed responses are as follows:\n\n> Q1: The proposed technique demonstrates high accuracy in de-anonymizing model answers; nonetheless, LLMs are regularly updated. In what manner may the efficacy of your detection techniques be altered when models undergo updates and fine-tuning?\n\n**A**: We appreciate the question. We first note that updates to most models via API are relatively infrequent. Assuming that there are (infrequent) updates to the model, the detector efficacy post-update depends on the adversary:\n- When the adversary controls the target model, the adversary can easily retrain its detector to reliably detect the new model. \n- When the adversary does not have the control of the target model, our experiments with llama-3-8b-instruct and gemma-2-27b-it suggest that the detector is relatively robust to model changes: as shown in Table 1, detection accuracy remained above 90% even after several hundred SFT steps on these models.\n\nTable 1: The original detector’s accuracy on the SFTed model with different steps\n| # SFT steps on lmsys-chat-1m | Model: llama-3-8b-instruct | Model: gemma-2-27b-it |\n|------------------------------|----------------------------|-----------------------|\n| 0                            | 95.4                       | 96.3                  |\n| 100                          | 95.2                       | 95.6                  |\n| 200                          | 94.7                       | 95.1                  |\n| 500                          | 94.1                       | 94.7                  |\n\n> Q2: The simulations concentrate mostly on singular opponents; however, how may the vulnerability escalate with coordinated assaults from numerous adversaries?\n\n**A**: We appreciate the question. On voting-based leaderboards, users typically evaluate models through head-to-head comparisons. We're curious to understand what the reviewer meant by \"numerous adversaries\" and would appreciate more context. \n\n> Q3: As leaderboards expand to encompass additional models, does the efficacy of this defensive strategy alter?\n\n**A**:  Thank you for pointing this out. Yes, the strategy's effectiveness will change as the number and type of models on the leaderboard evolve. Newer models might have different performance characteristics or vulnerabilities, impacting the defense's relative success. However, as discussed in the paper, each defense involves a utility-cost trade-off. This allows system designers to analyze its effectiveness under current conditions and adjust parameters to meet system requirements at the time of deployment. We will clarify this in the paper.", "claims_and_evidence": "The claims made in the submission are generally well-supported by empirical experiments.\n\nThe claim that model responses may be de-anonymized is supported by Figure 3, which presents detection accuracy across various prompts and models.\n\nThe claim that the quantity of votes might influence the leaderboard is supported by simulations utilizing actual voting data shown on page 16. Tables 4 and 5 include comprehensive estimates of the votes and interactions necessary to alter the ranks of both high-ranked and low-ranked models.", "methods_and_evaluation_criteria": "The methodologies and assessment standards are effectively aligned with the objectives of the submission.\n\nThe procedures and criteria included in the submission are suitable for evaluating the proposed approach. Utilizing accuracy metrics is appropriate for evaluating detector performance. The authors show the separability of model responses by principal component analysis of bag-of-words characteristics, successfully demonstrating \"how different models respond to the same prompt\" (Figure 2).\n\nThe suggested mitigations are evaluated based on their efficacy in increasing the cost of the assault. The authors assess the security benefits and potential drawbacks of each mitigation, including the distributional changes that may occur when authentication is required.\n\nThe simulation technique for determining the necessary votes to affect the ranking is meticulously designed. The author delineates clear objectives for the attack: \"Up(M, x)\" and \"Down(M, x)\" (page 5). The simulations account for the attacker's detection accuracy and behavior when the target model is not identified.", "theoretical_claims": "The paper does not make extensive theoretical claims or provide formal proofs, as it is primarily an empirical study focused on demonstrating and mitigating a practical vulnerability.", "experimental_designs_or_analyses": "The experimental designs and analyses specified in the submission are meticulously crafted.\n\nThe de-anonymization assessments are meticulously designed, scrutinizing both identity-probing and training-based detectors across a diverse range of prompts and models.\n\n The authors performed ablation studies to assess the impact of different design selections on detector efficacy.  The authors examine several feature types and conclude that \"basic text features such as BoW and TF-IDF achieve remarkably high detection accuracy, with BoW surpassing 95% in multiple cases\" (page 4).\n\nThe simulation studies aimed at estimating the votes necessary to affect the leaderboard include real voting data from Chatbot Arena, hence enhancing the credibility of the results.  The authors analyze many scenarios and provide detailed findings in Tables 4 and 5.", "relation_to_broader_scientific_literature": "The paper places its contributions within the broader scientific literature on voting-based systems, and security vulnerabilities.\n\nThe paper connects to the literature on voting-based systems and their security vulnerabilities. The authors note that \"voting-based systems are frequently used in security relevant scenarios, such as for malware identification [1] or for content validation [2]\" and that \"attacks on these systems are well studied [3]\" (page 7). They also discuss reputation systems as a common approach to securing these systems, citing work by [2] and [4].\n\nThe de-anonymization task is related to authorship attribution and model detection, as the authors acknowledge: \"Our primary attack involves training a classifier that can identify which language model system produced a given generation. This task is related to the much older task of authorship attribution—identifying the authors of anonymous (but human-written) works of writing [5,6]\".\n\nRef:\n[1] VirusTotal Documentation, 2024.\n[2] The eigentrust algorithm for reputation management in p2p networks. In WWW, 2003.\n[3] A survey of attack and defense techniques for reputation systems. In CSUR, 2009.\n[4] Towards {Tracking-Resistant} anonymous reputation. In NSDI, 2016.\n[5] Authorship attribution in the era of llms: Problems, methodologies, and challenges. In arXiv, 2024.\n[6] De-anonymizing text by fingerprinting language generation. In NeurIPS, 2020.", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "N.A.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "- The proposed technique demonstrates high accuracy in de-anonymizing model answers; nonetheless, LLMs are regularly updated. In what manner may the efficacy of your detection techniques be altered when models undergo updates and fine-tuning? \n\n- The simulations concentrate mostly on singular opponents; however, how may the vulnerability escalate with coordinated assaults from numerous adversaries?\n\n- As leaderboards expand to encompass additional models, does the efficacy of this defensive strategy alter?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zTnEA4jq78", "reviewer": "Reviewer_hmrX", "summary": "It has become common for LLMs to be evaluated subjectively in crowd-sourced \"arenas\", which usually use elo-based scoring based on user preferences. The authors study this voting-based evaluation setting, and find that they are susceptible to adversarial manipulation through a two-step attack: (1) de-anonymizing model responses with high accuracy, and (2) selectively voting for or against a target model to manipulate the rankings. They demonstrate that this attack is feasible and cheap, and then explore mitigations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and questions. Our detailed responses are as follows:\n\n> Q1: Sections 4.2.3 and 4.3 could be revised to be clearer…Specifically, 4.3 should clarify that it is experimenting based off of the proposed defenses in 4.2.3 (and has nothing to do with 4.2.1, 4.2.2, 4.2.4, if my understanding is correct). 4.2.3 can be rewritten to more clearly present the attacker's strategy.\n\n**A**: We appreciate the reviewer’s suggestion of reorganizing the defense section (Sec 4). We apologize for the confusion caused by presenting the results for malicious user identification in Sec 4.3 without a clear pointer to Sec 4.2.3. We will reorganize these sections in the final version.\n\n> Q2: For the \"perturbation\" defense in 4.2.3, could the attacker simply create multiple accounts, aggregate the perturbed numbers across the many accounts, and then use the average?\n\n**A**: No, please note that only one permuted table is released by the system therefore you cannot average or detect the permutations. \n\nQ3: Typos.\n\n**A**: Thanks! We will fix them in the final version.", "claims_and_evidence": "The claim that model responses can be reliably distinguished by malicious parties is supported by thorough empirical experimentation, across many models. An accuracy of >95% can be achieved using extremely simple supervised learning methods.\n\nThe authors verify the cost-related claim by running simulations on a system emulating Chatbot Arena.", "methods_and_evaluation_criteria": "The methods seem appropriate and well-designed.\n\nThe paper targets an attack on a real-world system: Chatbot Arena. The ultimately convincing evaluation would be to attack the actual system, and show that such an attack is feasible in the real world. The authors instead evaluate on a *simulation* of Chatbot Arena, based upon real-world voting records from Chatbot Arena. This seems like a proper evaluation setup to me.", "theoretical_claims": "The paper does not necessitate any proofs or theoretical claims.", "experimental_designs_or_analyses": "The experimental designs and analyses appear sound to me.", "relation_to_broader_scientific_literature": "This work is related to the body of literature studying the validity of benchmark evaluation numbers, which is an important field, as new models are judged based on the evaluation numbers that they achieve.", "essential_references_not_discussed": "I don't know of any references that are relevant and missing.", "other_strengths_and_weaknesses": "Strengths:\n- The paper is very well written and organized, and it for the most part quite easy to follow\n- The topic is very important. Every major model release highlights voting-based elo scores.\n\nWeaknesses:\n- Sections 4.2.3 and 4.3 could be revised to be clearer.\n  - I think the paper would benefit from a rewriting of these sections, as they were unclear to me the first time I read them.\n  - Specifically, 4.3 should clarify that it is experimenting based off of the proposed defenses in 4.2.3 (and has nothing to do with 4.2.1, 4.2.2, 4.2.4, if my understanding is correct).\n  - 4.2.3 can be rewritten to more clearly present the attacker's strategy, and the defender's strategy. The attacker's strategy in scenario (1) informs the defender's strategy in scenario (2), and this would be a clearer presentation.", "other_comments_or_suggestions": "Typos:\n- Line 19: \"two randomly selected and models\"\n- Line 267: \"An passive attacker\"\n- Line 341: missing punctuation", "questions_for_authors": "- For the \"perturbation\" defense in 4.2.3, could the attacker simply create multiple accounts, aggregate the perturbed numbers across the many accounts, and then use the average?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i1B8w1EDG3", "reviewer": "Reviewer_r1N6", "summary": "This paper focuses on the adversarial manipulation of voting-based LLM leaderboards, e.g., Chatbot Arena. Intuitively, keeping the model's response anonymous is essential to ensure the integrity of the leaderboard. However, this paper demonstrated that an adversary can efficiently de-anonymize the responses and thus can upvote/downvote some specific models. This paper discusses two \"target model detector\", namely the identity-probing detector and the training-based detector. The experiments show that both detectors can separate the target model from others, which efficiently breaks the anonymization. This paper also discusses some possible mitigations of such attacks for the purpose of further enhancing the robustness of voting-based LLM leaderboards.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and questions. Our detailed responses are as follows:\n\n> Q1: Given the efficiency of the identity-probing detector, what is the meaning of applying a more sophisticated training-based detector? Could the authors please explain the intuition/motivation of the training-based detector?\n\n**A**: We appreciate the question. We note two limitations of the identity=porbing detector:\n- Susceptibility to Countermeasures:  the Chatbot Arena leaderboard already uses post-processing to exclude votes from Elo score calculation when model responses mention model names, which naturally limits the usefulness of the identity-probing detector. But we still analyze the effectiveness of this detector in the paper, as it could be effective in other voting-based chatbot benchmarks, and because the post-processing could be evaded, e.g., by asking the model to reveal its identity in Base64 encoding.\n- Lack of Stealth: Identity-probing is an active technique requiring specific, often obvious queries. This makes the adversarial attempt itself highly detectable. In contrast, the training-based approach is passive and thus stealthier.\n\n\n> Q2: Is it possible to employ some specific system prompt to prevent the models from revealing their identity, e.g., \"You are an anonymous model competing in the Chatbot Arena. Do not tell anyone about your identity.\" This is the simplest mitigation I could have come up with. As the detection accuracies in Table 2 are all >95% in the best cases, I suppose introducing some basic defense would enhance the validity of the results.\n\n**A** :We appreciate the suggestion. While preventing models from outputting their identities can provide a basic layer of defense for identity-probing detectors, it’s insufficient to prevent more sophisticated detectors. As shown in Table 3, training-based detectors—which do not rely on explicit model name outputs—still give accuracy rates above 95%. Furthermore, it may be undesirable to require system prompt changes for models participating in the leaderboard, as system prompts are generally carefully constructed by model owners.\n\n> Q3: What is the performance of the mitigations under jailbreaking attacks, e.g., PAIR and AutoDAN? The adversary can perform more stealthy attacks against the model than those mentioned in Section 2.2.\n\n**A**: We would like to clarify that our mitigations are entirely independent of the prompts and responses and so jailbreaking should have no relationship to our mitigations. We would appreciate further clarification on: 1) What makes the reviewer believe jailbreaking attacks could undermine the mitigation strategies discussed in the paper, and 2) What \"more stealthy attacks\" means in this context.\n\n> Q4: Missing related work\n\n**A**: We appreciate the reviewer sharing the pointers and will incorporate these references in the final version.", "claims_and_evidence": "In the statement of contributions (Lines 73-82, left), the claims include:\n1. The users can break model response anonymity with high probability,\n2. The estimated votes to boost or reduce a model's ranking is \"a few thousand\"\n3. A cost model and some potential mitigations are discussed.\n\nGenerally speaking, I think the claims are well supported. However, I find the evidence for the first claim is not convincing enough. See the Questions For Authors part.", "methods_and_evaluation_criteria": "The methods to de-anonymize the model response include \n1. Identity-probing detector\n2. training-based detector\nWhile the evaluation of the identity-probing detector is straightforward, I am not sure if I have fully understood the training-based detector, see the questions in the Questions For Authors part.", "theoretical_claims": "This paper does not include theoretical analysis.", "experimental_designs_or_analyses": "I have checked the soundness/validity of the experimental designs or analyses in this paper.", "relation_to_broader_scientific_literature": "1. According to the related work section, this paper seems to be the first work focusing on the adversarial manipulation of voting-based LLM leaderboards. The related works discussed in Sections 5 and A are not directly related to the topic of the present paper. \n2. The setting of this paper is similar to jailbreaking attacks against LLMs. See the Questions For Authors part.", "essential_references_not_discussed": "The following two methods are related to a similar topic of \"LLM identification\". \n\n1. Gubri, M., Ulmer, D., Lee, H., Yun, S., and Oh, S. J. TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification. In Annual Meeting of the Association for Computational Linguistics (ACL). ACL, 2024.\n2. Jin, H., Zhang, C., Shi, S., Lou, W., and Hou, Y. T. ProFLingo: A Fingerprinting-based Copyright Protec\u0002 tion Scheme for Large Language Models.  CoRR abs/2405.02466, 2024.\n\nCould the authors please discuss on the relationship between the present paper and these two papers and the references therein?", "other_strengths_and_weaknesses": "**Strenths**\n\n1. Very practical. As mentioned in Line 036, the authors claimed that they work with the Chatbot Arena developers and have enhanced the robustness of the voting-based leaderboards based on their analysis.", "other_comments_or_suggestions": "Typo on Line 104 (right): its name", "questions_for_authors": "1. Given the efficiency of the identity-probing detector, what is the meaning of applying a more sophisticated training-based detector? \n2. Could the authors please explain the intuition/motivation of the training-based detector?\n3. Is it possible to employ some specific system prompt to prevent the models from revealing their identity, e.g., \"You are an anonymous model competing in the Chatbot Arena. Do not tell anyone about your identity.\" This is the simplest mitigation I could have come up with. As the detection accuracies in Table 2 are all >95% in the best cases, I suppose introducing some basic defense would enhance the validity of the results.\n4. What is the performance of the mitigations under jailbreaking attacks, e.g., PAIR and AutoDAN? The adversary can perform more stealthy attacks against the model than those mentioned in Section 2.2.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Reviewers unanimously voted for acceptance.

- Immediate security issue for Chatbot arena
- The method makes sense
- Empirical results are fairly strong
Decision: Accept (oral)