Improving Your Model Ranking on Chatbot Arena by Vote Rigging
Authors: Rui Min
Tianyu Pang
Chao Du
Qian Liu
Minhao Cheng
Min Lin
Venue: ICML 2025 poster (2025)
Keywords: Chatbot Arena
Vote Rigging
Large Language Models
Leaderboard Manipulation
Primary area: deep_learning->large_language_models
TL;DR: We demonstrate that we could improve model rankings on Chatbot Arena by vote rigging.

Abstract: Chatbot Arena is an open platform for evaluating LLMs by pairwise battles, in which users vote for their preferred response from two randomly sampled anonymous models. While Chatbot Arena is widely regarded as a reliable LLM ranking leaderboard, we show that crowdsourced voting can be *rigged* to improve (or decrease) the ranking of a target model $m\_{t}$. We first introduce a straightforward **target-only rigging** strategy that focuses on new battles involving $m\_{t}$, identifying it via watermarking or a binary classifier, and exclusively voting for $m\_{t}$ wins. However, this strategy is practically inefficient because there are over $190$ models on Chatbot Arena and on average only about 1% of new battles will involve $m\_{t}$. To overcome this, we propose an **omnipresent rigging** strategy, exploiting the Elo rating mechanism of Chatbot Arena that any new vote on a battle can influence the ranking of the target model $m\_{t}$, even if $m\_{t}$ is not directly involved in the battle. We conduct experiments on around *1.7 million* historical votes from the Chatbot Arena Notebook, showing that omnipresent rigging strategy can improve model rankings by rigging only *hundreds of* new votes. While we have evaluated several defense mechanisms, our findings highlight the importance of continued efforts to prevent vote rigging. [**Code**](https://github.com/sail-sg/Rigging-ChatbotArena) is publicly available to reproduce all experiments.

Improving Your Model Ranking on Chatbot Arena by Vote Rigging
Rui Min 12
Tianyu Pang" Chao Du
Qian Liu
Minhao Cheng?
Min Lin'
Abstract
15
wio Rigging
Chatbot Arena
Chatbot Arena is popular platform for evaluating LLMs by pairwise battles, where users vote for their preferred response from two randomly sam- pled anonymous models. While Chatbot Arena widely regarded as a reliable LLM ranking leader- board, we show that crowdsoureed voting can be rigged to improve (or decrease) the ranking of a target model 172 We first introduce straightfor- ward target-only rigging strategy that focuses on new battles involving my, identifying it via wa- termarking or a binary classifier, and exclusively voting for my wins. However, this strategy is prac- tically inefficient because there are over 190 mod- els on Chatbot Arena and average only about 1% of new battles will involve my To overcome this, we propose omnipresent rigging strategies, exploiting the Elo rating mechanism of Chatbot Arena that any new vote on a battle can influence the ranking of the target model my, even if m, not directly involved in the battle. We conduct experiments around 1.7 million historical votes from the Chatbot Arena Notebook, showing that omnipresent rigging strategies can improve model rankings by rigging only hundreds of new votes. While we have evaluated several defense mech- anisms, our findings highlight the importance of continued efforts to prevent vote rigging. Code publicly available to reproduce all experiments.
Ranking Improvement
12
Target-Only
After
Rank
Model
Omnipresent
741
m
:
(Atter rigging)
Rank :
Model m :
Befom
89
1
:
(Fdoore rigging)
of New Votes (x10²)
Figure I. We simulate rigging on new votes beyond the 1.7 million historical votes from the Chatbot Arena Notebook. In this demo, set the target model m; as Phi-3-small-8k -t-sstctl- Under the normal voting distribution (w/o rigging). the ranking remains steady. showing only single rank increase despite the addition of approximately 27,000 new votes. In contrast, vote rigging significantly boosts m, 's ranking. Using the omnipresent strategy, we achieve a 15-rank improvement while being more efficient compared the target-only strategy.
models in downstream applications (Ouyang et al., 2022). To meet this need, Chatbot Arena (Zheng et al., 2023a; Chi- ang et al., 2024) provides an open platform for conducting pairwise battles between LLMs, where users vote for their preferred response from randomly selected anonymous models. These votes used to compute Elo ratings for LLMs, with higher rankings on Chatbot Arena's 's leaderboard offering substantial promotional benefits.
Chatbot Arena is widely popular, but it relies on millions of user votes collected in the wild, which can be noisy and bi- ased. Several strategies have been implemented to enhance the leaderboard's reliability and reduce potential gameabil- ity, including controlling for output length and style (Dubois et al., 2024; Li et al., 2024a), detecting anomalous voting patterns and bot activity (Chiang et al., 2024), categorizing prompts for data curation (Li et al., 2024b;c), and invalidat- ing votes if anonymous model identities are revealed the responses (Chiang al., 2024).
1. Introduction
A variety of large language models (LLMs), both closed- source and open-source (OpenAI, 2024; Dubey et al., 2024), now available to the community. Evaluating their align- ment with human preferences is crucial for selecting suitable
'Equal contribution. The project was done during Rui Min's internship at Sea AI Lab. SSea AI Lab 'Hong Kong University of Science and Technology "Pennsylvania State University. Cor- respondence to: Tianyu Pang <tianyupang (ww.acccam...... Minhao Cheng <mmc7 749e posecssuudu....
Although these strategies have significantly reduced (mostly unintentional) voting biases and noise, this paper demon- strates that crowdsourced votes in Chatbot Arena can still be maliciously rigged to manipulate the ranking of a tar- get model m, either improving or decreasing it. We first introduce a straightforward target-only rigging strategy
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Improving Your Model Ranking on Chatbot Arena by Vote Rigging
that focuses solely on new battles involving m, identifying it via watermarking (Zhao et al., 2024) or binary classi- fier (Huang et al., 2025), and exclusively voting for m, wins. However, this strategy is practically inefficient because there are over 190 models on Chatbot Arena and on average only about 1% of new battles will involve my. Improving a sin- gle ranking position for a target model may require more than 10,000 rigged votes or interactions on Chatbot Arena, scenario that could be effectively mitigated by imposing daily voting limits per IP address (Chiang et al., 2024).
depends on previously collected votes. The user can query both sampled models ma and my with any prompt string E S, where & denotes the natural language space, and cast vote for their preferred response between ma and (*))). Then the vote set V will be updated according to the selected voting option:
i. a wins: Va>b = VU fe, - -,), V + Vasbi
ii. b wins: Vosb = VU fe, - ca), V t Vasbi
iii. Tie: Va=b = (e - Ufo ca), V + Va=bi
iv. Abstain: V is unchanged,
To solve this inefficiency, we observe that the Elo rating system in Chatbot Arena calculates Bradley-Terry (BT) scores (Bradley& & Terry. 1952) by fitting pairwise logistic relationships on all collected votes (Fq. (1)). This implies that when a sufficient number of votes have been collected, all models' BT scores become mutually connected through these pairwise logistic relationships. Consequently, any new vote for a battle can influence the ranking of target model me, even if me is not directly involved in the battle. Based on this observation, we propose umnipresent rigging strate- gies, which first de-anonymize all models by a multi-class classifier and actively manipulate every new vote, regardless of whether m, is involved in the battle.
where C₁ E IR K is the k-th basis unit vector and we slightly abuse the notation of J to denote the appending operation.'
Calculation of rating scores. Chatbot Arena applies the Elo rating system benchmark models. According to Chiang et al. (2024), Chatbot Arena initially used online Elo scores to calculate model ratings, but later switched to Bradley- Terry (BT) scores (Bradley & Terry, 1952) for better sta- tistical estimation. Given a collected vote set V, we can calculate BT scores for the K models on the Icaderboard, denoted in a vectorized form as BBT R.K, where r [x] is the BT score of the k-th model. The BT scores are derived from fitting the logistic relationships on V, formulated as
To prevent contaminating the actual voting records on the Chatbot Arena platform, we establish a reproducible vot- ing environment using the publicly available historical votes from Chatbot Arena Notebook. This dataset contains around 1.7 million voting records across 129 models. In the ex- periments, we thoroughly examine voting scenarios under various threat models, and empirical results show that our omnipresent rigging strategies can improve model rankings by manipulating only hundreds of new votes. These om- nipresent strategies are far more efficient than target-only strategies and other baselines, as illustrated in Figure 1. Additionally, we evaluate several defense mechanisms; how- ever, our findings underscore the need for ongoing efforts to develop stronger protections against vote rigging.
BT = arg min E (BBeee/(CCBEV))) rV
where Lacr(v, r) = - log is the binary cross- entropy (BCE) loss, and a()) is the Sigmoid function.
2.2. Threat Model
Throughout this paper, our adversarial rigging goal is to promote the ranking of a target model my on Chatbot Arena through vote rigging. This is achieved by submitting new votes, where each voting option is strategically selected to promote the target model's 's ranking.
Based on the adversary's accessibility, we pinpoint the key elements in our threat model as described below:
2. Preliminaries
Historical votes (V or v ): whether the adversary
has access to the historical voting data V or can only access to the BT scores BT
We first formalize the basic operations of Chatbot Arena in Section 2.1, including the mechanism for collecting pairwise uuman-annotated votes and calculating rating scores. Next, in Section 2.2, we introduce various threat models of vote rigging based on the adversary's 's accessibility.
V from the public leaderboard;
Model identities (Real-nanc or Anonymous): whether the adversary can directly access the identities of the sampled models m, and my in each new battle;
Sampling distribution (Py or Unknown): whether the adversary can know the sampling distribution Py or not;
2.1. Chatbot Arena
The Chatbot Arena leaderboard comprises K models, de- noted as (m1...., m&} }. with their rating scores calculated on a collection of user votes V. To collect new vote, pair of model indices and b is sampled from the joint distribu- tion Py, where the subscript V indicates that the distribution
Other users' votes (0 or Vo): when the adversary sub- mits (malicious) new votes, other users may also submit new votes simultancously, denoted as Vo.
II our notation, we use my and its index & interchangeably refer to the k-th model without causing ambiguity.
2

---

Improving Your Model Ranking on Chatbot Arena by Vote Rigging
For example, when the adversary aims to manipulate the real-world Chathot Arena platform, the threat model can be written as {r Anonymous, Unknown, Vo}.
define the vote manipulation function M
a wins if L MM = b wins if t, Passive otherwise,
Remark. We initially contacted Chatbot Arena and dis- closed the potential threat in September 2024. In our ex- periments, to avoid contaminating the actual voting records on the Chatbot Arena platform, we set up a reproducible voting environment using the latest historical votes (as of January 2025) that are publicly available in the Chatbot Arena Notebook. This dataset contains around 1.7 million voting records across 129 models. Within this environment, we divide 90% of the complete historical vote records as V h and the remainder as Vo throughout all simulations.
(2)
where the Passive option can be set to Tie (T-Tie), Abstain n (T-Abstain), random selection (T-Random), or aligned with the normal user voting distribution (T-Normal), In our following experiments, we treat these target-only rigging strategies as our baselines.
3.2. Omnipresent Rigging
3. Vote Rigging Strategies
While target-only rigging strategies are straightforward, they are inefficient in practice, as they manipulate only the new votes predicted to involve my For example, with over 190 models on the Chatbot Arena platform and a uniform model sampling distribution, the probability of a specific target model being involved in a battle is only about 1%. Conse- quently, target-only rigging strategies may passively select the voting options for approximately 99% of new battles. As reported in Huang et al. (2025), improving a single ranking position for a target model (e.g., from rank 129 to 128 or rank 5 to 4) requires over 10,000 votes for low-ranked mod- els and more than 20,000 votes for high-ranked models. ?
In this section, we discuss various vote-rigging strategies aimed at promoting the ranking of the target model m, Gen- erally, under a given threat model (where model identities are Anonymous), a rigging strategy manipulates new votes and consists of two key components:
The de-anonymizing function A(s, (s)) = To takes the user prompt 8 and the model response my (s) as inputs, aiming to de- ne-annnymize the true identity of my (or its index k)) through the predicted identity my (or the index k). This function is typically trained or designed to maximize the probability P(k- = k);
To enhance rigging efficiency, draw inspiration from the following observation on Chatbot Arena's 's rating mechanism (an informal proof is provided in Appendix A.I):
For each new vote between the sampled models ma and my. the vote manipulation function M(a, b) takes the identities ā and à predicted by A as inputs and returns one of four voting options: ā/a wins, b/b wins, Tie, or Abstain. Note that M may also depend on additional information, such as historical votes or ranks, as described in our omnipresent rigging strategy.
Observation (omni-property):: when the BT scores r are calculated on sufficient number of votes in V (by Eq. (1)), any new vote on a battle between m. and m& can influecce the ranking of the target model m4, even if my is not directly involved in the battle (i.e., t & {a, b)).
Based on this, we propose omnipresent rigging strategies, which actively manipulate every new vote, regardless of whether my is involved in the battle. We implement the de de-anonymizing function Aomni (8, mx E {l...., K} as a multi-class classifier (detailed in Appendix B). For the vote on each new battle, Aogni predicts the identities of the sampled models as ă and b. The design of the vote ma- nipulation function Mommi then depends on the adversary's accessibility to historical votes, as described below.
In the following, we elaborate on a vanilla target-only rig- ging strategy and our proposed omnipresent rigging strategy.
3.1. Target-Only Rigging
To promote the ranking of the target model m,, a straightfor- ward approach is to rig votes only for new battles predicted to involve m, (specifically, when t E {ā. b)). In this case, the de-anonymizing function focuses exclusively on identi- fying m,, formulated as Aponly (s, ms)) E {t,-t) -f), where - represents all other model indices.
BT-based omni rigging (Omni-BT). When the adversary has direct access to the historical voting data WH, it can combine its manipulated votes VMM to form V = VH U VM. For new battle between m. and myg the Omni-BT
Two concurrent works (Zhao et al., 2024; Huang et al., 2025) have explored similar target-only rigging strategies. These works implement the de-anonyyi function A,only using either watermarking/attriuutte techniques or a binary classifier. Based on the implemented Au they further
"Our measure of "votes" corresponds to the "interactions" de- fined in Huang et al. (2025), where they "votes" count only the battles predicted to involve m,.
3

---

Improving Your Model Ranking on Chatbot Arena by Vote Rigging
manipulation function can be expressed compactly as:
In our experiments, we perform additional ablation studies alternative design choices for R in Appendix A.3. After each vote manipulation, we can optionally update BT
M ommi = arg max R.BT( (r
3
the local BT scores as v [a] + r-"2"",, (9) and v [6] + r (7, (t). However, our empirical results in Appendix A.4 show that it would be better to keep r unchanged during each Omni- On manipulation.
where V' E acbb Vasb V T. } represents the four vot-
ing options: ā/a wins, b/b wins, Tie, and Abst a iin, as introduced in Section 2.1. Here, (.) denotes the rig- ging objective of Omni-BT. Throughout our experiments, we adopt the relative rating increase between m, and my that ranks one position ahead of it with R8 (rt)) r [t] ( as our rigging objective. We identify it through abla- tion studies and defer detailed comparisons in Appendix A.2. Note that since the adversary selects voting options based on the predicted ä and 6, these predictions may deviate from the ground truth updates (aas BT Vacc:
4. Sanity Check with Idealized Rigging
We start by rigging against the idealized scenario using the threat model (V Real-name, Py. 07. Results from this sanity check indicate our optimal rigging performance and serve as an upper bound for the capability analysis in Section 5. Without specific assumptions on the sampling distribution Py, we use uniform sampling with the marginal probability of a sampling my being Ps = z Addition- ally. to understand how effective vote rigging performs, we include the w/o rigging case in which votes are sampled using the normal user voting distribution comparisons. We report our initial results by rigging 20,000 new votes and defer results with larger numbers of votes to Appendix C.I. We demonstrate the ranking changes of diverse target mod- cls m, including Llama-2-13B- 1-B-Chatt (Touvron et al., 2023), Mistral- 1rnr-----nsttu--O... (Jiang et al., 2023), Qwen1.5. 14B- Chat (Bai et al., 2023), Vicuna-7B (Chiang et al., 2023b), Gemma-2-99 2-9BBit (Gemma et al., 2024b), and Phi-3-small-8k Instruct (Abdin et al., 2024) and defer rigging results with 22 extra models (used in Huang et al. (2025)) to Appendix C.2.
BBT BTT) Va=s>
Online-based omni rigging (Omni-On). When the adver- sary has access only to the up-to-date BT scores r. from the public leaderboard and not to V directly optimizing Eq. (3) in Omni-BY Mmnni-BTuue becomes intractable. To address this, we propose using online Elo scores (Elo, 1967) to approximate updates to the BT scores. This approach relies exclusively BT on y eliminating the need for access to Vn- Formally, after a new battle between m, and mp, the online Elo scores for ma and my are calculated as
AT (D)); r. (7: p)= V [a] a- (7- W [a), V On (7: ()) = v + (1 71-7-WP W BT [6], r [a])).
where is the step size, W(r v) = (1 + 10(0~=)/400) is a logistic function, and the base 10 and scaling factor 400 are adopted following Zheng et al. (2023b). The parameter y depends on the voting option: 7 = 1 for a wins, 7 = 0 for 6 wins, and 7 = 0.5 for Tie. When selecting the Abstain option, there is μ 0.
As shown in Figure 2, all rigging strategies effectively im- prove m, 's ranking compared to the scenarios without rig- ging. achieving an average of 6-rank improvement. Besides, our omnipresent strategies demonstrate significantly higher rigging efficiency against target-only strategies. For exam- ple, when rigging 20,000 new votes, the target-only rigging achieves only an average increase of 4, whereas both om- nipresent rigging strategies notably outperform it, resulting in an approximately 10-rank promotion.
Then the Omni-On manipulation function selects the voting option for the battle between m. and my as:
M
Un = arg max R (I [t], ( p), To pmni
On (T,14). (4)
7.4
5. On Exploring the Rigging Capability
where Y, Pl are constrained the values corresponding to the four voting options described ahove, and (O)(.) represents the rigging objective of Omni-On. A simple design for (O"(-) is a differentiable surrogate of the ranking function. Specifically, the ranking of m; is calculated as Rank(m;) =
However, practical vote rigging is typically conducted with limited adversary's 's accessibility, potentially reducing the manipulation effectiveness. Here, we conduct a series of stress tests to explore whether our strategies remain effective against these more demanding rigging scenarios. Specif- ically. in Section 5.1 we conduct rigging under the threat model (Vnr Anonymous, Pv, 0} to explore the impact of inaccurate de-anonymization with predicted probability P(k k) < 1, then in Section 5.2 we conduct rigging un- der the threat model {VH Real-nand Unknown, 0) to simulate the influence of Unknown sampling distribution, and finally in Section 5.3, we conduct rigging under the
n [k] > Vi [t]] where T[()) is the indicator function. This ranking function can be reformulated as Rank(m;) 1 + Exkes BT [t). v [k)) < 0.5)].
1+ ++E Everee [T
from which we can define Roa in terms of W. capturing the pairwise win rates. Consequently, Mon
ommi can be defined as
MOn arg max W (r BT 7.f
(1 [1), (7.P))
+ W (T [t) On (7,11))
4

---

Improving Your Model Ranking Chatbot Arena by Vote Rigging
Target Model: llama 2 13b chat
Target Model: mistral 7b instruct v0.2
Target Model: qwen1.5 1.5 14b chat
wi Rigging
wio Rigsing
wio Rigging
83
T-Abuliin
82-
T-Abstrin
63
T-Abstain
Model Ranking
T-Radom
T-Random
T-Random
TTee
I-Tie
T-Tle
86
T-Normal
84
T-Norpal
65
T-Normal
Omni-RT
85
Geya-RT
66
Ommi-RT
89
Omni
Pimni-
Omii--On
87
68
92
4000
8000
12000
16000
20000
4000
8000
12000
16000
20000
4000
8000
12000
16000
20000
Target Model: vicuna-7b
Target Model: gemma-7b-it
Target Model: phi----mnni-----mmtt--------
95
93
75
wis Rigging
wio Rigging
wia Rigging
Model Ranking
99
T-Abotain
95
T-Abstain
78-
T-Abstain
T-Randout
T-Random
T-Raudom
97
T-Tic
T-Tie
T-Tie
104
TNooraa
99
82
T-Normal
Omni BT
Origo BY
Omni BT
108
Ompll-in
102
CrnelOn
85
Omdii-On
113
89
0
4000
8000
12000 16000
20000
4000
8000
1 12000 16000
20000
4000
8000
12000
16000 20000
# of New Votes
# of New Votes
# New Votes
Figure 2. Ranking improvements under the idcalized rigging scenario with different target models my. Our omnipresent rigging strategies (Omni-BT and Omni-On) result in approximately double the ranking promotions compared to target-only strategies.
threat model {V Real-nallleeeee Pv, Vol to incorporate the influence of concurrent user voting.
no my will be sampled for new battles. As shown in Ta- ble 2, decreasing P; significantly reduces the effectiveness of target-only rigging. with most strategies failing com- pletely = 0.3. In contrast, omnipresent strategies show effective manipulation performance with over 5-rank im- provement even when my is not directly involved in battles.
5.1. Rigging with Inaccurate De-Anonymization
Since our rigging strategies rely on my predicted by the de-anonymizing function A(-) to select voting options, its predicted probability P(k = k:) thus directly impacts the rigging effectiveness. To examine whether vote rig- ging remains effective against inaccurate de-anonymization, i.e. P(k k) < 1, we set proportion of battles to Anonymous model identities. As shown in Table all rigging strategies exhibit decreased ranking promotion expected, with both T-Tie and T-Random achieving no ma- nipulation effect when half of the battles have Anonymous model identities. Besides, we observe that the Omni-On ex- hibits more resistance against inaccurate de-anonymization than other strategies. These results can be attributed to the usage of initial ry for Omni-On without updating with manipulated votes V MM, which makes it more resistant to the impact of previously misclassified model identities.
5.3. Rigging with Concurrent User Voting
In addition to manipulated votes V.M concurrent votes Vo from other users remain unknown to the adversary, which could affect the rigging effectiveness. For instance, they would lead to an inaccurate calculation of omnipresent rig- ging objectives RBT and (-),., thereby impacting the subsequent vote selection of their respective manipulation functions MBT (-) and Man (-). To incorporate the influ- ence of Vo. we use the combined votes V = V VHUVMUN to calculate the final rating. Our results in Table 3 demon- strate that the influence of Vo remains minor, which only introduces an average 1-rank decrease even with Vo con- taining 100,000 votes. These findings suggest the resilience of vote rigging against concurrent user voting.
5.2. Rigging with Unknown Sampling Distribution
Practical sampling distributions could be Unknown to users, for example, newly released models might acquire a higher sampling probability to collect enough votes (Zhao et al., 2024). As a result, these non-uniform sampling strategies might potentially reduce Pt, i.e., the marginal probability of sampling m4, thereby decreasing the number of sampled battles containing m, In this section, we sample new bat- tles using P. = 3. 2. where 3 E [0, 1] controls the degree of probability reduction. When 3 = 0, it indicates that
6. Case Study: Rigging Chatbot Arena
6.1. Towards Simulating Real-world Vote Rigging
To demonstrate how improve target model m,'s rank- ing in the realistic leaderboard, we simulate vote rigging against the practical scenario with the threat model being [ Anonymous, Unknown, VoJ. Through this case study, our preliminary findings would serve as a proof-of- concept that exposes the real-world rigging risks within the
5

---

Improving Your Model Ranking on Chatbot Arena by Vote Rigging
Table 1. Results of rigging performance against different propor-
Table 3. Rigging results against various scales of Yo.
tion of Anonymous battles.
Ranking, (Ranking Increaset)
Method
Rankingt (Ranking Increaset)
2 10
4 x 10
6 10*
8 10*
10
Method
10%
20%
30%
40%
50%
T-Tie
90 (+2)
90 (+2)
90 (+2)
90 (+2)
90 (+2)
T-Abstain
87 (+:)
87 (+5)
87 (4:5)
87 (+5)
88 (+4)
T-Tie
(+2) 91 (+1) 91 (+1) 92 (+0) 92 (+0)
T-Random
89 (+3)
89 (+3)
90 (+2)
90 (+2)
90 (+2)
T-Abstain 87 (+:5) 88 (+4) 89 (+3) 89 (+3) 90 (+2)
T-Normal
87 (+5)
87 (+5)
87 (+5)
87 (+5)
88 (+4)
T-Random
90 (+2) 90 (+2) 90 (+2) 91 (+1) 92 (+0)
Omni-BT
82 (+10)
82 (+10)
82 (+10)
82 (+10)
82 (+10)
T-Normal 87 (+:5) 88 (14) 88 (+4) 88 (+4) 89 (+3)
Omni-On
83 (+9)
83 (+9)
83 (+9)
83 (+9)
84 (+8)
Omni-BT 84 (+8) 84 (+8) 86 (+6) 87 (+5) 88 (+4)
Omni-On 84 (+8) 84 (+8) 85 (+7) 86 (+6) 86 (+6)
Table 4. Rigging with prompts from the Quora (Q) and HC3 (H)
datasets. We denote Target-Only" as the idealized T-Abstain.
Table 2. Vote rigging under various marginal probability Pt
8- R. When it indicates that no m, will be sampled.
Ranking! (Ranking Increase
Method
Method
Ranking| (Ranking Increase?)
Llama
Mistral
Qwen
Vicuna
8 = 0.0 8 = 0.3 8 = 0.5 B = 0.7 | = 0.9
w/o Rigging
15 (+1)
13 (+0)
12 (-1)
21 (+0)
T-Tie
95 (-3)
94 (-2)
93 (-1)
92 (+0)
90 (+2)
Target-Only"
(+1)
11 (+2)
10 (+1)
18 (+3)
T-Abstain
92 (+0)
91 (+1)
89 (+3)
89 (+3)
87 (+5)
T-Random
95 (-3)
92 (+0)
92 (+0)
91 (+1)
(+2)
Omni-BT -T (H)
11 (+5)
(+4)
8 (+3)
12 (+9)
T-Normal
92 (+0)
90 (+2)
89 (+3)
88 (+4)
87 (+5)
Omni-On On (H)
14 (+2)
(43)
9 (+2)
17 (+4)
Om
87 (+5)
86 (+6)
84 (+8)
84 (+8)
83 (+9)
Omni-BT BT (Q)
10 (+6)
9 (+4)
9 (+2)
12 (29)
Omni-On
86 (+6)
85 (+7)
84 (+8)
84 (48)
83 (49)
Omni On (Q)
14 (+2)
(+3)
10 (+1)
17 (+4)
Chatbot Arena. Specificallyy, we extract 25 models with around 23,000 English-specific votes from the complete historical records to set up the simulation environment. We present ranking improvements of the target model m,, which is set to be one of the four models including Llama-2-13B- Chat, Mistral-7B- -Band------nttut-O2222221
compared to the optimal performance of T-Abstain rigging.
6.2. Ablation Studies on Omnipresent Rigging
What if unseen prompts are employed for vote rigging? While our initial results use training prompts for rigging, reusing these limited prompts could be easily detected by the quality control in Chatbot Arena such as the simple prompt-deduupliation strategy (Chiang et al., 2024). As a result, we aim to investigate whether unseen prompts effective for rigging, especially without classifier retrain- ing. Our results in Figure 3 (a) show that both omnipresent strategies still effective ranking improvement even rigging with unseen prompts. These preliminary findings indicate the potential scalability of using the multi-class classifier for de- de-anonyyizing rather than directly relying upon the memorization of trained responses.
Qwenl.5-14B--Ci -Chat, and Vicuna- 7B. Details on the overall model selection can be found in Appendix B.2.
Setup for target-only rigging. We employ the idealized rig- ging with P(t t) = 1 since the de-anonymizing function Avonty( (-) in Zhao et al. (2024); Huang et al. (2025) typically achieves a high prediction performance. Besides, we adopt T-Abstain as it delivers a more stable ranking improvement.
Setup for omnipresent rigging. We fine-tune RoBERTa- based classifiers (Liu et al., 2019) to classify all 25 models with two datasets respectively, including the HC3 and Quora datasets. We generate the training corpus by querying each model with 4,000 training prompts. The fine-tuning process includes 20 epochs with a batch size of 64 which takes a few hours on 2x NVIDIA A100 GPUs When simulating the vote rigging, we reuse the training prompts to query model pairs within each sampled battle. We defer more details of the training corpus to Appendix B.3.
Explore the effectiveness of omni rigging with unrecog- nized models. Since the Chatbot Arena may constantly introduce new models to the leaderboard, which become unrecognized by our trained classifier. To investigate how these models will affect the rigging performance, we include 5 additional models (described in Appendix B.2) that are outside the classification range and conduct rigging within these 30 models. Figure 3 (b) shows that both omnipresent strategies outperform target-only rigging, demonstrating a degree of resilience against unrecognized models.
Vote rigging can be effective in practice. Table 4 demon- strates that both of our omnipresent strategies outperform the target-only strategy. Our Omni-BT and Omni-On strate- gies achieve an average of 5-rank and 3-rank promotion, respectively, yielding more than 50% ranking improvement
Rigging the length-control leaderboard. rongghhbconttoeeearbbbbrrr In addition to the original leaderboard, the Chatbot Arena offers the length-
6

---

Improving Your Model Ranking Chatbot Arena by Vote Rigging
Table Rigging results against the length-control leaderboard.
Table 6. The proportion of correctly classified responses obtained from the realistic Chatbot Arena platform (side by side).
Ranking! (Ranking Increaset)
Method
Llama3
GPT-40
Gemma
Liama
Mistral
Qwen
Vicuna
Dataset
Train
Unseen
Train
Unseen
Train
Unseen
Omni-BT (H)
13 (+3)
11 (+2)
9 (+2)
8 (+13)
HC3 Quora
30/30
30/30
30/30
30/30
25/30
22/30
Ommi-On (H)
15 (+1)
10 (+3)
9 (+2)
15 (+6)
25/30
25/30
28/30
27/30
29/30
27/30
Omni-BT (Q)
13 (+3)
II (+2)
9 (+2)
(+11)
Omni-On (Q)
14 (+2)
11 (+2)
9 (+2)
10 (+11)
Q-10
Detection Accurmcy (%)
100
Raaddn
Rizging Unseen Prompts
Rigping w/ Uurecognized Models
Ranking Improvennt
Q=25
80-
[NNamaal
Rankking Impro improvement
Ommi-177
Rankcng Improvemen
Ommi-AT
n= 100 Inf
60
Ozzzi
Omni--n
Cumi-On
Orssi BT (Njenal)
12
4:00
5010
12925
1900
10'
10
10'
# of New Votes
of New Votes
HC3
Quora
HC3
Quoma
(b)
(b)
Figure 3. The left figure depicts the rigging results simulated with unscen prompts, while the right figure shows rigging under the impact of unrecognized models. The dashed lines represent the performance of idealized T-Abstain rigging.
Figure 4. We present two strategies to detect anomalous voting behavior. The left figure detects and prevents users from submitting duplicate votes while the right figure identifies malicious voting behavior that deviates from the normal user voting distribution.
control version, which explicitly disentangles the effect of response length in rating calculation. Here we aim to in- vestigate whether our vote rigging still effective when conducted on the length-control leaderboard. As illustrated in Table 5, our omnipresent rigging still maintains ef- fective rigging effect achieving an average of 4-rank im- provement. Furthermore, we observe an intriguing phe- nomenon in which Vicuna-7B's 's ranking is significantly boosted compared to rigging the original lcaderboard. While our initial strategies are not specifically designed achieve this, our findings highlight another vulnerability within the length-control mechanism, where adversaries could opti- mize prompts to reduce length discrepancies between re- sponses, thereby increasing the importance of rigged votes in updating the length-control leaderboard.
in practice. For evaluation, we generate 1,000 responses for each model using unseen prompts and report their av- erage accuracy. Results in Table 7 show that the classifier maintains comparable performance even when the training dataset is reduced to half of its original size.
7. Defense against Vote Rigging
To mitigate the risks of ranking manipulation, we discuss several methods to defend against vote rigging, including detecting malicious users and filtering anomalous votes.
Detect users with duplicate vote submission. Given that both the T-Abstain/T- Tie rigging strategies consistently vote for the Passive ve options, a straightfor- ward defense involves detecting and preventing such du- plicate voting behavior. For instance, if a continuous voting duplication detected over r) battles, the user will be sus- pended from voting for period (e.g., we discard the follow- ing 200 new votes from the violating user in our demo). A larger 1) implies weaker detection but less impact on normal voting, with 1) = inf indicating no detection of duplicate votes. Our defense results against T-Abstain in Figure (a) show that our simple mechanism can effectively eliminate the ranking increase by 80%, even with a large 7 = 100.
De-anonymizing responses from the Chatbot Arena. While previous simulations use locally generated responses for de-anonymizing, it remains unclear whether our classi- fier performs effectively on realistic responses from Chatbot Arena. To investigate it, we use the first 30 training and unseen prompts to generate responses from three models including Llama-3--BB-Instuue (Dubey et al., 2024), GPT- 4o-mini-2024-07-18 (Achiam et al., 2023), and Gemma- 2-9B-it (Gemma et al., 2024b) through Chatbot Arena's ss APIs. We provide response examples from the HC3 in Ap- pendix D. As shown in Table 6, our classifier still effectively distinguishes these responses, highlighting the practical ef- fectiveness of our maassffierbaseed de-anonymizing function.
Identify malicious users. While detecting duplicate votes is effective against T-Abstain/T-Tic practical adversaries may submit random Passive options (T- (T-Random) to sim- ply bypass the detection. To overcome the challenge, Chi- ang et al. (2024); Huang et al. (2025) have discussed an identification mechanism, which detects anomalous voting behavior that deviates from the normal user voting distri-
Efficiency analysis of classifier training. this section, we leverage various scales of training corpus for classifier training to demonstrate the efficiency of classifier training
7

---

Improving Your Model Ranking on Chatbot Arena by Vote Rigging
Table 7. The Top-1 and Top-5 accuracy against various scales of the training corpus.
Table 8. The mitigation results of the vote filtering strategy.
Ranking (Ranking Increaset)
Training Prompts per Model
Method
Accuracy
0.7
0.75
-= 0.8
0.85
7 = 0.9
2000
2500
3000
3500
4000
T-Tie
91 (+1)
90 (+2)
89 (+3)
89 (+3)
89 (+3)
T-Abstain
89 (+3)
88 (+4)
87 (+5)
87 (+5)
87 (+5)
Top-1
76.92%
78.34%
79.09%
79.96%
79.23%
T-Random
(+2)
(+3)
89 (+3)
89 (+3)
89 (43)
Top-5
96.34%
97.08%
97.28% 97.72%
97.32%
T-Normal
88 (+4)
(+4)
87 (+5)
87 (+5)
87 (+5)
Omni-BT
86 (+6)
85 (+7)
84 (48)
83 (+9)
82 (+10)
Omni-On
85 (+7)
84 (+8)
83 (49)
83 (49)
83 (49)
bution. Here we follow the implementation of Huang et al. (2025), which leverages a likelihood test against the null hypothesis where votes are from normal users. Results in Figure 4 (b) demonstrate its effectiveness in detecting the T-Random and Omni-BT rigging. However, as suggested by Huang et al. (2025), the adversary could bypass it by cast- ing normal votes with the public ranking (T-Normal). For Omni-BT, when casting around 20% of normal votes, we successfully reduce the detection accuracy to 20% despite less than 15% decrease in ranking promotion. Additionally, our original Omni-On is challenging to detect without adap- tion, indicating its stealthier rigging behavior in practice.
with concerns such dataset contamination (Yang et al., 2023; Sainz et al., 2023) and Out-Of- Dff-iistribution robust- ness (Yuan et al., 2023). To address these challenges, recent progresses (Zheng et al., 2023a; Li et al., 2023; Dubois et al., 2024) employ LLM-as-a-Judge where a strong lan- guage model such as GPT-4 (Achiam et al., 2023) serves as referee for model assessment. While reducing the need for human annotation, these automatic evaluators might suffer from spurious features, such as verbosity and posi- tion bias (Dubois et al., 2024; Chen et al., 2024; Liu et al., 2024). Unlike traditional benchmarks, Chatbot Arena (Chi- ang et al., 2024) devises an online platform that allows site users to vote between a pair of anonymous models based on preferred responses. By leveraging crowdsourced voting, the leaderboard aggregates high-diversity suman-annotated votes, which features Chatbot Arena the most popular and widely recognized LLM benchmark.
Vote filtering with pairwise win rates. Since practical ad- versaries may have multiple accounts to rig which reduces the effectiveness of anomaly user detections. To address this issue, we propose a simple vote filtering designed to remove anomalous votes that deviate from the historical win rate: For each collected vote, if it satisfies W(r [a], [a], r [6]) > with my wins or W( [b], 5T [a]) >> T with ma wins, we then discard it for leaderboard updating. The controls the filtering proportion and the basic intuition here is to reduce unlikely voting results calibrated by pair- wise win rates. We provide a detailed implementation in Appendix E. As shown in Table 8, while vote filtering re- duces the overall ranking improvement, it still suffers from completely eliminating the rigging effect, where our om- nipresent strategies still achieve over 6-rank improvement even with 0.7. In conclusion, our findings demonstrate the difficulty of thoroughly defending against vote rigging, implying that more effective defenses should be developed to improve Chatbot Arena's integrity.
V
Vulnerability of LLM evaluation. Previous studies (Raina ct al., 2024; Shi et al., 2024; Zheng et al., 2024) have ex- posed the vulnerability of the LLM-as-a-Judge by adver- sarially cheating the LLM evaluator. While these studies primarily concentrate on identifying vulnerabilities in auto- matic evaluation paradigms, our paper distinguishes them with focus on rigging the human- Chatbot Arena. In concurrent with our work, Zhao et al. (2024); Huang et al. (2025) leverage strategies such as watermarking and binary-classifier to identify and exclusively vote for the tar- get model my, which can be absorbed within our general target-only rigging strategy. Additionally, we provide more unified rigging framework along with an in-depth anal- ysis of rigging capability. Our proposed omnipresent rig- ging strategy significantly improves the rigging efficiency and is effective even m, is not directly involved in battles.
8. Related Work
LLM evaluation. Developing LLMs benchmarking is a crucial task for measuring their intrinsic capabilities. Con- ventional benchmarks like GLUE (Wang et al., 2018), Hu- manEval (Chen et al., 2021), MMLU (Hendrycks et al., 2020), and GSM-8K (Cobbe et al., 2021) assess LLMs in static manner, where they typically rely on predefined test cases. Although convenient, these benchmarks are dif- ficult to comprehensively capture the open-ended genera- tion capabilities (Liang et al., 2023; Peng et al., 2022) of emerging advanced models, and are typically associated
Vulnerabilities of social voting systems. Social voting has become an effective method for gathering crowdsourced opinions, but it is often challenged due to potential vul- nerabilities (Nitzan, 1985; Islam et al., 2010; Bassi, 2015), such as strategic manipulation. Gibbard (1973) analyzes the general vulnerability of preference aggregation to voting manipulation: Hartvigsen (2008) examines vulnerabilities in political elections; Peters & Veselova (2023) discusses
8

---

Improving Your Model Ranking Chatbot Arena by Vote Rigging
the threat of group manipulation; and Wang al. (2015); Feng et al. (2010) propose defense mechanisms enhance the trustworthiness of voting systems. While these studies focus on various forms of strategic manipulation, our work introduces several unique challenges when manipulating the Chatbot Arena. Specifically,, unlike previous manipu- lation settings (Bassi, 2015), where voting candidates are visible voters, the sampled models our case are ini- tially anonymous before any rigging occurs. To address this challenge, we design and implement several effective model identification strategies, denoted as A. Additionally, to improve the rigging efficiency in practice, we design two general-purpose rigging objectives RR and RO",, which significantly outperform baselines and the two most relevant prior works in terms of efficiency.
Impact Statement
Due to Chatbot Arena's widespread popularity in LLM eval- uation, it is possible that practical adversaries could exploit our rigging strategies to improve their own target models' ranking for substantial promotional benefits. These mali- cious behaviors would put other normal model developers' interests at risk, and even worse, undermine the reliability and trustworthiness of Chatbot Arena. On the other hand, while we have discussed several defense methods against vote rigging, our initial attempts at rigging defense highlight the difficulties in completely eliminating the manipulation effect. As result, we encourage the community to focus developing more robust defense mechanisms to miti- gate the rigging vulnerabilities of Chatbot Arena as well as strengthening the integrity of emerging voting-based evalu- ation systems such as Copilot Arena and WebDev Arena.
LLM identification through model responses. Identify- ing the source of LLM through model responses has been widely researched. Active identification methods such as LLM watermarking (Kirchenbauer et al., 2023; Yoo et al., 2023; Fernandez et al., 2023; Christ et al., 2024; Kirchen- bauer et al., 2024) and L.M backdoor (Shu et al., 2023; Li et al., 2024d; Hubinger et al., 2024; Yan al., 2024; Xu al., 2024; Rando & Tramèr, 2023) embed traceable information into LLM responses, facilitating further detec- tion through predefined statistical metrics. On the other hand, passive strategies (Dou et al., 2022: Guo et al., 2023; Chen et al., 2023a; Ghosal et al., 2023; Chen et al., 2023b; Verma et al., 2024) analyze hidden text style without alter- ing the generation process. For instance, Guo al. (2023)
References
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan,
A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: highly capable lan- guage model locally on your phone. arXiv preprint arXiv:2404.1422 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.082 2023.
fine-tuned RoBERTa-basel model (Liu al., 2019) to distinguish between output preference for LLM-generated responses and human-written documents.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv:2309.1660
2023.
9. Conclusion
Bassi, A. Voting systems and strategic manipulation: An experimental study. Journal of Theoretical Politics, 27 (1):58885, 85, 2015.
In this paper, we expose the vulnerability within Chatbot Arena where rankings of target model m, can be improved through simple target-only rigging strategy. However, given the large number of models on Chatbot Arena, this strategy could be practically inefficient. To tackle this, we further propose the omnipresent rigging strategy by re- designing rigging objectives with omni-property, which sig- nificantly improves the rigging efficiency and is effective even without directly rigging m,. While our study primarily presents proof-of-concey experiments, practical adversaries could simply use the multi-class classifier or more advanced de-anonymizing functions Aommi() (-) to predict model identi- ties and cast malicious new votes to boost m, 's ranking with substantial promotional benefits. In conclusion, our findings highlight the challenges of providing a faithful LLM evalu- ation with human-annotated votes. Furthermore, devising effective anti-rigging defenses would be critical in future research to preserve the integrity of not only the Chatbot Arena but also emerging voting- based evaluation systems.
Bradley, R. A. and Terry, M. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Chen, G. H., Chen, S., Liu, Z., Jiang, F,, and Wang, B. Humans or Ilms as the judge? a study on judgement biases. arXiv preprint arXiv:2402.10665 2024.
Chen, M., Tworek, J., Jun, H., Yuan, Q. Pinto, H. D.D.O. Kaplan, J.., Edwards, H.,, Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03333 2021.

---

Improving Your Model Ranking Chatbot Arena by Vote Rigging
Chen, Y., Kang, H., Zhai, V., Li, L.,, Singh, R., and Raj, B. Gpt-sentinel: Distinguishing human and chatgpt gener- ated content. arXiv preprint arXiv:2305.07969, 2023a.
Feng, Q. Sun, Y. L., Liu, L., Yang, Y., and Dai, Y. Voting systems with trust mechanisms in cyberspace: Vulnera- bilities and defenses. IEEE Transactions Knowledge and Data Engineering, 22(12):1766-1788 2010.
Chen, Y., Kang, H., Zhai, V., Li, L.., Singh, R., and Raj, B. Token prediction as implicit classification to identify IIm-generated text. arXiv preprint arXiv:2311.0872 2023b.
Fernandez, P., Chaffin, A., Tit, K., Chappelier, V., and Furon, T. Three bricks consolidate watermarks for large lan- guage models. In 2023 IEEE International Workshop on Information Forensics and Security (WIFS), pp. 1-6. IEEE, 2023.
Chiang, W.-L.,, Li, T. L.., Gonzalez, J. E.,, and Stoica, Chatbot arena: New models & elo system up- date, 2023a. URL https:/ /lts///mmsssss//// wwwpssorg/booooooo 07- 2000a--------aaaraaaaaar
Gemma, T., Mesnard, T., Hardin, C., Dadashi, R., Bhupati- raju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini re- search and technology. arXiv preprint arXiv:2403.08200 2024a.
Chiang, W.-L.., Li, Z.., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L.,, Zhuang. S., Zhuang, Y., Gonzalez, J. E.,, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, See https://vicuna. Imsys.org (accessed 14 April 2023), 2(3):6, 2023b.
Gemma, T., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, I.., Mesmard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint Ttt1:240080011188 2024b.
Chiang, W.-L.., Zheng, I.., Sheng. Y., Angelopoulos, A. N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonza- lez, J. E., et al. Chatbot arena: An open platform for evaluating Ilms by human preference. In International Conference on Machine Learning (ICML), 2024.
Ghosal, S. S., Chakraborty, S., Geiping, J., Huang, F, Manocha, D., and Bedi, A. A survey on the possibilities & impossibilities of ai-generated text detection. Transac- tions on Machine Learning Research, 2023.
Christ, M., Gunn, S., and Zamir, O. Undetectable water- marks for language models. In The Thirty Seventh Annual Conference Learning Theory, pp. 1 1125-1139. PMLR, 2024.
Gibbard, A. Manipulation of voting schemes: a general result. Econometrica: journal of the Econometric Society, pp. 587-601, 1973.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Zhang. D., Rojas, D., Feng, G., Zhao, H., et al. Chatglm: A family of large language models from glm- 130b to glm-4 all tools. arXiv preprint arXiv:2406. 12793, 2024.
Cohere. The command r model, 2024. URL https:// re. www.cooooooooooo wwto///oc//// coooss/commad-r...
Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y., Yue, J., and Wu, Y. How close is chatgpt human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597, 2023.
Dou, Y., Forbes, M., Koncel-Kedzionski, R., Smith, N. A., and Choi, Y. Is gpt-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 7250-7274, 2022.
Hartford, E. URL. conniirecomptttiioss/bbllll 1-mistral-7b.
2023. https: https://huggingg ngf.geiiinnnnn.... co/ 2.2.
2olphin-2.2.11miin
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Hartvigsen, D. The manipulation of voting systems. Journal of business ethics, 80:13-21, 2008
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D.,, and Steinhardt, Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.00 033000030000,0000000000000000000 2020.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, B. Length-comtrolled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.0444
2024.
Huang, Y., Nasr, M., Angelopoulos, A., Carlini, N., Chiang, W.-L.. Choquette- -hyuu, C. A., Ippolito, D., Jagielski, M., Lee, K., Liu, K. Z., et Exploring and mitigating
Elo, A. E. The proposed usef rating system, its development, theory, and applications. Chess life, 22(8):242-247, 1967.
10

---

Improving Your Model Ranking Chatbot Arena by Vote Rigging
adversarial manipulation of voting-based leaderboards. arXiv preprint arXiv:2501.07490 2025.
Liang, X., Tang, Z., Li, J., and Zhang. M. Open-ended long text generation via masked language modeling. In The 61 Annual Meeting Of The Association For Computa- tional Linguistics, 2023.
Hubinger, E., Denison, Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., et al. Sleeper agents: Training deceptive IIms that persist through safety training. arXiv preprint arXiv:2401.05566 2024.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M.. Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv e e-prints, pp. arXiv-1907, 2019.
Islam, J., Mohajan, H., and Moolio, Methods of voting system and manipulation of voting. 2010.
Liu, Y., Yao, Z., Min, R., Cao, Y., Hou, L., and Li, J. Rm- bench: Benchmarking reward models of language models with subtlety and style. arXiv preprinl urXiv:2410. 16184, 2024.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.,, et al. Mistral 7b. arXiv preprint arXiv:23 10.V:2330.00825. 2023.
Mosaic. Introducing mpt-7b: A new standard for open- source, commercially usable Ilms, 2023. URL https: / //www. www.w..atbbbi.. cks. com /blog http//llgg//ppppp/////p///p/ mpt-7b.
Kim, D., Park, C., Kim, S., Lee, W., Song. W., Kim, Y., Kim, H., Kim, Y., Lee, H., Kim, J., et al. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312. 1666, 2023.
Nitzan, S. The vulnerability of point-voting schemes to preference variation and strategic manipulation. Public choice, 47:349-3700 -304, 1985.
Kirchenbauer, J., Geiping. J., Wen, Y., Katz, J., Miers, I., and Goldstein, T. A watermark for large language models. In International Conference on Machine Learning, pp- 17061-17084. PMLR, 2023.
OpenAL. URL
Introducing openai ol preview, 2024. pttts:://ppeai.com cow/inde//////////oooooocc -ore
Kirchenbauer, J., Geiping, J., Wen, Y., Shu, M., Saifullah, K., Kong, K., Fernando. K.. Saha, A., Goldblum, M., and Goldstein, T. On the reliability of watermarks for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Ouyang, L.., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Li, T., Angelopoulos, A., and Chiang, W.-L. Does style matter? disentangling style and substance in chathot arena, 2024a. URL attps://mmsy: org/blog/ 2024-08-2 28- (el---syyl--ooooeeee
Peng, X., Xie, K., Alabdulkarim, A., Kayam, H., Dani, S., and Riedl, M. Guiding neural story generation with reader models. In Findings of the Association for Computational Linguistics: EMNLP 2022. pp. 7087-7111, 2022.
Li T., Chiang, W.-L., and Dunlap, L. Introduc- ing hard prompts category in chatbot arena, 2024b. URL https://Imsys http:://smsssss..//o...///o 2024-55-17-categoryy
Peters, H. and Veselova, Y. On the safety of group ma- nipulation. Social Choice and Welfare, 61(3):713-733 2023.
Li, T., Chiang, W.-L.,, Song, Y., Jain, N., Dunlap, L.., Li, D., Frick, E., and Angelopoulos, A. N. Chatbot arena categories definitions, methods, and insights, 2024c. URL htkhhttss://bloo arena-categozy/.
Raina, V., Liusie, A., and Gales, M. Is llm-as-a-judge ro- bust? investigating universal adversarial attacks on shot IIm assessment. arXiv preprint arXiv:2402.14016, 2024.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. AlpacaEval: Automatic Evaluator of Instruction-folloiinng Models, May 2023.
Rando, J. and Tramèr, F. Universal jailbreak back- doors from poisoned human feedback. arXiv preprint 1..14523 2023.
Sainz, O., Campos, J. A., Gareía-Ferrero, I., Etxaniz, J., de Lacalle, O. L., and Agirre, E. Nlp evaluation in trou- ble: On the need to measure 1lm data contamination for each benchmark. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Li, Y. Li, T., Chen, K., Zhang, J., Liu, S., Wang. W., Zhang, T., and Liu, Y. Badedit: Backdooring large language models by model editing. In The Twelfth International Conference on Learning Representations, 2024d.

---

Improving Your Model Ranking Chatbot Arena by Vote Rigging
Shi, J.. Yuan, Z., Liu, Y., Huang. Y., Zhou, P., Sun, L., and Gong, N. Z. Optimization-based prompt injection at- tack to llm-as-a-judge. arXiv preprint arXiv:2403.17711 2024.
Yan, J., Yadav, V., Li, S., Chen, L.., Tang, Z., Wang, H., Srini- vasan, V., Ren, X., and Jin, H. Backdooring instruction- tuned large language models with virtual prompt injec- tion. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6065-6086, 2024.
Shu, M., Wang, J., Zhu, C., Geiping, J., Xiao, C., and Goldstein, T. On the exploitability of instruction tuning. Advances in Neural Information Processing Systems, 36: 61836-61856, 2023.
Yang, S., Chiang, W.-L.., Zheng, L., Gonzalez, J. E., and Stoica, I. Rethinking benchmark and contamination for language models with rephrased samples. arXiv preprint arXiv:2311.04855 2023.
Teknium.
2.5teeermne-2...mmtzii 2023. URL
https: /hhggi ng saggffcces. wtt.:cniittneteeteeee
OpenHermes 2. 5 Mistral-7B.
Touvron, H., Martin, I.., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Yoo, K., Ahn, W., Jang, J., and Kwak, N. Robust multi-bit natural language watermarking through invariant features. Proceedings of the 6lst Annual Meeting of the Asso- ciation Computational Linguistics (Volume 1: Long Papers). pp. 2092 2015,, 2023.
Tunstall, L.. Beeching. E., Lambert, N., Rajani, N., Ra- sul, K., Belkada, Y., Huang, S., von Werra, I.., Fourrier, C., Habib, N., et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310. 144944440.,6 2023.
Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., et al. Yi: Open foundation models by 01. ai. arXiv preprint htl::2003.046522 2024.
Verma, V., Fleisig, E., Tomlin, N., and Klein, D. Ghost- buster: Detecting text ghostwritten by large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technologies (Volume 1: Long Papers), pp. 1702-1717, 2024.
Yuan, L., Chen, Y., Cui, G., Gao, H., Zou, F., Cheng, X., Ji, H., Liu, Z., and Sun, M. Revisiting -uito--distributio robustness in nlp: Benchmarks, analysis, and llms eval- uations. Advances in Neural Information Processing Systems, 36:58478-58507, 2023.
Zhao, W., Rush, A. M., and Goyal, T. Challenges in trust- worthy human evaluation of chatbots. arXiv preprint arXiv:2412.043633 2024.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and anal- ysis platform for natural language understanding. In International Conference on Learning Representations, 2018.
Zheng, L., Chiang, W.-L.,, Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z.,, Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances Neural Information Processing Systems, 36: 46595-46623, 2023a.
Wang, D., Muller, T., Irissappane, A. A., Zhang. J., and Liu, Y. Using information theory to improve the robustness of trust systems. In AAMAS. pp. 791-799, 2015.
Zheng, L.,, Sheng, Y., Chiang, W.-L., Zhang, H., Gonzalez, J. E., and Stoica, I. Chatbot arena: Benchmarking llms in the wild with elo ratings, 2023b. URL https:// // imsys. org/ httgg/blor bttp//ll/////// /atp://223-a2 00--0005003-----
Wang, G., Cheng. S., Zhan, X., Li, X., Song, S., and Liu, Y. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309. 12235, 2023.
Zheng, X., Pang, T., Du, C., Liu, Q. Jiang, J., and Lin, M. Cheating automatic Ilm benchmarks: Null models achieve high win rates. arXiv preprint arXiv:2410.07137, 2024.
Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C.. and Jiang, D. Wizardlm: Empowering large language models follow complex instructions. arXiv preprint a304v:2304. 1 12244, 2023.
Zhu, B., Frick, E., Wu, T., Zhu, H., Ganesan, K., Chiang, W.-L., Zhang, J., and Jiao, J. Starling- 7b: Improving help- fulness and harmlessness with rlaif. In First Conference on Language Modeling, 2024.
Xu, J., Ma, M., Wang, F., Xiao, C., and Chen, M. Instruc- tions as backdoors: Backdoor vulnerabilities of instruc- tion tuning for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp.
3111-3126, 2024.
12

---

Improving Your Model Ranking Chatbot Arena by Vote Rigging
A. Ablation Studies of Different Rigging Objectives for Omnipresent Rigging
A.1. An Informal Proof of Omni-Property
Given a collected voting set V and the target model m,, we assume, without loss of generality, that a (malicious or normal) user votes for wins in new battle between ma and my, where É fa, b). After this new vote, the voting set updated to V a>> J as described in Section 2.1.
It is directly evident by Eq. (1) that the BT scores ma and my will change, i... r [a] * r r. |a| and [6] + v.ss [6]. Then since the sampling distribution P is always non-zero all battle pairs and the collected voting set V is assumed to be sufficiently large, it is reasonable to conclude that at least one vote on the battle between my and m. or my and my is included V. Consequently, the value of rv
[t] depends on r [a] and/or r [b). and BT
Vaza [+] depends on Vass [a] and/or
T... [6]. Thus, we conclude that r [t] 전 rBT [t], indicating that new vote the battle between ma and my will influence the BT score of the target model mn!-
r
A.2. Does Improving Relative Rating Increase Better than Improving Absolute Rating Increase for Omni-BT
In this section, illustrate why choose R RRT(1 (x r [t] r [ that measures the relative rating increase between and mg, i.e., the model that ranks one position ahead of
our rigging objective. For comparison, we implement straightforward objective RBT (r-)) r [f] that directly maximizes the absolute rating increase. We reconduct experiments under the setting in Section 4 and present comparison results of their average ranking increase across all manipulated votes Figure 5. observed that by maximizing the relative rating increase, we achieve a more stable and efficient ranking promotion. In practice, the adversary may explore more effective rigging objectives, which worth discussing future studies.
Average Ranking Improvement Across all Votes
10
Omni-BT (Relative Increase)
Omni-BT (Absolute Increase)
Ranking Improvement
8
Target-Only (T-Abstain)
6
2
0
Llama
Mistral
Qwen
Vicuna
Gemma
Phi
Model
Figure 5. We show the average ranking improvement across all new votes for two rigging objectives, where Relative Increase indicates the rigging objective TH (x*#) x [t] 1ng that maximizes m, 's relative rating inerease between and my and Absolute Increase
indicates the rigging objective Br (1 []] that maximizes the absolute rating increase of m,,
A.3. What if Maximizing the Win Rate of One Model for Omni-On
Our original Omni-On strategy in Eq. (5) aims maximize the average pairwise win rates m. and my- Here, investigate intriguing question: what if we only consider maximizing the win rate of one model (either m. my)? We formulate our problem into two straightforward rigging objectives: the first involves maximizing the win rate over the model with higher ranking, with the objective as
Mn-Min
( BT [t], O000 ( 1). (1, n)) min(W (r,T, - (7: W (I [. x (7, ())))))
(6)
and the other one focuses on maximizing the win rate over the lower-ranking models, with the following objective
On-Max
BT [t) 0 (7. r
20 (7,14) max(W (T [t] r (0 (7.1)) W (r M1 rg (7: p)))
13

---

Improving Your Model Ranking Chatbot Arena by Vote Rigging
Table Rigging results against various numbers of votes.
Vote Numbers
Method
5000
10000
15000
20000
25000
30000
35000
40000
45000
50000
w/o Rigging
92 (+0)
92 (+0) 92 (+0)
92 (+0)
92 (+0)
92 (+0)
92 (+0)
92 (+0)
92 (+0)
92 (+0)
T-Tie
92 (+0) 91 (+1) 90 (+2)
90 (+2)
89 (+3)
88 (+4)
86 (+6)
83 (+9)
82 (+10)
79 (+13)
T-Sastain
91 (+1) 89 (+3) 88 (+4) 87 (+5)
86 (+6)
85 (+7)
83 (+9)
81 (+11) 80 (+12)
79 (+13)
T-Random
92 (+0) 90 (+2) 90 (+2) 89 (+3)
88 (+4)
86 (+6)
84 (+8)
83 (+9)
(0122 79 (+13)
T-Normal
91 (+1) 89 (+3) (+4) 87 (+5)
86 (+6)
86 (+6)
84 (+8)
(+10)
(+11) 80 (+12)
Omni-BT
87 (+5) 86 (+6) 84 (+8) 82 (+10)
80 (+12)
78 (+14)
75 (+17)
73 (+19) (+20)
70 (+22)
Omni-On
87 (+5) 86 (+6) 84 (+8) 82 (+10)
81 (+11)
78 (+14) 76 (+16) 74 (+18) 72 (+20) 71 (+21)
Average Ranking Improvement Across all Votes
8
Omni-On (w/o Update)
Omni-On (Update)
Ranking Improvement
6
Target-Only (T-Abstain)
2
0-
2
Llama
Mistral
Qwen
Vicuna
Gemma
Phi
Model
Figure 6. The results show that updating oer leads significantly inferior rigging performance, which explains why we do not update with online Elo scores in our Omni-On rigging strategy.
We experiment against six models used in Section 4 and observe that applying R results in an average of 7 ranking increase. However, switching to Mnn-inn yields much inferior and unstable rigging results, even lowering the average ranking by 4. These findings validate the effectiveness of using the average win rate as our objective for more stable rigging performance.
A.4. Explanation of Why We Do not Update BT when Using the Omni-On Strategy
In Figure 6, we compare the results of updating (Update) and not updating (w/o Update) the r strategy. It is observed that updating vv results in significantly inferior rigging performance, even when compared the T-Abstain. This is due to the instability of the online Elo updating which is also discussed in Chiang et al. (2023a), leading to an inaccurate calculation of pair-wise win rates and thus affecting the vote selection of Omni-On.
BT when using the Omni-On
B. Omnipresent Rigging based on Multi-Class Classifier
B.I. Mechanisms of De-Anonymizing Functions Aommii
We train classifier acquire LLM identities based on their individual responses. Formally, let fa(-) N denote the classifier parameterized by 6, where N < K indicates the number of models being classified. We construct the training dataset D by prompting each considered model m,, with set of prompts and labeling their responses with corresponding indexes (1,..., N). These labeled corpus d = (m, n) can then be utilized to optimize 0 by minimizing the Cross-Entropy (CE) Loss:
0" arg min EdED! log( exp( (m,
(s))(m) (m),))m)
-)]
(8)
exp(fo(m {))))
14

---

Improving Your Model Ranking Chatbot Arena by Vote Rigging
Table 10. Rigging results against additional target models.
Rigging Strategies
Model
w/o Rigging
T-Abstain
T-Random
T-Tie
T-Normal
Omni-BT
Omni-On
claud-335-sxomeee222440222
(+0)
(+0)
5 (+0)
(+1)
(+0)
4 (+1)
4 (+1)
claude-3-haiku-22224000
39 (+0)
38 (+1)
38 (+1)
38 (+1)
38 (+1)
38 (+1)
34 (+5)
19 (+0)
19 (40)
190)
19 (+0)
19 (90)
19 (+0)
18 (+1)
gemini-l.S-pro-api-O5
(40)
9+0)
(+22)
(42)
940)
(+2)
(+1)
gemma-2-2h-it
56 (+0)
(42)
(+1)
56 (+0)
(42)
50 (+6)
(42)
gemma-2-9b-it
33 (+0)
(+1+1)
(+1)
(-1-
(+1)
32 (+1)
330)
gemma-2-27b-it
(-1
(+1)
(+1)
202)
20 (+1)
21 (+1)
8pt-3.5-turbo-0613
63 (+0)
(42)
(+4)
59 (+4)
(42)
57 (+6)
57 (+6)
gpt-4-0125-preview
17 (+00
(6(+1)
14(+3)
143)
16(+1)
15(+2)
11 (16
8p1-4-1106-previee
12 (+0)
(40)
12
(11+1)
(+
12
5+7)
ap--4-turbo-2224-044-99
11 (+0)
(01+))
(+1)
(43)
(01+1)
7(44)
9+2)
$pt-4o-20024055-13
(+0)
(+0)
(+0)
(+0)
(
(+0)
(+1)
$p1-4o-2024-08--66
(+0)
(+2)
(+1))
(+11)
(+2)
(+3)
6+2)
8pD-4o-mimi-2024-07-188
(+0)
(40)
4 (+0)
(+0)
(+0)
4 (+0)
(+0)
(+0)
47 (40)
(-1)
(-1)
(+0)
(41)
47 (+0)
Hama-3--7b-inssie
27 (+0)
27 (+0)
26(+1)
25 (+2)
27 270)))
(+2)
21 (+6)
(40)
(00)
400)
(40)
40
40 (+0)
40 (+0)
Ilama-33 3.1-tob-inttruct
(40)
144)
133)
13 (43)
124)
12
142)
llama-3, 3-40----i--t
(40)
(+0)
(+00)
(40)
7400)
(+2)
7+0)
mixtral-888 1aarral--877-issttuttt00..
64 (+0)
(40)
(-2)
64 (+0)
(42)
62 (+2)
(+0)
Sinrral-8x22b--istruue--00.
(-1)
48 (+4)
51 (+1)
(+4)
(+4)
48 (44)
50 (+2)
gwen2-72b-instruet
(+0)
(+1)
34 (+0)
(+0)
(+1)
32 (+2)
33 (+1)
Average Ranking Improvement
-0.1
+0.9
+0.8
+1.1
+1.0
+2.0
+2.1
where fo(-) [n]] indicates its n-th logit. Then the omnipresent de-anonymizing mechanism can be formulated as Aomni my (s)) arg max, fe- [s)]
B.2. Details Model Selection in the Case Study
Here, we elaborate on the models utilized in our case study in Section 6. For classifier training, we use total of 25 models, including 1imm--3-B-Inntruus (Dubey al., 2024), Llama-2-7B- 2B--at,, Llama-2-13B-Chail (Touvron et al., 2023), Mistral- IB-Instu 1B-nn-stcttttttto..22222 (Jiang et al., 2023), Command-r (Cohere, 2024), Gemma-2B-i (Gemma et al., 2024a), Gemma-2-9B-ir. Gemma-2-27B-ir (Gemma al., 2024b),
Yi-34B-Chat Yi- Ff-155334-CCha (Young (Bai et al., 2023). Starling-LM-7B-aappp Starling- LM--e--M-B--bbbb (Zhu et al., 2024), Chatglm3-6B (GLM et al., 2024), Zephyr-7B-alppe
et al., 2024), Qwenl.5-7B-CII Swenl.5-14BBCIT
Zephyr-7B-beta (Tunstall et al., 2023), Openchat- 3.5 (Wang et al., 2023). Vicuna-7B 7B-v1. (Chiang et al., 2023b). Mpt- Mpt-7B-Chat Chat (Mosaic, 2023), Wizardlm- (Xu et al., 2023), Solar- 10.7B-instruct- 1.0 (Kim et al., 2023), and GPPT-4o-mini-2224-07-4 (Chen et al., 2023a). Additionally, involve five extra models simulate newly added models that fall outside our initial classification range including (Brown et al., 2020), Douma-13B-vil (Chiang al., 2023b), 2hi-3-medium-44 4k-Instruct (Abdin et al., 2024), 2olphin-2.2.11-00 1-21--Mittral-BB (Hartford, 2023), and Openhermes-2.5-- 2.5-Mistaal- (Teknium, 2023). In Figure 7, demonstrate the classification performance against responses generated with unseen prompts from the HC3 and Quora datasets respectively. We evaluate each model using 1,000 responses generated from unseen prompts.
B.3. Details of Training Dataset Construction
In our experiments, both the HC3 and Quora' datasets are sourced from the Hugging Face. For HC3, we utilize the prompts from the reddit_eliS dataset split for response generation. We use two simple data-cleansing strategies in our training
hhtpps: h//hgggi///////// htt.aa....o/tttttet//aeataea/ hhtpps: hottps://huuge httss::///gggggg/gggeeeee wtt....ooaaaatttteaeeeee quoraa compet ono ons/quora quora
mp1 eAT /HC3
15

---

Improving Your Model Ranking Chatbot Arena by Vote Rigging
corpus: discarding responses with fewer than 100 tokens, these responses show less promotion for classifier training, and removing three markdown symbols from the text responses including markdown header, markdown bold, and markdown list symbols followed by (Li et al., 2024a) to make our classifier less biased towards these potentially spurious features.
C. Additional Rigging Results
C.1. Rigging Results with Various Numbers Votes
We provide the average rigging performance against various numbers of new votes in Table Our omnipresent strategies remain effective across different numbers of votes compared target-only rigging strategies.
C.2. Rigging Results with Additional Target Models
We provide additional comparisons of rigging performance between the target-only and omnipresent rigging strategies on target model my, which set be one of the diverse models used in Huang et (2025). Our experimental results in Table 10 demonstrate the effectiveness of omnipresent rigging strategies, which achieve over double the ranking promotion compared to the target-only rigging strategies.
Algorithm 1 Vote-filtering Strategy
input Collected voting records V; Historical voting records V Threshold output Filtered voting records VF. F.
1: Initialize VF 1p=0
2: for in do
3: Extract m. and my from v
4: Calculate (r [a]. rTT [b])
5: Calculate (r
BT [b]. Vu [a]) 6: if W(r [a].
BT
AT [b) > T and my wins then
7: continue
8: else if W
BT VH
[b,
BT [a)) > t and ma wins then B
9: continue
else
11: VF Uv
12: end if
13: end for
D. Example of Different Model Responses
We provide examples of prompts from the HC3 dataset, well corresponding responses generated by querying model APIs on the Chatbot Arena (side by side). We show the model responses of Llama-3-8B-Inns Instruct in Figure 8, GPT-4o-mini- 2024-07-18 in Figure and Gemma-2-9B-it in Figure 10 respectively.
E. Details of Vote Filtering
We provide detailed description of our fote-filtering strategy Algorithm 1.
16

---

Improving Your Model Ranking Chatbot Arena by Vote Rigging
Classification Accuracy on HC3 dataset
00
4964
0,980
0.336.4
0.9233
0
Accuracy
610
0 6666
0.8489
0 12z84
769
0.33
0.8163
0,735
0.7397
0.6913
6.7329
6.6438
0.5
606
0.5956
0. 5.5466
0.5 5208
Llama-2-7b-chata
Llama-2 1ama-2-13b-chatte
Meta Meta-Llama Laa--LLaaa------ 3-88--nstrrrtt
c4ai-command-rwu0)
gpt 10---00-mi-2222 2014-ii-202----111
Mistral- Eeatra--B-innse 7---Innstuutt-00
Mistral- 71----lnnss 1B-Instruct-v0.1
gemma-2-227..t
gemma-2b.. -b-it
gemma 2-9b-it
Qwen1.5 Qwen1.5-78-Ch 5-7B-hha
Qwen1. 5-148-4666 Chat
Starling LM- MM----aa0 78-appha
Starling 7B-baa
Yi- Yi-34B-Chal
Yi-1. Yi-1.5-344 Chh-BBBB
chatgim3-6t
zephyr- zephyr-7b-ber 7b-beta
zephyr- 7b-alpha
Phi-3 3-s-all---l----- ik-isrruut
vicuna-T Drcuna-7b-v... 7b-11.3
mpt- mpt-7b-cha 7b-chat
openchat 3.5
WizardLM- M-1 M-133-11.22
SOLAR-10 1...-10.78-nnstruttte 1nsttrrrut-v11..
Model
Classification Accuracy on Quora dataset
0.6973
0,903
0. 0.89974
0. 0.9528
0 0.9236
Accuracy
0,7831
0 77.66
0, 017755
022
0. 06.669
0.7258
0.7384
0.6446
7088
0.5
0. 0.5499
0.5759
0.5209
0. 0.4063
0.4696
0. 0.3555
1.3626
(lama-2-7b-chat-ht
Llama- hama-2-13b-chata-
Meta-Llama-3-geet 3-a-a----B--nsttuc
(4ai-command-cvv0)
gpt-4o 1ilo-miiii-22----1111
Mistral- Diistrl-78-Instruucke
Mistral-72 Mistral-7B-Inste B--aalrtttnntt
gemma-2-22 2--b-it
gemma-2b-it
gemma-2-8 2-9b-it
Qwen1.5 5-7BChhat
Qwen1 5-148-4 4h-Chat
Starling- LM L.M-M---lph
Starling BaanngLLL-BBB
Yi-34B- Chat
Yi-1 5-348 Chat
chatglm3 6-
zephyr- 7p-hy--7b-alpha
Phi- Phi-3- 1il-3--mall----- -inssl--k-instrrut
vicuna-7b-ut 7b-v1.3
openchat 3.5 mpt-7b-chat
Wizard WizardL M---dLM--11-1111
SOLAR-10 78 3-Instrucct-v1.0
Pephyr-7b-ber
LM--B--bllt
Model
Figure 7. Classification accuracy of the text classifier against the HCC3 and the Quora dataset. Our evaluation results are evaluated with
responses generated with unseen prompts, which are not used for training corpus generation.
17

---

Improving Your Model Ranking Chatbot Arena by Vote Rigging
Examples from Llama-3-8B 8B--Istructt
Prompt (Train)
Why every
BLIS Khy every book 1 bear about NY Times # 1 Best seller Should n't there only be one best seller Please explain Like 2'm give.
about
Model Response
What a great quest fon!
Imagine you're at big lemonade stand with lots of different flavors. Each flavor is different book, and people buying them read.
super-important lemonada stand keeps track of how many people are buying each flavor of lemonade (bock) each week. They make a apecial 1iat called the *Beat Sellers Liat" that shows which flavors (oooka) are the popular.
New
IXYT)
thare many different categories of books, like ficsion, non-fiction, mystery, romance, and more. The NYT has separate 1ist tor each category, kind of like a different of
for each ...
Prompt (Train)
If selt is bad for why
it the roads ? the title states way do we use it 2 is there ne other option or what ? Flease explain like I'm five.
Model Response
Let me explain it to you in a way that'ss easy to understand!
Salt is used on roads for really importent reasen: it helps keep us safe by causing accidenta. Here's why:
preventing slippery
nuring the winter months, the ground can get really cold, and when does, can make the roads super slippery, This is because the water in the air freezes and turns into ice or frost. When we drive on these slippezy reads, it's like trying to drive on lose traction and slide all the place!
gient benana
That's
Nhen sprinkled roads,
important things:
Prompt (Unseen)
The appeal of NASCAR I= 's just one of those things 've never understood Explain five.
Model Response
OR BOY, LET XB TELL YOU ABOUT BASCAR!
you know how there cars, right? like your parents drive you around in
Well, NASCAR is like big where people drive those cars REALLY, REALLY FAST!
But insteed of going to the grececy store, they go around special track called "racetrack". it's like big oval-shaped playgreund!
Tmagine yru'rre playing with your toy nolaes, these caza are actually going "VR00000M!" zeally, zeally fast! Some of them can go up to
instead naking
per hour! ...
Figure 8. Examples of model responses generated by querying Llama-3-8B B-ae--3-----nteee Instruct using HC3 dataset prompts. The upper two examples the training prompts, whereas the bottom example uses the unseen prompt.
18

---

Improving Your Model Ranking Chatbot Arena by Vote Rigging
Examples from GPT-40-miin-22224----B
Prompt (Train)
why ia avery bock hear about a BY Times t Best Soller EL2S why in every book hear about Times Rest Sellar " Should n't there only best seller Please explain like 2'm five.
Model Response
Great questiva! Inagine you have a big box of Loys, and every time you play with toy, you your friends have different games and ways play with those toys. Sama days, play with blocks, other days with dolla, and aometimes with cars.
is. New, let's say you your friends
When it corea to books, the New York Times has different Hata for best sellers, just like you different games.
lists differen: types books, like:
(madeup stories)
NonFiction (real storles or information]
Children's Books
Graphic Novels
And more!
So, even if book is
Prompt (Train)
If salt is so bad for cars why do we use it on the roads ? Aa the title atates way explain like five.
we
other option or
Model Response
ice drive ovez ice, they can slip and slide, which can be very dangerous! So, to help keep safe, people sprinkle on
Okay! Tmagine it's
and there's 1ot
Here'ss s why salt used:
1. Malts Tees Salt to like magic for ics! When you sprinkle it on iea, 't makes the melt. helps
roads so drive safely.
melts, it's not slip. This meana fewer accident.s!
Mekes Reads
cars stay on road
Prompt (Unseen)
The appeal of NASCAR To ta just one of those things 'ue never understood explain
like I'a rive.
Model Response
Okay! Tmagine you auper fast around big circle, like giant track. NASCAR is like that, but with real drivers!
toy think about laz chose toy racing
People to watch because:
1. Speed: The cara go really, really fast, like a cheatah running!
Excitement: Sometimes the hump into each other spin out, which is thii to aee!
NASCAR togethez, cheering their favorite drivers, just like cheering for team in a game,
Friends Family: Many people
Cool cars: cars are pat bright colors and have fun designa, making them 100k
Figure Examples of model responses generated by querying GPT-4o-mimi-2224-077-1 using HC3 dataset prompts. The upper examples use the training prompts, whereas the bottom example uses the unseen prompt.
19

---

Improving Your Model Ranking Chatbot Arena by Vote Rigging
Examples from Gemma-2-9B-4
Prompt (Train)
Why avery book hear about meme Reat Seller RTTS Khy every book I heaz about NY Times 4 1 Beat Beller Should nft theze only be one # 1 seller
explain like five.
Model Response
different toy stores, each chart showing most popular toys that wack.
imagine
The New York Tines ls like big store that makes 1ist of the most popular books each week. They have different charts for different kinds of books, like picture books, chapter bocks, and growuup books.
So, can many bestsellers because
are many different lists!
It's like having 1 toy in the action rigure sect section, coy in the dell section, and in the building blecks section. They are sll in their category!
Prompt (Train)
If selt is bad for why do we use it ? is there ne other option or what ? Flease explain like I'm five.
it the roads ? the title states way
Model Response
Imagine you're ze playing outside on snowy day, Your shoes get all Ley and slippezy, right?
Roads get same way super slippery and dangercus for cars.
it snows and freezes, roads become
Salt is like magic powder that melta the ice! It makes the ice tuzn iate water, care can drive safely.
Tt's not perfect though. Salt can be bad for cars because to can rust them. Rut it'a little
have car because ley
Prompt (Unseen)
The sppeal of NASCAR 1: 's just one of those things 've never understood Explain
like I'm five.
Model Response
Tmagine you have bunca of toy cara, all the same, but with different colors and numbers.
Now imagine you race then around à big, oval track.
Some people like watching these races beccuse:
The cars are auper fast! They zoom azound the track really quickly, which is exciting
The drivers are really good! They have to be super careful and skilled to deive these Sest.
There's lots action! Somet bit like big, noisy gane of Lag.
sump into each ocher, which little
It'a like big party! Lots of people NASCAR races with their Frienda and ramily, and they cheer for
Figure 10. Examples of model responses generated by querying Gemma-2-9B-in using HC3 dataset prompts. The upper two examples use the training prompts, whereas the bottom example uses the unseen prompt.
20


Reviews:
[{"review_id": "3xKu22XDr9", "reviewer": "Reviewer_QcBh", "summary": "The paper presents a method to manipulate the chatbot arena leaderboard platform, demonstrating the conceptual ability to alter a target model's ranking through strategic voting. The authors propose three rigging schemes: The first is target-specific, involving voting for the target model whenever it appears in a pairwise comparison. The other two schemes exploit the scoring dynamics of the Bradley-Terry model (with and without historical voting data) to modify the target model's score by strategically voting for other candidates.\n\n## Update after rebuttal:\nConsidering the novelty and technical contribution of the paper, I have decided to keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review for recognizing our paper and invaluable suggestions. Below we respond to the comments in **Weaknesses (W)**.\n\n---\n\n***W1: (Essential References Not Discussed) Missing literature review of social voting and its vulnerabilities.***\n\nWe appreciate your valuable suggestions, which inspired us to explore earlier works on social voting systems and their vulnerabilities. We identify several relevant studies, for example, [1] analyzes the vulnerability of preference aggregation to strategic manipulation; [2, 3, 4] systematically introduce various voting systems and their susceptibility to manipulation; [5] examines vulnerabilities in political elections; [6] discusses the threat of group manipulation; and [7, 8] propose defense mechanisms to enhance the trustworthiness of voting systems. Following your suggestion, we will incorporate detailed discussions of these works in the paper revision.\n\n---\n\n***W2 (a): Concerns on method novelty; whether they are known in previous papers.***\n\nFollowing your valuable suggestions in $\\\\textrm{\\\\color{green}W1}$, we conduct a detailed survey on social voting systems and their vulnerabilities. While the referenced papers focus on various forms of strategic manipulation, our work introduces several unique challenges in the problem setting and methodological innovations. These are discussed in detail in $\\\\textrm{\\\\color{blue}Section 3}$ (page 3), and we briefly recap them as follows:\n\n- *The de-anonymizing function* $\\\\mathcal{A}$: Unlike previous manipulation settings [2], where voting candidates are visible to voters, the sampled models in our case are initially **anonymous** before any rigging occurs. To address this challenge, we design and implement several effective model identification strategies, denoted as $\\\\mathcal{A}$.\n\n- *Omnipresent manipulation function* $\\\\mathcal{M}_{\\\\textrm{omni}}$: In addition to improving the model ranking, we also emphasize **rigging efficiency** in practice. To this end, we design two general-purpose rigging objectives ($\\\\mathcal{R^{\\\\textrm{BT}}}$ and $\\\\mathcal{R^{\\\\textrm{On}}}$), which significantly outperform baselines and two most relevant prior works in terms of efficiency.\n\n---\n\n***W2 (b): Concerns on whether the rigging methods are relevant to other voting and scoring models.***\n\nOur rigging methods can be applied to other voting/scoring models, such as those using online Elo scores. To demonstrate this, we conduct simulations on a leaderboard updated with online Elo scores. Results in **Table A** show that both methods (Omni-BT and Omni-On) effectively improve the target model $m\\_t$’s ranking by rigging 1,000 new votes. Since the Chatbot Arena adopts the Bradley-Terry model for scoring, we primarily report results based on this model in the main paper to better align with the practical setting.\n\n\n**Table A: Rigging simulation on the leaderboard with online Elo scores. The results are absolute ranking (ranking increase).**\n\n||Llama-2-13B-Chat|Mistral-7B-Instruct|Qwen1.5-14B-Chat|Vicuna-7B|Gemma-2-9B-it|Phi-3-small-8k-Instruct|\n|-|-|-|-|-|-|-|\n|*Omni-BT*|88 (+4)|74 (+13)|54 (+14)|108 (+5)|87 (+15)|73 (+16)|\n|*Omni-On*|78 (+14)|65 (+22)|46 (+22)|97 (+16)|81 (+21)|67 (+22)|\n\n---\n\n***W3: Whether rigging strategies are applicable under practical defenses.***\n\nOur method is applicable under practical defenses. First, since our rigging mechanism operates by strategically selecting voting options and submitting votes just like normal users, network-level defenses (e.g., firewalls and DDoS mitigation) and application-level defenses (e.g., bot detection via Cloudflare or CAPTCHA) cannot directly detect or distinguish the rigged votes.\n\nAdditionally, practical attackers can incorporate techniques such as IP rotation, Slowloris attacks, and CAPTCHA farms into their rigging scripts to bypass many of Cloudflare’s and CAPTCHA’s traditional network- and application-level defenses. This *effectively bridges the gap between our experimental simulations and real-world vote rigging*, indicating the practical applicability of rigging methods. \n\nMore importantly, we chose not to attack the actual platform due to ethical considerations, as we do not wish to contaminate Chatbot Arena’s voting data. Instead, our simulations offer reproducible results that serve academic purposes and can support future efforts to design more robust and trustworthy voting-based LLM leaderboards.\n\n---\n\n***References:*** \\\n[1] Manipulation of Voting Schemes: a General Result \\\n[2] Voting Systems and Strategic Manipulation: an Experimental Study \\\n[3] The vulnerability of point-voting schemes to preference variation and strategic manipulation \\\n[4] Methods of voting system and manipulation of voting \\\n[5] The Manipulation of Voting Systems \\\n[6] On the safety of group manipulation \\\n[7] Using Information Theory to Improve the Robustness of Trust Systems \\\n[8] Voting Systems with Trust Mechanisms in Cyberspace: Vulnerabilities and Defenses", "claims_and_evidence": "Claims made in the submission supported by clear and convincing evidence", "methods_and_evaluation_criteria": "Proposed methods make sense for the application at hand.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "All the experiments in the main body of the paper appear valid.", "relation_to_broader_scientific_literature": "The methods proposed by the authors specifically demonstrate how the ranking of a target model can be altered on the Chatbot Arena platform. The main contribution of the paper concerns voting and manipulating voting scores. More broadly, this paper addresses the scoring of LLMs, focusing on model identification, evaluation, and relative ranking.", "essential_references_not_discussed": "The key contribution is a method to manipulate the Bradley-Terry model as a scoring system. However, there is a lack of a literature review on the vulnerabilities of scoring systems, such as their susceptibility to strategic voting, the addition of new candidates, and similar issues. Given that the Bradley-Terry model is well-established, there should be known attacks that are effective against it.", "other_strengths_and_weaknesses": "Strengths: \n1) The paper is well-written, clear, and concise.  \n2) It demonstrates a relevant weakness in a well-known system.  \n3) It bridges the problem of social voting (and its vulnerabilities) with the current need to score LLMs.\n\nWeaknesses:\n1) There is a missing literature review related to social voting and its vulnerabilities.  \n2) The novelty of the proposed rigging strategies is unclear:  \n   a. The approach is based on strategic voting, but it is unclear whether the methods are already known from previous papers discussing vulnerabilities in social voting and scoring systems (e.g., specifically for the Bradley-Terry model or other generalized models).  \n   b. If the methods are indeed novel, it is uncertain whether they are relevant to other voting and scoring models.  \n3) There are multiple defense mechanisms in place, such as Cloudflare, CAPTCHA, and user authentication (some of which are discussed, and some are not). Therefore, it is unclear whether the proposed methods can be applied in practice outside of a simulated environment.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p6NKnv5Syq", "reviewer": "Reviewer_Cw8u", "summary": "The paper investigates the problem of manipulating the ranking of a target model on the anonymous voting platform Chatbot Arena. The authors begin by examining a straightforward approach called target-only rigging, which involves attempting to influence votes only in battles where the target model is present. They demonstrate the limited effectiveness of this method due to the low frequency of such battles. To overcome this limitation, the authors propose more efficient omnipresent rigging strategies. These strategies leverage the Elo rating system used by Chatbot Arena, showing that even votes in battles not involving the target model can impact its overall ranking. The authors conducted experiments by simulating attacks to showcase the performance of their techniques and discuss potential defense mechanisms against such attacks.\n\n## updates after rebuttal\nI decided to remain the score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your very positive and supportive review. Below we respond to the comments in **Weaknesses (W)**.\n\n---\n\n***W1: Absence of actual, live experiments on the platform.***\n\nWe acknowledge that we did not conduct live experiments on the platform, primarily due to ethical considerations. Instead, we opted for practical simulations, which offer more reproducible results and can better support future research in further investigating rigging vulnerabilities. In fact, as mentioned in the *Remark* paragraph (Lines 110–111), we proactively reached out to the authors of Chatbot Arena in September 2024 to share our preliminary findings on rigging vulnerabilities. They also recommended conducting simulations based on historical data. Below, we provide detailed explanations:\n\n- **Rigging the actual platform raises ethical concerns**: As one of the most widely used LLM benchmarks nowadays, submitting rigged votes to Chatbot Arena would compromise the platform’s integrity and unfairly impact model developers whose models may be pushed down in the rankings due to manipulated results. Our objective is to expose potential vulnerabilities in Chatbot Arena and encourage the community to develop robust defenses against such attacks. To this end, we provide practical simulations that closely mirror the real platform, ensuring reproducibility without interfering with the actual leaderboard or affecting real voting data.\n\n- **Vote rigging can be conducted in practice**: While real-world platforms may employ standard defenses such as Cloudflare and CAPTCHA, practical attackers can leverage well-established techniques, such as IP rotation, Slowloris attacks, and CAPTCHA farms, within rigging scripts to bypass many of the traditional network- and application-level protections. These capabilities *effectively narrow the gap between our simulated experiments and real-world vote rigging*, indicating that our methods are applicable in practical scenarios.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Sounding.", "relation_to_broader_scientific_literature": "This paper contributes to the growing body of literature on the security and reliability of anonymous voting systems, particularly in the context of evaluating large language models (LLMs). The authors specifically address the Chatbot Arena platform, which is widely used as a benchmark for LLM performance.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper excels in providing a clear and well-defined framework for understanding the different scenarios involved in manipulating a target model's ranking. The authors effectively categorize these scenarios based on factors such as the anonymity of the models and the availability of historical data. The detailed and intuitive presentation of the concepts makes the paper easy to follow and understand.\n\nThe paper's findings are significant as they highlight a potential vulnerability in a widely used benchmark for evaluating LLMs. By demonstrating the effectiveness of even relatively small-scale vote rigging attacks, the paper raises important questions about the trustworthiness of anonymous voting platforms and underscores the need for robust defense mechanisms.\n\nWeaknesses:\n\nWhile the authors present compelling simulations based on historical data from Chatbot Arena, the absence of actual, live experiments on the platform could be considered a limitation.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i9xwYJwJnU", "reviewer": "Reviewer_btrf", "summary": "This paper investigates how Chatbot Arena can be manipulated to artificially boost the ranking of a target model. The authors first describe a target-only rigging strategy, which detects and votes exclusively for the target model whenever it appears. They then present an **omnipresent** strategy that manipulates every new vote by first identifying which two models appear in each matchup and selectively voting in a way that indirectly benefits the target model—even if it is not involved in that specific matchup. Overall, the paper underscores the vulnerability of voting-based rankings to adversarial manipulation, calls for further research on robust defenses, and highlights broader implications for any crowdsourced evaluation system.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and for recognizing our work. Below, we respond to your comments in **Concerns (C)**, **Weaknesses (W)**, and **Questions (Q)**.\n\n---\n\n***C1: No supplementary materials.***\n\nWe would like to clarify that we have provided *Supplementary Material* and the folder named *rigging_code* contains the code.\n\n---\n\n***C2: Lack of discussions of LLM-based text classification.***\n\nThanks for the helpful suggestion. We have included related works on LLM-based text classification in the paragraph *Discussions on strategies to identify LLM through model responses* (page 15). In the final revision, we will move this discussion to the main paper and expand it in detail.\n\n---\n\n***W1 & Q2: Can rigging strategies generalize to other voting platforms?***\n\nOur strategy generalizes effectively to other voting platforms. To verify this, we simulate on *WebDev Arena* and *Copilot Arena*. Since they do not provide historical voting data, we initialize the Bradley-Terry model’s coefficients with public scores and update the leaderboard with newly submitted votes. Results in **Table B** show consistent improvements by rigging 500 votes, validating the effectiveness.\n\n**Table B: Rigging simulation on other voting-based leaderboards. The results are absolute ranking (ranking increase).**\n\n||Model|T-Random|Omni-BT|Omni-On|\n|-|-|-|-|-|\n|WebDev|o1-mini|8 (+3)|1 (+10)|6 (+5)|\n||Gemini-Exp|8 (+7)|1 (+14)|8 (+7)|\n|Copilot|Gemini-1.5-Flash|3 (+7)|1 (+9)|1 (+9)|\n||Qwen2.5-Coder|7 (+5)|1 (+11)|2 (+10)|\n\n---\n\n***W2: The defense methods are somewhat rudimentary.***\n\nWe acknowledge that current defenses are relatively straightforward; however, results in $\\\\textrm{\\\\color{blue}Figure 4}$ (page 7) show strong effectiveness against several baselines, including T-Abstain, T-Tie, T-Random, and even the vanilla Omni-BT.\n\nWhile devising advanced defenses against Omni-On and improved Omni-BT is indeed challenging and remains an open problem, we systematically highlight rigging vulnerabilities in Chatbot Arena—a platform widely used and trusted by the community. By exposing these flaws, our paper can spark broader discussion and inspire future research to develop deeper and more robust defenses.\n\n---\n\n***W3: Efficiency of classifier training if new models appear frequently.***\n\nOur RoBERTa-based classifier is lightweight and efficient to train. As shown in $\\\\textrm{\\\\color{blue}Table 7}$ (page 8), training corpus generated using *2,000 prompts* (or even fewer) per model is sufficient for high accuracy on *unseen prompts* and the training cost is around 4 GPU hours on a single A100 GPU.\n\nTo further address the scalability concern, we propose a hierarchical classification design. Specifically, we can construct multiple sub-classifiers, each distinguishing among $N$ known models and an additional class labeled as *other models*. This allows us to incrementally accommodate new models without retraining the entire system. The total number of classifiers needed to cover $M$ models is $\\\\lceil M/N \\\\rceil$.\n\nWe also appreciate your insightful suggestions regarding more online approaches. Techniques such as class-incremental learning [1] can indeed sequentially update the classifier using data from newly added models while mitigating the risk of catastrophic forgetting.\n\n---\n\n***W4: The ethical concerns of releasing rigging strategies.***\n\nWe understand and appreciate your concerns regarding the ethical implications. However, our work is positioned as a red-teaming study, with the primary goal of exposing critical vulnerabilities in widely used LLM leaderboards. As noted in the *Remark* paragraph (Lines 110–111), we proactively informed the authors of Chatbot Arena about the vulnerability prior to submission.\nBy bringing these issues to light, we aim to raise awareness within the community and encourage the development of stronger defense to safeguard future evaluation platforms against such manipulation.\n\n---\n\n***Q1: How feasible is it to maintain undetectable rigging behavior?***\n\nRigging behavior is indeed difficult to detect, particularly under our *omnipresent rigging* which improves the ranking of a target model $m\\_t$ by manipulating battles that *do not* involve $m\\_t$ directly. As a result, detection is challenging for defenders focusing on the voting patterns of $m\\_t$ alone. For instance, attackers can avoid detection by behaving normally in battles involving $m\\_t$.\n\nMoreover, modern LLMs exhibit strong generative capabilities and often provide responses that differ subtly in style rather than in correctness. This makes it difficult for users to definitively judge which response is better. Consequently, voting outcomes can vary significantly depending on user preferences. Without a *ground-truth* ranking for reference, it is **difficult to distinguish between a ranking increase caused by vote rigging and one driven by genuine user preference**.\n\n---\n\n***Reference:*** \\\n[1] Class-Incremental Learning: A Survey", "claims_and_evidence": "The claims made are supported through empirical evidence. The authors demonstrate vote rigging by altering model rankings. Such evidence is provided through simulation experiments on around 1.7 million historical votes, showing that omnipresent rigging outperforms the simpler, target-only method.", "methods_and_evaluation_criteria": "The authors rely on a two-part procedure for “omniscient” manipulative voting. First, they train or use a classifier that predicts a model’s identity from its generated responses (even though the platform anonymizes these models). Second, once the identity of each model in a matchup is predicted, a manipulative voting rule is applied. For instance, an Elo-based or Bradley-Terry–based objective guides which model should “win” to indirectly raise the target model’s score.\n\nEvaluation primarily measures how many new votes are required to shift the target model’s ranking upward by a certain amount (e.g., from rank 20 to rank 10). The authors provide side-by-side comparisons among different rigging approaches (target-only vs. omnipresent) and document how robust each approach is under various conditions.", "theoretical_claims": "The main approach of the paper relies on the inter-connective-ness of the Bradley-Terry model, which is theoretically sound. Although the paper’s arguments are mostly kept at an intuitive or “informal proof” level as mentioned by the authors, the reasoning is coherent and standard results on Bradley-Terry models generally support the claim that any single vote in a pairwise battle influences, in principle, the final Bradley-Terry scores for all models, including those not involved in the particular matchup.", "experimental_designs_or_analyses": "**Partitioning the data**: The authors take around 1.7 million historical votes, dividing them so that 90% become the “frozen” baseline while 10% simulates other user votes happening concurrently. This is fine.\n\n**Threat Models**: They vary sampling distributions (uniform, non-uniform), anonymity levels, concurrent user votes, and the presence of unrecognized models. This is also mostly fine.\n\n**Comparison to Baselines and Defenses**: The experiments compare “target-only” vs. “omnipresent” vote-rigging strategies, and they measure how well simple detection or vote-filtering defenses can mitigate the rigging in various tables and figures. This is also fine.", "relation_to_broader_scientific_literature": "Recent studies on LLM watermarking and attribution (e.g., Zhao et al., 2024; Huang et al., 2025) have discussed the possibility of identifying a single target model and exclusively voting for it (“target-only” rigging). This paper situates those findings within the broader framework of Bradley-Terry–style ratings and extends them by introducing an “omnipresent” rigging method that can alter every new vote.", "essential_references_not_discussed": "Although the paper cites several watermaking and LLM-attribution approaches, it might benefit from a more direct discussion of LLM-based text style classification used in deepfake text detection. Incorporating references on text classifier reliability (for instance, those from the broader generative text detection literature) could help contextualize how robust or fragile the “de-anonymizing” classification step might be in the long term. Clarifying these references would strengthen the discussion around how stable model identification remains as new models emerge or older models receive updates.", "other_strengths_and_weaknesses": "**Strengths**:\n1. The paper systematically outlines a new and important vulnerability in widely used LLM leaderboards.\n2. The experimental methodology is rigorous. They run multiple ablations, threat models, and defense attempts, providing a clear picture of the rigging ecosystem.\n\n**Weaknesses**:\n1. The work mainly address Chatbot Arena, while a popular benchmark, constrain the scope of the work.\n2. Defense methods explored are somewhat rudimentary—primarily detection and filtering. While the paper concludes that robust mitigation is non-trivial, the work would be stronger if it proposed deeper or more systematic solutions.\n3. The method’s reliance on a large labeled corpus for the classification step might be more challenging in real-world practice if brand-new models appear frequently. The authors do mention some limited resilience to “unrecognized” models but maybe a more online approaches could be beneficial. \n4. The social and ethical implications of releasing or describing rigging strategies remain non-trivial, although the authors do responsibly disclaim that their demonstration is for educational purposes only.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Given the authors' findings, how feasible do the authors think it would be for a practical adversary to maintain undetectable rigging behavior over extended periods? A detailed response could clarify the practical implications of your strategies.\n2. Can the omnipresent rigging strategy generalize effectively to other voting-based evaluation platforms beyond Chatbot Arena? An affirmative answer could significantly enhance the broader impact of the work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s2CGCLx1MJ", "reviewer": "Reviewer_SYVZ", "summary": "The authors find that an attacker can meaningfully boost (or diminish) a target model’s ranking even when that model does not appear in the rigged battles. New votes cast in entirely different matchups, where the target never competes, can still change the target’s overall standing because all of the models’ ratings become tightly interconnected in the Elo/Bradley-Terry system.\n\n## update after rebuttal\nI keep my score as a strong accept after reviewing comments made during the rebuttal period.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your strongly supportive review and insightful suggestions. Below we respond to the comments in **Weaknesses (W)**, and **Questions (Q)** and will fix the typos in the paper revision.\n\n---\n\n***W1: About the claim in the experiments validated \"showing that omnipresent rigging strategies can improve model rankings by rigging only hundreds of new votes\".***\n\nIn our demonstration (Figure 1), our omnipresent rigging strategy requires around 900 rigged votes to achieve a one-rank promotion for the *Phi-3-small-8k-Instruct* model from its initial ranking position. Following your suggestion, we will revise the wording in the paper to clarify this claim and better reflect the experimental evidence.\n\n---\n\n***Q1: Does this method work if the leaderboard simply adds a delay before revealing to the attacker how their voting impacted results? How would it impact the Omni-On and Omni-BT methods?***\n\nThank you for the insightful question. In fact, our experiments in $\\\\textrm{\\\\color{blue}Section 5.3}$ (page 5), which investigate *rigging with concurrent user voting*, are designed to reflect the exact scenario you described. In this setting, the attacker does not have access to the real-time impact of their votes due to interference from other users’ concurrent voting (denoted as $\\\\mathbb{V}\\_{O}$ as in Lines 100-103), which effectively simulates a delay in leaderboard updates and creating what we refer to as a *perturbed leaderboard*.\n\nThe more concurrent votes $\\\\mathbb{V}\\_{O}$ there are, the greater the perturbation to the leaderboard, which is analogous to introducing a *longer delay* before the attacker can observe the results of their rigging. Despite this, our results in $\\\\textrm{\\\\color{blue}Table 3}$ (page 6) show that both our omnipresent strategies Omni-On and Omni-BT maintain robust and stable rigging performance, even when up to 100,000 concurrent votes are introduced.\n\n---\n\n***Other Comments Or Suggestions: \"Chtbot Arena\" and \"Copilot Areana\"***\n\nThank you for pointing out the typos. We will correct them in the revised version of the paper.", "claims_and_evidence": "The authors compare a straightforward target-only rigging approach (manipulating only battles involving the target model) against their proposed omnipresent rigging strategies (Omni-BT and Omni-On) that manipulate every vote. Their experiments demonstrate that the omnipresent methods achieve significantly higher ranking improvements.\n\nTo assess robustness, they simulate various adversarial scenarios by:\n-Varying the accuracy of the de-anonymization classifier (introducing a mix of anonymous votes) (Table 1)\n-Altering the sampling distribution so that the target model appears less frequently (Table 2)\n-Incorporating concurrent genuine user votes (Table 3)", "methods_and_evaluation_criteria": "The authors set up a series of simulation-based experiments using real historical votes from Chatbot Arena (about 1.7 million recorded votes). In those simulations:\n\n-Compare different rigging strategies (target-only vs. “omnipresent”) on various target models to see how effectively each method can boost (or lower) a model’s ranking.\n\n-Vary the “threat model”: for instance, they assume different levels of attacker knowledge (access to raw vote data vs. only the leaderboard, perfect vs. imperfect model-identity detection, etc.).\n\n-Introduce concurrent user votes (other normal votes keep rolling in) to see if real-time voting dilutes or blocks an attacker’s manipulations.\n\n-measure how many rigged votes are needed to get a certain rank increase, comparing the efficiency of voting only in matchups with the target model (“target-only”) vs. voting in all matchups (“omnipresent”)—the latter proves more efficient.\n\n-try simple defenses (like filtering suspicious votes or identifying malicious users) and see whether those actually prevent the rigging, which further highlights the system’s vulnerability.\n\nThey show that with only a few thousand malicious votes, an attacker can achieve a meaningful jump in a model’s rank.\n\nThe authors propose three methods of attack:\n\n-Target-Only Rigging: This method only manipulates battles in which the target model directly participates. When the target is detected, the strategy votes in its favor. While straightforward, it’s inefficient because the target appears in only a small fraction of all battles.\n\n-Omni-BT Rigging: This approach leverages the interconnected nature of the rating system. Instead of only affecting battles involving the target, it manipulates every battle by first de-anonymizing the participants. The strategy then chooses the vote outcome that, when applied, maximally increases the target model’s rating relative to the model immediately ahead of it. It requires access to the complete historical vote data to compute how each potential vote would affect the overall rankings.\n\n-Omni-On Rigging: Designed for situations where only the current public leaderboard is available (and not the full historical vote records), this method approximates how a new vote would change the ratings using an online update mechanism. It then selects the vote outcome that best boosts the target model’s position relative to its closest competitor. This method is more practical when the adversary has limited access to detailed vote histories.", "theoretical_claims": "No theoretical claims need be checked in this work.", "experimental_designs_or_analyses": "I only reviewed the material in the uploaded PDF file.", "relation_to_broader_scientific_literature": "I am not an expert in the leaderboard methods but am familiar with the lmsys leaderboard.", "essential_references_not_discussed": "I am not an expert in the leaderboard methods but am familiar with the lmsys leaderboard.", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper demonstrates that adversaries can dramatically manipulate model rankings by strategically rigging a relatively small number of votes. This is particularly important as these ranking systems (and lmsys' leaderboard in particular) have large impacts on the perception of  how these models are perceived. \n\nI appreciate the novelty in the fact that the proposed method is particularly effective at impacting the reliability and integrity of vote-based evaluation systems because it does not require rigged votes that involve the target model only. \n\nWhere is this claim in the experiments validated \"showing that omnipresent rigging strategies can improve model rankings by rigging only hundreds of new votes\"? This is mentioned in a couple of areas in the text. It seems like the number of rigged votes is on the order of thousands based on the experiments.", "other_comments_or_suggestions": "\"Chtbot Arena\"\n\"Copilot Areana\"", "questions_for_authors": "Does this method work if the leaderboard simply adds a delay before revealing to the attacker how their voting impacted results? How would it impact the Omni-On and Omni-BT methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper targets systems that rely on voting mechanisms, suggesting ways to rig the scoring (and preliminary defence mechanisms). While the methods proposed are largely impractical, the reviewers believe that the ideas this paper carries about the susceptability of social choice mechanisms could affect the community.

During the reviewing phase, the reviewers raised several important concerns, most notably the lack of a relevant literature review on social voting and its vulnerabilities. The authors' rebuttal has addressed all of these concerns. Particularly, the authors have promised to extend the literature review as needed.

Overall, I share the reviewers' positive opinion about this paper, and I believe it could be interesting to the community and generate fruitful discussions.
Decision: Accept (poster)