Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Authors: Wayne Chi
Valerie Chen
Anastasios Nikolas Angelopoulos
Wei-Lin Chiang
Aditya Mittal
Naman Jain
Tianjun Zhang
Ion Stoica
Chris Donahue
Ameet Talwalkar
Venue: ICML 2025 poster (2025)
Keywords: evaluation
code
llm
Primary area: general_machine_learning->evaluation
TL;DR: Copilot Arena is a platform for conducting realistic evaluations of code LLMs, collecting human preferences of coding models with real users, real tasks, and in realistic environments.

Abstract: Evaluating in-the-wild coding capabilities of large language models (LLMs) is a challenging endeavor with no existing solution. We introduce Copilot Arena, a platform to collect user preferences through native integration into a developer's working environment. Copilot Arena comprises a novel interface for comparing pairs of model outputs, a sampling strategy to reduce experienced latency, and a prompting scheme to enable code completion functionality. Copilot Arena has served over 4.5 million suggestions from 10 models and collected over 11k pairwise judgements. Our results highlight the importance of model evaluations in integrated settings. We find that model rankings from Copilot Arena differ from those of existing evaluations, which we attribute to the unique distribution of data and tasks contained in Copilot Arena. We also identify novel insights into human preferences on code such as an observed consistency in user preference across programming languages yet significant variation in preference due to task category. We open-source Copilot Arena and release data to enable human-centric evaluations and improve understanding of coding assistants.

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Wayne Chi Valerie Chen
Anastasios Nikolas Angelopoulos 2
Wei-Lin Chiang² Aditya Mittal
Naman Jain
Tianjun Zhang? Ion Stoica Chris Donahue
Ameet Talwalkar
Abstract
Copilot Arena Arena
Evaluating in- wild coding capabilities of large language models (LLMs) is a challenging en- deavor with no existing solution. We introduce Copilot Arena, a platform to collect user pref- erences through native integration into devel- oper's working environment. Copilot Arena com- prises novel interface for comparing pairs of model outputs, sampling strategy to reduce ex- perienced latency, and a prompting scheme to enable code completion functionality. Copilot Arena has served over 4.5 million suggestions from 10 models and collected over 11k pairwise judgements. Our results highlight the importance of model evaluations in integrated settings. We find that model rankings from Copilot Arena dif- fer from those of existing evaluations, which we attribute to the unique distribution of data and tasks contained in Copilot Arena. We also iden- tify novel insights into human preferences on code such observed consistency in prefer- ence across programming languages yet signifi- cant variation in preference due to task category. We open-source Copilot Arena and release data to enable human-centric evaluations and improve understanding of coding assistants.
C
Real Users
Real Tasks
Real Environment
Static benchmarks do not have users in the loop
User studies operate on limited, prescribed set of tasks
Preference evaluations do not occur realistic coding envs
Limitations of existing evaluations
Figure 1. Copilot Arena is platform for conducting realistic eval- uations code LLMs, collecting human preferences coding models with real users, real tasks, and in realistic environments, aimed at addressing the limitations of existing evaluations.
evidence identified by electronic discovery systems (Yang et al., 2024a). Increasing deployment of models in produc- tivity tools demands evaluation that more closely reflects real-world circumstances (Hutchinson et al., 2022; Saxon et al., 2024; Kapoor et al., 2024). While newer benchmarks and live platforms incorporate human feedback to capture real-world usage, they almost exclusively focus on evaluat- ing LLMs chat conversations (Zheng et al., 2023; Dubois et al., 2023; Chiang al., 2024; Kirk al., 2024). Model evaluation must move beyond chat-based interactions and into specialized user environments.
1. Introduction
As model capabilities improve, large language models (LLMs) increasingly integrated into user environments and workflows. For example, software developers code with Al in integrated developer environments (IDEs) (Peng et al., 2023), doctors rely on notes generated through ambient listening (Oberst et al., 2024), and lawyers consider case
In this work, we focus on evaluating LLM-based coding assistants, specifically focusing on their ability to generate code completions. Despite the popularity of these tools- millions developers use Github Copilot (Github, 2022) existing evaluations of the coding capabilities of new mod- els exhibit multiple limitations (Figure 1, bottom). Tradi- tional MI benchmarks evaluate L.M capabilities by mea- suring how well model can complete static, interview-style coding tasks (Chen et al., 2021; Austin et al., 2021; Jain et al., 2024a: White et al., 2024) and lack real users. User studies recruit real users to evaluate the effectiveness of
Co-senior Authors. Carnegie Mellon University *UC Berkeley. Correspondence to: Wayne Chi <waynechi @ andrew.cmu..duu. Valerie Chen <<va- leriechen cmu.
Equal contribution
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
User IDE
Backend Server
21
Class OpenAIC Lient (IBaseClient);
Sampling
52
async def create(
distribution
model
model
latency (ms)
53
self, state: State, model: str,
10%
gpt-4g
gpt-4o-mini
1110
54
5%
gpt-4o
sonnet-3.
55
3%
sonnet-335
1ama-3-770
70B
1875
options: LLMOptions,
sornet-3.5
despsook
1320
57
LLMResponse:
sonnet-3355
codestral
1270
58
try:
codestral
deapsank
850
59
response await self.client.chett cww..ccctt...ooceeeoooooo conpeetttttttsscreeeeeeee
60
Ressages=
sessagesmpenere pttpptteppppp____eeeeeee wwr_forroodde for_nodel(__ (state, nodel, mode Lssodel,
Prompting
The correct outcone (zed) nust be fomatted correctly. To help with fomatting,
xeep =
await completions create( nessages-ntats get nesssges()) podel-nodel,
texperaturen eurreooptios tions,tnnns......oooo.oooooss.. texperature,
nax tokensmoptions cioosonn.. nax. tokens,
select
tab
strear-False
Repeated prefix enhance readability
messageswstate. wwwplleee__essseee (esssagssss,
gonarata the antire snippet and ranave and overlapping (blue)
resy await completions createl messagesnstatt wetesttttteteeteeeeeeeeeeeeeeeee get_neseeges() model -nodel, strean-False
Rode Imnodel,
#toptions. to dict(),
select bottom
shift tab
Figure 2. We introduce Copilot Arena, VSCode extension to collect human preferences of code directly in a developer's IDE. Copilot Arena enables developers to use code completions from various models. The system comprises a) the interface in the user's IDE which presents paired completions to users (left), b) sampling strategy that picks model pairs to reduce latency (right, top), and c) prompting scheme that allows diverse LLMs to perform code completions with high fidelity. Users can select between the top completion (green box) using ta or the bottom completion (blue box) using shift+tab.
LLMs as coding assistants, but are often limited simple programming tasks (Vaithilingam et al., 2022; Ross et al., 2023; Mozannar et al., 2024b) as opposed to real tasks. Recent efforts to collect human feedback such as Chatbot Arena (Chiang et al., 2024) are still removed from a real- istic environment, resulting in users and data that deviate from typical software development processes. We introduce Copilot Arena address these limitations (Figure 1, top), and we describe our three main contributions below.
benchmarks compared our leaderboard, while perfor- mance among larger models is mixed. We attribute these differences to the fact that Copilot Arena is exposed to users and tasks that differ drastically from code evaluations in the past. Our data spans 103 programming languages and 19 natural languages as well as a variety real-world appli- cations and code structures, while static benchmarks tend to focus on specific programming and natural language and task (e.g. coding competition problems). Additionally, while all of Copilot Arena interactions contain code contexts and the majority involve infilling tasks, much smaller frac- tion of Chatbot Arena's coding tasks contain code context, with infilling tasks appearing even more rarely.
We deploy Copilot Arena in-the-wild to collect human preferences on code. Copilot Arena is Visual Studio Code extension, collecting preferences directly in developer's s IDE within their actual workflow (Figure 2). Copilot Arena provides developers with code completions, akin to the type of support provided by Github Copilot (Github, 2022). Over the past 3 months, Copilot Arena has served over 4.5 mil- lion suggestions from 10 state-of-the-art L.M.ss gathering 1 1600 votes across 1642 users. To collect user preferences, Copilot Arena presents a novel interface that shows users paired code completions from two different LLMs, which are determined based on sampling strategy that aims to mitigate latency while preserving coverage across model comparisons. Additionally, we devise prompting scheme that allows a diverse set of models to perform code comple- tions with high fidelity.
We derive new insights into user preferences of code by analyzing Copilot Arena's diverse and distinct data distribution. We compare user preferences across differ- ent stratifications of input data (e.g., common versus rare languages) and observe which affect observed preferences most. For example, while user preferences stay relatively consistent across various programming languages, they dif- fer drastically between different task categories (e.g. fron- tend/backend versus algorithm design). We also observe variations in user preference due to different features related to code structure (e.g., context length and completion pat- terns). We open-source Copilot Arena, all user votes, and a curated subset of code contexts.' Altogether, our results highlight the necessity of model evaluation in realistic and
We construct a leaderboard of user preferences and find notable differences from existing static benchmarks and human preference leaderboards. In general, we observe that smaller models seem overperform in other static
All code and data is available at ma maarena/cop imarena/copilot arena.
2

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
domain-specific settings.
Acceptance Rate (9)
2. System Design
0.5
0.4
Copilot Arena is a VSCode extension that provides users with pairs of inline code completions from various LLMs. In return, users provide their votes on which completion is better suited for their task. To avoid interrupting user workflows, voting is designed to be seamless- users use keyboard shortcuts to quickly accept one of the two comple- tions into their code, which we interpret as a vote in favor of the underlying model that produced it. Designed to allow for developer's 's day-to-day usage, the three core components Figure 2-are the 1) User Interface, 2) Model Sampling, and 3) Model Prompting.
0.3
0.0
Pairwise Latency (seconds)
The likelihood of users accepting one of the two com- pletions as a function of empirical pairwise latency (determined by the slower the two models). As latency increases, users are less likely to accept completion. We devise sampling strategy described in Section 2.2 which reduces pairwise latency by 33% while also ensuring sufficient coverage of unique model pairs.
Figure 3.
of Copilot Arena-overvinn
2.1. User Interface
Traditional code completion tools (e.g., GitHub Copi- lot (Github, 2022)) only show one completion at a time. However, showing Iwo code completions simultaneously en- ables us to collect preference judgments on the same context and facilitate a more seamless user experience (Chiang et al., 2024; Lu et al., 2024). As such, we propose an interface that allows user to view two completions in a head-to-head manner; to our knowledge, we are the first to introduce an interface that does so. We propose a design inspired by Git Diff-a well-established tool familiar to many developers- which displays code from the current commit and code from the incoming commit stacked vertically, one on top of the other. In a similar manner, given an existing code context, we also stack responses from two different model outputs. This allows users to examine both completions together (an example of how the completions are visualized in Fig- ure 2). The user can accept the top suggestion using tab and the bottom suggestion using shi ftttab, or decide neither is appropriate and continue typing. The only distinc- tion between our system and conventional inline completion systems is the inclusion of second suggestion, resulting in a user experience that is familiar overall.
2.2. Model Sampling
A key challenge in building a realistic environment for cod- ing assistance is providing responsive code completion. De- veloper expectations for low latencies impact not only user satisfaction and retention, but also directly affect their likeli- hood to provide preference data. The slower the completions are returned to the user, the less likely users are to vote (i.e. users select neither completion) (Figure 3). However, many model providers do not optimize their API endpoints for low-latency use cases, requiring us to explore a sampling strategy that improves our system-wide latency.
Since the Copilot Arena interface shows two code com- pletions together, the slowest completion determines the latency. Thus, given a set of M models {l...., M,. we let Finux (1; j, denote the cumulative density function (CDF) for the maximum latency between models i and j. Because latencies tend be long-tailed, we model Fmax (4; j) as a log-normal CDF with parameters estimated from our his- torical data. Our objective will then be minimize the expected latency of the chosen model pair under the distri- bution induced by our observed data,
We make several other notable design decisions. First, we repeat the first line in the ghost text of the top completion so that both top and bottom completions text. Not repeating the text-as is the case with single completion- was an alternative we considered, but our ini- tial pilot studies indicated that the discrepancy between top (partial ghost text) and bottom (full ghost text) completions was more likely to confuse users. Second, we always wait for both completions to finish generating before showing them to the user to reduce the effects of latency on user preference, which we aim to study separately in Section 5.2. Lastly, we randomize the ordering of the completions to remove top-bottom bias from our preference evaluation. We discuss additional design decisions in Appendix A.I.
((0) = E(id' Farn [L],
(1)
entirely ghost
where pe is a distribution over model pairs,
exp(01g/T)
pe (i(i.) Ener exp(0sa/7)
(2)
Above, 7 is a temperature parameter that interpolates be- tween a latency-optimized distribution and a uniform dis- tribution, allowing us to trade off latency and coverage of unique model pairs. The parameters 0 E ((M) are opti- mized via gradient descent to minimize (Eq. 1). In practice, we set 7 to values between 5 and 10 to ensure sufficient cov- erage. By deploying our algorithm, we observed a decrease
3

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
cess of standard prompt templates varies greatly between models (Table 2). This is not necessarily indication that models cannot code as clearly many sate-of-the-art chat models are proficient coders (Jain et al., 2024a; Lin et al., 2024). Instead, the vast majority of the errors result in for- matting issues or duplicate code segments rather than logical errors, indicating that FiM performance is inhibited more by low-level formatting issues than high-level coding capa- bilities: see examples of these errors in Appendix A.2.3.
0.8
0.7
0.6
Stip Passigh
0.5
0.4
Post- Post-processing using Snip-It. While it is not feasible to retrain these models because many of them offer API ac- cess only, we explore alternative approaches via prompting to improve chat models' abilities to complete FiM tasks. Specifically, we allow the model to generate code snippets, which is more natural format, and then post-process the snippets into FiM completion. Our Srrtoach--nippI----1 as follows: the model is prompted with the same prompts as above (e.g. prefix-sufix-middle but with instructions to begin by repeating a portion of the prefix and similarly end by repeating a portion of the suffix. Then, we remove any portion of the output code that already exists in the input, similar to recent agentic search-replace tools (An- thropic, 2024). As shown in Figure 4, we found that, rel- ative to the baseline, Snip-It provides robust performance gains for infilling: performance improved in 93% of the conditions. High-performing models improve substantially (c.g., Claude-3.5-Sonee improves from 56.1% to 73.0%), while initially struggling models improve dramatically (e.g.. 1.mm--31170BB from 7.4% to 49%). While offline eval- uation is not a perfect metric, find that these drastic improvements enable these models for FiM tasks.
0.0
0.2
0.4
0.6
Vanilia Pass@1
Figure 4. We evaluate the effectiveness of our Stip-It method by comparing LLM performance on infilling tasks (using pass I) before and after applying Snip-It. We evaluate 9 different models of varying performance across 4 different prompt templates (i.e., ways of encoding the prefix and suffi in the prompt): each point represents one model and one prompt template pair. We observe that, across the board, all pairs generally benefit from the Snip-It method (c.g., g., lic above the diagonal line).
in median experienced latency by 33% (from 1.61 to 1.07 seconds) compared to a uniform distribution.
2.3. Model Prompting
During real development processes, developers frequently modify or expand upon existing code which requires models to infill between code segments. However, many popular coding models such as GPT-4o or Sonnet 3.5 are instruction- tuned (Wei et al., 2022) and trained output text left- to-right autoregressively. rather than "fil-dn-the-mmidlle" (FiM) (Fried et al., 2023; Gong et al., 2024). In prelimi- nary experiments, we observed poor, essentially unusable performance of instruction-tuned models on FiM tasks. Ac- cordingly, we use offline datasets to improve chat models' infilling capabilities through prompt engineering. We in- clude full experimental details and results in Appendix A.2.
3. System Deployment
Deployment Details. The Copilot Arena extension is adver- tised in online open-source communities and made available on the VSCode extension store, where it is free down- load. Similar to the set-up employed by Chiang et al. (2024) and Lu et al. (2024), participants are not compensated for using the extension, as in traditional user study, but in- stead receive free access to state-of-the-art models. In ad- dition to logging all preference judgments made by users of Copilot Arena, we also log the latency of each model re- sponse, the type of file the user writing, the prefix and suf- fix length (characters and tokens), each completion length, which model was in the top versus bottom position, and unique userll)-al all of which allows users to utilize the extension without revealing the content of what the user is working on. Given the sensitive nature of programming, we established clear privacy controls to give users the ability to restrict our access to their data. Depending on privacy settings, we also collect the user's code context and model responses. Appendix B provides a copy of the specific user
Offline Evaluation Set-up. Our set-up uses the HvmanEval- infilling dataset (Bavarian et al., 2022) which consists of 1640 examples where random spans in a completed portion of code are masked out to simulate FiM behavior. To incor- porate prefix and suffix information, we began with several prompt templates from Gong et al. (2024) with modifica- tions to align the prompts with chat models (c.g., initial instruction and few-shot examples). The templates capture different ways to encode information about the given code context. For example, nrefix-suffx-micee presents the code context in the order of prefix and then suffix, and the LLM is asked to output the middle.
Vanilla performance on FiM tasks. We find that the suc-
4

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
instructions and privacy guidelines. The data collection pro- cess is fully compliant with appropriate laws and regulations regarding user privacy within our institution and locale.
Chamot Arena (general)
Chatbot Arna (coding)
UvatadeBench BiyCadeBench
Capilot Arena
LiveBench
Data collection process. We select 10 state-of-the of-the--rtt-e art mod- els to balance a set of open and commercial models, as well as generalist and code-specific models. In latter anal- ysis, we refer to LLMs by shortened names to conserve space: please check Table 3 for full model names. Across 1642 users, we have served over 4.5 million suggestions and collected 1 11604 votes over the course of 3 months. Overall, we find that all models received between 2-5K votes, satis- fying our coverage objective. In general, the median time to vote-the the time taken after the completion is displayed to the user-was 7 seconds, suggesting that users did not accept all suggestions immediately and considered both completions. A more in-depth overview of data analysis is in Appendix C.
deepseek-coder
cleude-3..rsonnet
1
2
44
codestral-
2
-6
Ia-me3.1--03b-
-3
gashni--
-1
3
-1
-2
-3
42
0
gpt-40-7024-08-00
4
43
43
-3
33
liama-3.1
(wen-2.5-coder322
9
+7
+8
+6
0
+1
9
+3
+1
+4
+6
+5
r-0.10 P=-0.15
r=-0.10 r=0.48
rw0.62
Figure 5. We compare model rankings in Copilot Arena (Ist col- umn) existing evaluations, both static benchmarks (2nd-4th column) and live preference evaluations (last two columns). For existing evaluations, we show the change in rank relative Copi- lot Arena rank, with positive values in green denoting models performing better on existing evaluations, negative values in red denoting models performing worse, and dash indicating that the model is not present in the live leaderboard. We also report the Spearman rank correlation coefficients between Copilot Arena and other leaderboards.
Data Release. Upon publication, we will release the full dataset of I 11604 votes, with non-deanonymizing features (e.g., context length or file type). Despite giving users full control over their privacy, we take a conservative approach to data release given the potential sensitivity of coding data. To demonstrate the type of code users write using Copilot Arena, we will also release hand-curated set of 500 ex- amples that contain the prefix, suffix, and both completions. This portion of the dataset spans all 10 models and cap- C
model's rank is determined by which other models' lower bounds fall below its upper bound.
tures a variety of downstream applications- shows multiple examples. Two authors carefully checked this set of examples to ensure the code also contained no sensitive information or personally identifiable information. We intend continue slow release in the future.
Constructing our leaderboard (Figure 5, 1st column), we find that our leaderboard is segmented into multiple tiers based on the estimated 3, values (Table 4). In the first tier, DeepSeek Coder and Claude Sonnet-3.5 are at the top, with Codestral following closely behind. In general, we observe that code-specific models (e.g.. DeepSeek Coder and Code- stral) are competitive with general-purpose state-of-the-art models (e.g. Claude Sonnet-3.5), especially if they are trained to infill. In the second tier, there are 5 models of varying sizes and from different model providers that have relatively similar strengths. In the final tier, users preferred two models the least. In particular, Qwen-2.5-coder is an ex- ception, performing notably worse than other code-specific models. Implementation of leaderboard computation and additional ablations on provided Appendix D.
4. Model Rankings
4.1. Copilot Arena Leaderboard
We construct a leaderboard using our user preference judge- ments. Let n denote the number of judgments and M the number of models. For each battle E [n], we define: X, (-1,0, 1): Xim if model m is presented in the top position, Xim
11 if presented in the bottom posi- tion, and 0 otherwise. The outcome Y, E {0, 1}, where indicates the top model won. As is standard in other work on pairwise preference evaluation (Chiang et al., 2024; Lu et al., 2024), we apply a Bradley-Terry (BT) model (Bradley & Terry, 1952) to estimate the relative strengths of models R.M where the probability Pij that model i beats model j can be modeled as:
4.2. Comparison against Prior Evaluations
We compare our leaderboard to existing evaluations which encompass both live preference leaderboards with human feedback and static benchmarks (Figure 5, 2nd-5th col- umn). For human preferences, compare against Chat- bot Arena (Chiang et al., 2024) in-the-wild platform for evaluating human preferences on chat capabilities- across both the general leaderboard and the coding subset. For static coding benchmarks, we select three that are re- cent and continue to be maintained (of which we have at
Pij
B + e,
We bootstrap the battles in the BT calculation to construct a 95% confidence interval for the rankings, which are used to create leaderboard that ranks all models, where each
5

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Programming Languages
Natural Languages
Table 1. We compare Copilot Arena with prior evaluations in terms of scale, context length, task type, and code structure. Copilot Arena provides broad coverage across programming languages
Python/488s
pesscippt (19%))
Code Only (50%)
HH (MMa/MMaaoooo n 1750
English (45%)
-jeve 1356)
(PL), natural languages (NL), context length in characters, mul- tiple task types, and structural dimensions--ween contains code and mildiee (FiM) tasks present. Chatbot Arena (code) only contains code 40% and infilling in 2.6% of its input and is denoted by vr. In Figure compare against benchmarks that are updated with the latest models (denoted by *).
4HHP (3%)
Russian (2%)
the context
Go
CSS (250
Chinese (156)
Plaintext (1%%)
Spanish (156)
C (155)
languages (2%)
languages (156)
10
20
50
10
40
Froquencyy (86
Frequencyy (86
Tasks
Context Lengths
Backend Development 126
<100 characters (11%)
Frontend Development (20%)
100 1, 1,000 characters (30%)
Scale
Context Len
Task
Structure
Algorithm Des kn (18%)
1,000 2,500 characters (19%)
Benchmark
# PL
# NL
p50
p95
Multi
Code
FiM
Data Processinng (15%)
Copilot Arena
103
1.6k
18k
v
V
Handding (118)
2,500
5,000 characters (24%)
HumanEval
0.4k
0.9%
V
X
Machine Learning (?56)
5,000
10,000 characters (1 196)
HumanEval-XL
12
0.4k
0.9%
V
X
SAFIM
5.9k
V
Game Dev and Simulatiios (3%)
10,000 characters (13%)
LiveCodeeencc*
1.4k
2.5k
V
X
25
LiveBench"
2.3k
3.9%
X
Frequuincy (96)
Frequency (56)
RigCceeeench
1.1k
1.9k
Last-line Contexts
Completion Structures
Chatbot Arena (generaly
2 17
2 49
0.7k
2.9K
V
Chatbot Arena (code)
2 17
2 39
1.4k
7.8k
V
Docstrings Block comments (5%)
Completion-onl) only (35%)
Comments
(34%)
Code Only
FIM (65%)
(6116)
least 8 out of 10 overlapping models): LiveBench (White al., 2024)-a collection of benchmarks designed to mit- igate test set contamination Lititattttoo-iieeeeeeBeBeeee (Jain et al., 2024a)-a code benchmark designed mitigate test set contaimnation and BigCodeBench (Zhuo et al., 2024)-a benchmark for code generation which focuses on function calls. We do not compare to rankings from any user studies because they are difficult to keep updated in comparison to both static benchmarks and live comparative systems.
Figure 6. Copilot Arena data is diverse in programming and natural languages, downstream tasks, and code structures (e.g., context lengths, last-line contexts, and completion structures).
considered in prior evaluations (Table 1).
We find the highest correlation (r, 0.62) with Chatbot Arena (coding) (Chiang et al., 2024) and similarly high correlation (r, = 0.48) with Chatbot Arena (general), How- ever, we find a low correlation (r, S 0.1) with most static benchmarks. The stronger correlation with human pref- erence evaluations compared to static benchmarks likely indicates that human feedback captures distinct aspects of model performance that static benchmarks fail to measure. We notice that smaller models tend to overperform (e.g., GPT-4o mini and Qwen-2.5-Coder 32B), particularly in static benchmarks. We attribute these differences to the unique distribution of data and tasks that Copilot Arena evaluates over, which we explore in more detail next.
Programming and Natural Language: Previous bench- marks such as HumanEval (Chen et al., 2021) cover a lim- ited number of languages, primarily focusing Python and English (Bavarian et al., 2022; Jain et al., 2024a; White et al., 2024; Zhuo et al., 2024). While recent work such as HumanEval-XI. (Peng et al., 2024) and SAFIM (Gong et al., 2024) has expanded coverage to up to a dozen program- ming languages, Copilot Arena covers 103 programming languages which is an order of magnitude more than most other benchmarks. While the distribution of programming languages is not uniform, there is still core set of pro- gramming languages which contain a significant number of votes (Appendix C). Similarly, while the majority of Copi- lot Arena users (45%) write in English, we also identify 19 different natural languages which comparable to Chat- bot Arena (general) (Chiang et al., 2024) and benchmarks focused on multilingual generation (Peng et al., 2024).
5. Data Analysis
5.1. Exploring Copilot Arena Data
Evaluating models in real user workflows leads to diverse data distribution in terms of programming and natural lan- guages, tasks, and code structures e.g., context lengths, last-line contexts, and completion structures (Figure 6). We discuss how our data distribution compares against those
Existing benchmarks tend to source problems from coding competitions (Jain et al., 2024a; White et al., 2024), handwritten programming chal- lenges (Chen et al., 2021), or from a curated set of GitHub repositories (Gong et al., 2024). In contrast, Copilot Arena
Downstream Tasks:
6

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
users are working on a diverse set of realistic tasks, includ- ing but not limited frontend components, backend logic, and ML pipelines (we provide representative examples of the different task clusters in Appendix C.I). Coding style problems (i.e., algorithm design) comprise a much smaller portion-18% of Copilot Arena's data. Further, the dis- tribution of downstream tasks for our in-editor suggestions differs from questions raised by chat conversations, e.g., in Chathot Arena (Chiang al., 2024), where coding questions also focus on code explanation or suggesting commands.
Front/Backend
Long Context
Non-Python
deepseek-coder
6, 3
+2. e
+1
9.
cloude-3. Souude-3.--sonnee
44, 0
6,-1
12.
+1,
codestral
+2, 0
+1,11
8, 0
9.
llama-3. 1-405b
+5,-4
+1,-1
8, 0
8.
gemini-ffash-Uu 002
+1,-2
0,
+1,-2
0,
gemini-pra-002-
+1. 0
+3. 8
12, 0
0,
gpk-4o-2024-08-06
+2, 0
0,-2
0.-2
+1.
1ama-3.1-700
44.
+1, 0
+1,-2
0,00
-wen-225xoddee336 S-coder-326 326-
qwen-2
0,-2
0,:3
9.
0.-2
Code Structures and Context Lengths: Most coding benchmarks follow specific structures, e.g., taking struc- tured doestrings as input (Chen et al., 2021: Zhuo et al., 2024; Jain et al., 2024a; White et al., 2024) or infilling tasks (Bavarian et al., 2022: Gong et al., 2024). This means that most benchmarks have relatively short context lengths (e.g,, all HumanFival (Chen et al., 2021) problems are less than 2k characters). Similarly, Chiang et al. (2024) focuses on natural language input collected from chat conversations, with many prompts not including any code context (e.g., 40% of Chathot Arena's s coding tasks contain code context and only 2.6% focus on infilling). As such, input prompts are also relatively short, with 95% of prompts falling be- tween 1-3k characters. Unlike any existing evaluation, Copi- lot Arena is structurally diverse, comprising mixture of infilling versus code completion and forms of docstring tasks. Since users are working in actual IDEs, they work on significantly longer inputs: the median context length is around 1.6k characters and 95% of inputs fall within 18k characters.
apt-4o-mini
+
9,-1
+1.
% Total Changes:
31.1
17.8
15.6
6.7
Figure 7. Significant win-rate changes as a result of different data partitions: frontend/backend versus algorithmic problems, long versus short contexts, FiM vs non-FiM, non-Python vs Python. We report the number of positive and negative changes (e.g.. +1/-2 means that a model improved over model and worsened against models). In general, we observe the largest percentage of total changes as result of differences in task, while the smallest effects as result of differences in programming language.
our dataset. We stratify the data as follows: For tasks, we compare frontend/backend against algorithm design. For context length, we compare the top 20% against the bottom 20%. For FiM, compare FiM against completion only. For programming languages, we compare all other program- ming languages against Python. We stratify these input fea- tures to highlight differences between the data distribution in Copilot Arena compared to static benchmarks (Table 1), where a positive win-rate indicates increased model perfor- mance on data that may be considered out of the distribution of typical static benchmarks. See Appendix E. for full data win-rates.
5.2. Understanding User Preferences of Code
Given our diversity of input features, we evaluate how each impacts user preference. We partition each feature into contrasting subsets (e.g. FiM non-FiM), which we refer to as X and X. For each subset, we compute the 2 matrix W E MM M where W ((X) represents the win-rate matrix of subset X and Wij E [0, 1]. For each feature, we compute a win-rate difference matrix A E MM,, which represents the number of substantial differences in the win- rate between W (X) and W(X).
Downstream task significantly affects win-rate, while programming languages have little effect. Changing task type significantly affects relative model performance, with 28 significant win-rate changes (31.1% of all possi- ble changes). This gap may indicate that certain models are overexposed to competition-style algorithmic coding problems. On the other hand, the effect of programming language on win-rates was remarkably small, resulting in only 6 (6.6%) significant changes, meaning that models that perform well on Python will likely perform well on another language. We hypothesize that this is because of the inherent similarities between programming languages, and learning one improves performance in another, aligning with trends reported in prior work (Peng et al., 2024). Context length and FiM have moderate effects to win-rate, which lead to 16 (17.8%) and 14 (15.6%) significant changes respectively.
A. = 1 (X),, (x)( > cl
In our analysis, substantial changes are those in the top 90th percentile of win-rate changes (c = 0.166). Since
M = 10, the maximum amount of significant changes is 90 (|A) < 90).
We compute A for four input features task type, context length, FiM, and programming language-wher contrast- ing strata are present in sufficient quantity (2 10%) within
2 Inspecting win-rates helps circumvent potential issues that may arise from applying BT regression to slices with fewer votes.
Smaller models tend to perform better on data similar to static benchmarks, while the performance of larger

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
models is mixed. For example, Qwen-2.5 Coder performs noticeably worse on frontend/backend tasks (-2), longer con- texts (-3), and non-Python settings (-2). We observe similar trends for the two other small models (Gemini Flash and GPT-40 mini) across multiple features. We hypothesize that overexposure may be particularly problematic for smaller models. On the other hand, performance amongst larger models is mixed. For example. Gemini-1.3 Pro performs noticeably better (+3) on long context which aligns with its goal of long context understanding (Team et al., 2024). However, Llama-3.1 405B underperforms on frontend/back- end tasks (-4).
6. Related Work
Human Preferences for Evaluations. A diverse set of hu- man cisterennee-includiinn binary preferences (Bai et al., 2022), fine-grain feedback (Wu et al., 2023; Kirk al., 2024), and natural language (Scheurer al., 2022) are in- creasingly used for training and fine-tuning LLMs (Ouyang et al., 2022). Preferences are also important for human- centric evaluation, especially as LLMs are deployed in con- texts that involve human interaction. Platforms like Chatbot Arena (Chiang et al., 2024) and Vision Arenas (Chou et al., 2024; Lu et al., 2024) provide a way for users inter- act with LLMs and provide paired preference judgments. However, existing arenas lack integration into actual user environments to reflect the diverse data that may appear in a user's workflow. We study the use case of LLMs as coding assistants and introduce Copilot Arena to ground preference evaluations in a developer's 's working environment.
Surprisingly, models explicitly trained for infilling do not experience large changes to win-rate. Neither DeepSeek Coder, Codestral, nor 5wen-2.5 Coder sees any noticeable performance gains due FiM. We an experiment using DeepSeek Coder's Chat API with Snip-It (rather than FiM) and observe that relative model performance remains con- sistent (Table 6). These results suggest that Copilot Arena captures signals about code quality or usefulness rather than just formatting.
Evaluations of LLM coding capabilities. Static benchmarks, e.g., HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021), largely focusing on interview- style programming problems have been the most commonly used to evaluate coding capabilities (Lu et al., 2021; Ni- jkamp et al., 2023; Zhu et al., 2022; Wang et al., 2023;
5.3. Insights and Takeaways
Based on our findings, we discuss two key insights and their implications for developing new models:
Liu et al., 2023; Jimenez et al., 2023; Khan et al., 2023;
Yan et al., 2023: Cassano et al.. 2023; Muennighoff et al., 2023; Dinh et al., 2023; Yang al., 2024b), measured using pass@k. Recent benchmarks aim to create more realistic problems, which include multi-turn program eval- uations (Nijkamp et al., 2023) and repository-level chal- lenges (Jimenez et al., 2023; Jain et al., 2024b), and create live benchmarks that reduce contamination risks (Jain et al., 2024a; White et al., 2024). Our evaluation platform comple- ments the existing suite of benchmarks by contextualizing model evaluations in an actual user's 's workflow as coding assistants, measuring a model's 's quality based on user pref- erences. Preference data retains signal when models output slightly incorrect, but still useful answers as opposed to a strict or all or nothing when evaluating using test cases.
Model performance is dictated by task, context, and code structure. For example, while Claude 3.5 Sonnet excels at frontend and backend tasks, Gemini Pro and Deepseek out- perform at longer contexts. With the exception of different programming languages, we observed that any other varia- tion to the setting or use case affected model performance. Since there seems to be no model that is sone-size-fis-alll it important for end users and developers of LLM-powered applications choose their model depending their use case. Alternatively, routing approach based on input code context (Frick et al., 2025) is an interesting direction for future research.
Human preference data is essential for effective code generation models. The discrepancy between our leader- board and static benchmarks indicates that current models are likely not trained on human preference data. Recent approaches have considered this, but still have significant gaps (e.g., Qwe Coder trains on LLM-as-a-Judge pref- erences as a proxy for human preferences (Qwen et al., 2025)). We believe that training code models on human preference data is a promising future research direction. By releasing Copilot Arena data, we hope help address the need for human preference data in real-world coding contexts.
A growing set of user studies aim to study human inter- actions with LLMs (Lee et al., 2023), particularly how programmers use LLM assistance for software develop- ment (Barke et al., 2023; Vaithilingam et al., 2022; Ross et al., 2023; Peng et al., 2023; Mozannar et al., 2024a; Mu- rali et al., 2024: Chen et al., 2024). A notable work by Cui et al. (2024) conducted a field study on GitHub Copilot with many users. However, these studies generally face challenges of scale in terms of the number of users and the models considered, primarily relying commercial tools like GitHub Copilot or ChatGPT. Mozannar et al. (2024b) conducted study to evaluate six different LLMs of varying performance and Izadi et al. (2024) similarly conducted a study with three different LLMs, but the models evaluated
8

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
in both studies longer considered state-of-the-art. Our platform aims to address these challenges by building and deploying actual coding assistant that allows for scalable and adaptable evaluation as new models emerge.
itate further understanding of how user preferences vary, highlighting the importance of evaluating new models with real users, tasks, and environments.
A Acknowledgments
7. Discussion
This work was supported part by the National Science Foundation grants IIS IIS170512,, IIS 18$1838177 IIS2046613, IIS21 12471, and funding from Sony AI, Meta, Morgan Stan- ley, Amazon, Google, and Scribe. Any opinions, findings and conclusions or recommendations expressed in this mate- rial are those of the author(s) and do not necessarily reflect the views of any of these funding agencies.
Limitations. Although we have a diverse set of users and cases, it is unclear to what extent results encapsu- late all real-world use cases. We run extensive pilot tests to ensure platform usability, but we recognize that certain aspects- our pairwise completions and slower latency-do not perfectly mirror real-world platforms such as Github Copilot. Further, while we rank models based on user preferences, this should not be treated as the sole defining metric of model quality, but instead informa- tive one. In this work, we evaluate multiple LLMs with strong coding capabilities; however, we are unable to in- clude Github Copilot because the model powering Github Copilot is not available via API. Additionally, since this is a live service, model performance directly affects user retention (and votes), rendering it difficult for us to evaluate weaker coding models. Finally, due to privacy considera- tions, choose not release all code contexts collected in the study without careful post-processing, We strive to make all data open through periodic releases.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal con- sequences of our work, none which we feel must be specifi- cally highlighted here. However, we would like to highlight the ethical considerations of user privacy involved with storing and releasing user code data. We detail such consid- erations extensively in both Section 3 and Appendix B.
References
Anthropic. Raising the bar on swe-bench ver- ified with claude 3.5 sonnet, 2024. URL
Future Work. Our analyses of Copilot Arena data stress the need to create a diverse set of questions, including multiple written and programming languages, downstream applica- tions, and code structures. Copilot Arena findings highlight the importance of conducting evaluations with real users, tasks, and environments. To extend this platform, future evaluations may also consider building on the Copilot Arena system in multiple ways: more nuanced forms of feedback the programming setting, including measuring trajectories and code persistence metrics, and more forms of interaction, including inline prompt editing and chat dialogue within an IDE. We open-source Copilot Arena to facilitate these future extensions.
https://wwwz anthropic....................ct. https://ww.attrrrgi..coom wwnttroppc..ommreeetsssseeeeeeee com/research swe-bench-sonnet.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07733: 2021.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
8. Conclusion
Barke, S., James, M. B., and Polikarpova, N. Grounded copilot: How programmers interact with codc-generating models. Proceedings of the the ACM on Programming Lan- 2023.
We introduce a platform, Copilot Arena, to evaluate L.LMs in the wild using live human feedback for the use case of coding assistants. Copilot Arena is deployed and has col- lected over 11604 votes across 10 models; will release curated dataset to showcase the diversity of user preferences. We show that evaluating the coding capabilities of LLMs in Copilot Arena leads to rankings that differ from existing approaches which rely on static benchmarks or base- interactions, demonstrating how these differences could be attributed to the shift in distribution between Copilot Arena and prior evaluations. These different contexts also facil-
guages, ((00PSLA1):85-11
Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., and Chen, M. Efficient training of language models to fill the middle. arXiv preprint arXiv:2207.1422 2022.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-34 1952.

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps- Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q. et al. Multipl-e: a scalable and polyglot approach to benchmarking neural code genera- tion. JEEE Transactions on Software Engineering, 2023.
Gong, L.., Wang, S., Elhoushi, M., and Cheung, A. Evalua- tion of Ims syntax-aware code fill-in-the-midee tasks. arXiv preprint arXiv:2403.0444 04841, 2024.
Guo, D., Zhu, Q. Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, K., Luo, F., Xiong, Y., and Liang, W. Deepseek-coded When the large lan- guage model meets programming the rise of code in- telligence, 2024. URL https: http///axvvvvv//// /abs/ 1.01.14190
Chen, M., Tworek, J., Jun, H., Yuan. Q., Pinto, H. P. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint (xXv:21077003744 2021.
Chen, V. Zhu, A., Zhao, S., Mozannar, H., Sontag, D., and Talwalkar, A. Need help? designing proactive ai assis- tants for programming. arXiv preprint arXiv:2410.04596, 2024.
Hutchinson, B., Rostamzadeh, N., Greer, C., Heller, K., and Prabhakaran, V. Evaluation gaps in machine learning practice. In Proceedings of the 2022 ACM conference on fairesss, accountability, and transparency, pp. 1859- 1876, 2022.
Chiang, W.-L.., Zheng. L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonza- lez, J. E., et al. Chatbot arena: An open platform for evaluating Ilms by human preference. arXiv preprint arXiv:2403.04132, 2024.
Izadi, M.. Katzy, J., van Dam, T., Otten, M., Popescu, M., and van Deursen, A. Language models for code completion: A practical evaluation, 2024. URL https: httpss//axxvvvoovo abs /20... 16197.
Chou, C., Dunlap, L.., Mashita, K., Mandal, K., Darrell, T., Stoica, I., Gonzalez, J. E., and Chiang, W.-L. Vi- sionarena: 230k real world user-vim conversations with preference labels. 2024. URL ptatttss//ariiiv abs/2412.0 08687.
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang. S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free eval- uation of large language models for code. arXiv preprint 2:XXv:2403009911 2024a.
Cui, K. Z., Demirer, M., Jaffe, S., Musolff, L., Peng, S., and Salz, T. The productivity effects of generative ai: Evidence from field experiment with github copilot. 2024.
Jain, N., Shetty, M., Zhang, T., Han, K., Sen, K., and Stoica, I. R2e: Turning any github repository into a programming agent environment. In Forty-first International Confer- ence on Machine Learning, 2024b.
Dinh, T., Zhao, J., Tan, S., Negrinho, R., Lausen, L., Zha, S., and Karypis, G. Large language models of code fail at completing code with potential bugs. Advances in Neural Information Processing Systems, 36, 2023.
Jimenez, C. E.,, Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2023.
Dubois, Y., Li, C. X., Taori, R., Zhang. T., Guirajani, I., Ba, J.. Guestrin, C., Liang, P. S., and Hashimoto. T. B. Al- pacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2023.
Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., and Narayanan, A. Ai agents that matter. arXiv preprint arXiv:2407.01502, 2024.
Khan. M. A. M., Bari, M. S., Do, X. L.. Wang, W., Parvez, M. R., and Joty, S. xcodeeval: A large scale multilingual multitask benchmark for code understand- ing, generation, translation and retrieval. arXiv preprint arXiv:2303.0 000000000004....0 2023.
Frick, E., Chen, C., Tennyson, J., Li, T., Chiang, W.- L., Angelopoulos, A. N., and Stoica, I. Prompt-to- leaderboard, 2025. URL
abs/25022.55 2582. 14855.
Fried, D., Aghajanyan, A., Lin, J., Wang. S., Wallace, E., Shi, F,, Zhong, R., tau Yih, W., Zettlemoyer, L., and Lewis, M. Incoder: A generative model for code infilling and synthesis, 2023, URL httpt:ttp/// httpp:://arxiv.xt. abs/2204 05999.
Kirk, H. R., Whitefield, A., Röttger, P., Bean, A. M.. Margatina, K., Mosquera, R., Ciro, J. M., Bartolo, M.. Williams, A., He, H., Vidgen, B., and Hale, S. A. The PRISM alignment dataset: What participatory, represen- tative and individualised human feedback reveals about the subjective and multicultural alignment of large lan- guage models. In The Thirty hight Conference on New- ral Information Processing Systems Datasets and Bench-
Github. Github copilot your ai pair programmer, 2022. URL https: htt///gttthhh/t com/ features/
copilots
10

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
marks Track, 2024. URL
Murali, V., Maddila, C., Ahmad, I., Bolin, M., Cheng, D., Ghorbani, N., Fernandez, R., Nagappan, N., and Rigby, P. C. Ai-assisted code authoring at scale: Fine-tuning, deploying, and mixed methods evaluation. Proceedings of the ACM on Software Engineering, 1(FSE):106611085 2024.
net/forund d=DFrShteojxx
Lee, M., Srivastava, M., Hardy, A., Thickstun, J., Durmus, E., Paranjape, A., Gerard-Ursin, I., Li, X. L.,, Ladhak, F,, Rong, F., Wang, R. E., Kwon, M., Park, J. S., Cao, H., Lee, T. Bommasani, R., Bernstein, M. S., and Liang. P. Evaluating human-language model interaction. Transac- tions on Machine Learning Research, 2023. ISSN 2835- 8856. URL https: htap::://opeeee http://pppeeeeeeeeeeeeeeeeeeeeet net forum? id=h
Nijkamp, E., Pang. B., Hayashi, H., Tu, L.,, Wang, H., Zhou, Y., Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi-turn pro- gram synthesis. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: / /oopneeeeew.eccccm net/ wtt.//frrmmmiio// wweeiiidddiiiaieee -d-iiaccKKYY
Lin, B. Y., Deng, Y., Chandu, K., Brahman, F., Ravichander, A., Pyatkin, V., Dziri, N., Bras, R. I.., and Choi, Y. Wild- bench: Benchmarking Iims with challenging tasks from real users in the wild, arXiv preprint arXiv:2406.0477 2024.
Oberst, M., Liang, D., and Lipton, Z. C. The science of AI evaluation at Abridge. https: //www//wwwwwww/ http://www.ab.bbadec abridge. com hom/ai/sciienen
ai-evaluation. 2024.
Liu, J., Xia, C. S., Wang, Y., and Zhang, I.. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2023.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L.,, Zhou, L., Tufano, M., GONG, M.. Zhou, M., Duan, N., Sundaresan, N., Deng, S. K., Fu, S., and LIU, CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. In Thirty-fifth Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https: http//ppnevee.w.e.eeeee id= 6 E4dgxaUcb.
Peng, Q. Chai, Y., and Li, X. Humaneval-xl: A multilingual code generation benchmark for cross-lingual natural lan- guage generalization. arXiv preprint arXiv:2402. 16694, 2024.
Peng. S., Kalliamvakou, E.,, Cihon, P., and Demirer, M. The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint arXiv:2302.0655 2023.
Qwen, Yangg, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang. J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q. Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang. Z., and Qiu, Z. Qwen2.5 technical report, 2025. https: warxiv..org/aaee 251.bb.s...2....5.....
Lu, Y. Jiang, D., Chen, W., Wang, W. Y., Choi, Y., and Lin, B. Y. Wildvision: Evaluating vision-language mod- els in the wild with human preferences. arXiv preprint arXiv:2406. 1069, 2024.
Mozannar, H., Bansal, G., Fourney, A., and Horvitz, E. Reading between the lines: Modeling user behavior and costs in ai- assisted programming. In Proceedings of the CHI Conference on Human Factors in Computing Sys- tems, pp. 1-16, 2024a.
Ross, S. I., Martinez, F., Houde, S., Muller, M., and Weisz, J. D. The programmer's assistant: Conversational in- teraction with large language model for software de- velopment. In Proceedings of the 28th International Conference on Intelligent User Interfaces, pp. 491-514, 2023.
Mozannar, H., Chen, V., Alsobay, M., Das, S., Zhao, S., Wei, D., Nagireddy, M., Sattigeri, P., Talwalkar, A., and Sontag. D. The realhumaneval: Evaluating large language models' abilities to support programmers. arXiv preprint arXiv:240.002886, 2024b.
Muennighoff, N., Liu, Q. Zebaze, A. R., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., Von Werra, L., and Longpre, S. Octopack: Instruction tuning code large lan- guage models. In The Twelfth International Conference on Learning Representations, 2023.
Saxon, M., Holtzman, A., West, P., Wang, W. Y., and Saphra, N. Benchmarks as microscopes: A call for model metrology. In First Conference on Language Modeling, 2024. URL https: https:///ppeeeeee oppereviewwweeeeeeeeeeeeee net forum? id=bttKwCZDDmm

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Scheurer, J., Campos, J. A., Chan, S., Chen, A., Cho, K., and Perez, E. Training language models with language feedback. arXiv preprint arXiv:2204. 14146, 2022.
component in legal document review. In 2024 IEEE In- ternational Conference Big Data (BigData), pp- 4779- 4788. IEEE, 2024a.
Singhal, P., Goyal, T., Xu, J., and Durrett, G. A long way to go: Investigating length correlations in rlhf. arXiv 2023.
Yang. J., Prabhakar, A., Narasimhan, K., and Yao, S. Inter- code: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36, 2024b.
preprint arXiv:2310.077
Stahl, P. M. Lingua: The most accurate natural language detection library for Java, Kotlin, Rust, Swift, Python and Go. https: /ttgg/gttub/.////// com/ www/pommsstttteeeeecoeeeeee.oooe pemistahl/ lingua, 2024.
Zheng. I.., Chiang, W.-L.., Sheng, Y., Zhuang, S., Wu, Z.,, Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https:
Team, G., Georgiev, P., Lei, V. I,, Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., Mariooryad, S., Ding, Y., Geng, X., Alcober, F., Frostig, R., and and, M. O. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL https:////// arxiv. org/ abs/ 2403. 05530.
/ /openreview net/ forummidduccc
Zhu, M., Jain, A., Suresh, K., Ravindran, R., Tipimeni, S., and Reddy, C. K. Xlcost: A benchmark dataset for cross-lingual code intelligence, 2022. URL https:// arxiv. wwxiv.org///vo.o. abs//////////// 2206 08474.
Vaithilingam, P., Zhang, T., and Glassman, E. L. Expec- tation vs. experience: Evaluating the usability of code generation tools powered by large language models. In CHI Conference on Human Factors in Computing Sys- tems Extended Abstracts, pp. 1-7, 2022.
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I,, Brun- ner, S., Gong, C., Hoang. T., Zebaze, A. R., Hong, X., Li, W.-D., Kaddour, J., Xu, M., Zhang, Z., Ya- dav, P., Jain, N., Gu, A., Cheng, Z., Liu, J., Liu, Q.. Wang, Z.,, Lo, D., Hui, B., Muennighoff, N., Fried, D., Du X., de Vries, H., and Werra, L. V. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions, 2024. URL https:
Wang, S., Li, Z., Qian, H., Yang. C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P., et al. Recode: Robustness evaluation of code generation models. In Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13818-1388 13843, 2023.
Wei, J., Bosma, M., Zhao, V. Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned lan- guage models are zero-shot learners. In International Conference on Learning Representations, 2022.
White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K. Naidu, S., et al. Livebench: A challenging, contamination-free Ilm benchmark. arXiv preprint arXiv:2406.19334 2024.
Wu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P., Smith, N. A., Ostendorf, M., and Hajishirzi, H. Fine- grained human feedback gives better rewards for language model training. Advances in Neural Information Process- ing Systems, 36:59008 590333, 2023.
Yan, W., Liu, H., Wang, Y., Li, Y., Chen, Q. Wang, W., Lin, T., Zhao, W., Zhu, L., Deng, S., et al. Codescope: An execution-basee based multilingual multitask multidimensional benchmark for evaluating Ilms on code understanding and generation. arXiv preprint arXiv:2311. 211.:28881...8888. 2023.
Yang, E., Omrani, R., Curtin, E.,, Emory, T., Gray, L., Pick- ens, J., Reff, N., Traylor, C., Underwood, S., Lewis, D. D., et al. Beyond the bar: Generative ai as transformative
12

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
A. Additional System Details
describe further implementation details and considerations for each the three key system components: user interface, model routing, and model prompting
A.1. User Experience
We make several additional design decisions surrounding our user interface.
We cache generated completions. If the user continues typing, we try to retrieve a matching pair of completions.
We set a 0.5 second delay before automatically generating completions.
If two completions are identical, then we return only one copy.
If one model returns empty string, then we only display the other, non-empty completion.
We limit the number of lines each completion can generate (default of 20), but allow users to customize this limit.
After the user votes, reveal the model pair and their choice to the user. We also show the user history of their votes.
We limit the input file size to be 8,000 tokens, which covers nearly all user file lengths.
A.2. Prompting Diverse Models
To enable the evaluation of a diverse set of models, we devise prompting scheme to facilitate code completion functionality of models that were not necessarily trained to "fil-in-tthe-midllee Note that we focus on prompting based approaches and do not fine-tune any LLMs, as many models we evaluate are commercial, closed models.
Prompt templates. The following are an overview of prompt templates from Gong et al. (2024) that we experimented 3
1. Prefix-Suffix-Middee (PSM). PSM presents the code context in the order of prefix and then suffix, using XML notation to demarcate prefix, suffix, and middle segments (e.g.. <PREFIX> and </PREFIX>). The LLM is then asked to output the middle segment given the prompt.
2. Suffi--rrefx-Middle (SPM). SPM is identical PSM except that the suffix appears before the prefix, which may be more natural than having the suffix appear directly before the output as is the case with PSM.
Mask. Rather than using start and end tokens denote the prefix and suffix, the Mask prompt uses a special "sentinel" token to indicate the masked (i.e. middle) code segment (Guo al., 2024). The LLM is then requested fill in the masked code segment.
Instructed Prefix Feeding (IPF). IPF begins with the Mask prompt and then repeats the prefix as a "prefill" of the completion for the language model.* This is similar IPF in Guo al. (2024), except with instructions adjusted better align with chat models. This approach allows non-FiM- FiM-trained models the ability to better tackle FiM tasks (Fried et al., 2023).
'Full prompt templates are provided in our code: httpps://g com 1oar/mmaeeaaa crope/coppiiiiiii lootoreenn
'Nowadays, many completion APIs deprecated; however, many chat APIs provide the ability "pre-fill" tokens in the response which is similar to forcing the L.LM to do a completion
13

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
A.2.1. PSM EXAMPLE
Fill in code and output nothing clse. Respect spacing, new lines, and indentation. Start with <CODE> and end with </CODE>. Be VERY mindful of indentation. Make sure it is correct.
Example
<PREFIX>class Calculator {(
add (number) II
thistresult +<</PREIX>>>>>
<SUFFIX> subtract (number) ((
reistt number;
return this;
}}
}]</SUFFIX>
<CODE> number;
return this;
1></CODE>
example
<PREF <PREFIX> X>frrm typing import List
def cas_elose elements (numbers: List threshold: float) -> bool: Check if in given list numbers, are any two numbers closer to each other than
given threshold.
>>> has_cllse_eelments ([1.0, 2.0, 3.0), 0.5)
False
>>> has
([1.0, 2.8, 3.0, 4.0, 5.0, 2.0), 0.3)
True
for idx, elem in enumerate (numbers) for idx2, elem2 eau</PREFIX>
<BUFFIX> idx2:
distance abs (elem elem2) if distance threshold:
return True
return False</SJFFIX> <CodE>merato (numbers): if idx4/CODE>
Task: <PREFIX> (p/RF <SUFFIX> (/SJFF
14

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
A.2.2. EVALUATION RESULTS
On these various templates, we evaluate models on the HumanEEvalinfilling dataset (Bavarian al., 2022), which consists of 1640 examples where random spans in completed portion of code masked out to simulate FiM behavior.
Table pass@ code completions with different prompt templates (PSM, SPM, Mask). We observe that for all models and most
prompt templates, our Snip-It methed improves pass@1. 1.
Group Model
psm
spm
mask
ipf
snip-psm
snip-spm
snip snip.mask
snip-ipf
Open Code
Deepseek-Coder- V2.5 Qwen-2.5-32B
0.551
0.519
0.414
0.2299
0.585
0.584
0.597
0.614
0.169
0.065
0.113
0.005
0.563
0.611
0.534
0.521
Open
Alama-3.1-40555
11-40B--nstruet-Tubbo
0.254
0.224
0.145
0.038
0.553
0.583
0.531
0.463
3.1-70B-
0.074
0.079
0.061
0.029
0.490
0.527
0.312
0.323
Commercial
Gemini-1_5-Proon
0.620
0.599
0.562
0.338
0.561
0.659
0.259
0,,991
GPT-4o
0.607
0.477
0.505
0.033
0.620
0.670
0.609
0.524
Claude-3.5-Sonin
0.561
0.565
0.552
0.374
0.730
0.710
0.705
0.507
Gemini- 1.5-lii-,--FFash-000
0.434
0.376
0.277
0.286
0.409
0,403
0.301
0.394
GPPT-40mini
0.099
0.055
0.088
0.019
0.429
0.480
0.361
0.342
A.2.3. ERRORS WITHOUT SNIP-IT
Below examples of errors without Snip-It from GPT-4o mini. Red indicates the incorrect code that the model filled in.
from typing import List
def belox_zere (operations: [ist(int]) -> bool:
un. You' given list of deposit and withdrawal operations on bank account any point the balance of account fallls below zero, and at that point function should return True. Otherwise it should return False. >>> below_ (11, 2, 31) False >>> belov_zero 2, -4, 5]) True balance balance op
that starts with zero balance. Your task is to detect
if balance 0:
balance += op
if balance 0:
True return False
from typing import List
def parse_nested__are (paren_string str) -> [ist]]nt]: а*. Input to this function is string represented multiple groups for nested parentheses separated by spaces. For each of the group, output the deepest level of nesting parentheses. E.g. (() ()) has maximum two levels of nesting while ((O)) has three. >>> parse nested parens (1 (0 () ((0)) 0 ((0)0 12, 3, 1, 3]
def harse_paren_groulsse (s)
depth +=
max.depth max (max.depth,, depth) if char ')':
B. User information
Below we provide copy of the general instructions and privacy instructions for users.
15

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Table conserve space, we refer models by shortened name in the main text provide full model name below completeness.
Full Model Name
Shortened Name
deepseck-coder-fing
dcepscek-coder
claude-3.5-sonnet
codestral-2405
codestral
1aima-3.-4455b-nnsmee
llama33.14455b
gemini-1.5-flash-000
gemini-flash-002
gemini-pro-002
spt-40-2024408-066
gpt-4o-2024-08-066
llam--3.1-70b-isstnutt
Gama-3.1-706
qvenn-2.5-ooee--32biinntue-
qwen-2.5-coder-32.
gpi-4--mini-2224-07-11 gpt-4o-mini
B.1. General instructions
Step 1: Install the extension and restart Visual Studio Code after installation. If installed successfully, you will see Copilot Arena show up on the bottom right corner of your window and the check mark changes to a spinning circle when a completion being generated, Note, if you are using any other completion provider (e.g. Github Copilot), you must disable them when using Copilot Arena.
Step 2: Copilot Arena currently supports two main feature: read autocomplete and in-line editing (beta) below to understand how to use each one. Since we show paired responses, the way you use them are slightly different than your standard AI coding tools!
Step 3: This step is optional. If applicable, you can change what data is saved by Copilot Arena by following the instructions in "Privacy Settings".
Step 4: Create username by clicking the Copilot Arena icon the sidebar; detailed instructions also in "Create account". Your username will be used for future leaderboard to compare individual preferences.
B.2. Privacy Instructions
Privacy Settings. Your privacy is important to us. Please read carefully to determine which settings are most appropriate for you. To generate completions, the code in your current file is sent our servers and sent to various API providers. This cannot be changed.
By default, we collect your code for research purposes. You can opt-out of this. If you are working on code containing sensitive information, we recommend that you opt out of data collection. To opt-out of data collection, please change SedeeriiacySeettings to Debug. We will only log your code for debugging. To disable logging entirely, please change SedterisacySettinns to Private. Opting-out means any bugs you encounter will be reprodeuoeee on our end. You find these settings by searching for Copilot Arena in your vscode settings clicking the gear button of the Copilot Arena extension Extension Settings.
Data Collection.
Removing your data. If you would like to have the option in the future for us to delete any of your data, you must create an account on Copilot Arena following instructions described in "Create an account." To remove your data, you can email any of the Copilot Arena maintainers with your username.
Data Release. Prior to releasing any collected code snippets to enable future research efforts, we will run PII detector and remove any identified entities further ensure no personal information is released.
C. Data Analysis
Natural Language Detection. To detect natural languages, we used the lingua language detector (Stahl, 2024). We set the detector to all available languages (except for Latin due to false positives), and picked the language with the highest confidence that was greater than 0.7. Additionally, we filtered for languages that appeared at least 5 times. Results are
16

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
in Figure 8. For Table since Chatbot Arena does not track natural languages, we the same detection algorithm for Chatbot Arena.
Distribution of Languages in Research Votes
4081
Narbor Vo
Riglish
Roralan
Chioeet
Speutds
Garato
Voralee
Frinch
Palich
Esparanto
Tagaloy
Ukrainlen
Terkles
Catalan
Humgurtin
Koman
Portagurtae
Language
Figure 8. Natural languages in Copilot Arena
Programming Language Detection. We detect programming languages in Copilot Arena by using the file's extension type (Figure 9). For Table 1, since Chatbot Arena does not track programming languages, we checked for the language of codeblocks instead.
Distribution of Top 10 File Types
5000
Number of Filos
4000
3000
13 12
Govad Pribos Hinaiimadaaan mankalonn
plaistest
lnters
Bi
paun
powerrsbell
diange groperr Irmtur spore
aadeco
Oppnc
Jocker
Type
Figure Programming languages in Copilot Arena. For image clarity, only show programming languages that appear more than 10 times.
17

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Task detection, Since Copilot Arena code contexts fairly long, we employ multi-step process cluster code contexts, via LLM--as-a-judee (Zheng al., 2023). We specifically use GPT-4o-min n due to its speed and price.
First, we summarize all code contexts into short one-sentence descriptions.
System Prompt
You are a helpful assistant that describes code files single, concise sentence. Focus on the main purpose and functionality of the code. Keep descriptions clear, technical, and under 100 characters. not mention file names extensions in your description.
General Prompt
Describe this code in one sentence
Next, we prompt model to cluster all one-sentence descriptions.
General Prompt
You are code organization expert.
Analyze the provided code descriptions and:
1. Identify 5-10 main functional clusters or themes
2. Assign each description the most appropriate cluster Provide brief name and description for each cluster
3.
4. Format the response valid JSON with the following structure:
"clusters";
"name": "cluster_names
"description": "brief cluster description", "descriptions": ["description", "description2")
Finally, we provide the full code context and ask the LLM to categorize the context given aforementioned clusters. Note that we sanity-checked clusters manually and removed redundant ones.
18

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
System Prompt
Please categorize the following code into of these categories:
User Interaction and Input Handling: Code that manages user inputs, prompts, and basic interaction with the system
Frontend Development and UI Design: Code snippets focused on designing user interfaces and creating interactive components.
Backend Development and APIs: Server-side logic, data management, and API integration for applications.
Algorithm Design and Problem Solving: Code implementing algorithms to solve computational problems or optimize tasks.
Data Processing and File Operations: Code that reads, writes, or processes data from files and other data sources.
Game Development and Simulations: Code focused on creating games, simulations, and managing game dynamics.
Artificial Intelligence and Machine Learning: Code related Al and machine learning for training, inference, and application.
General Prompt
Only respond with the exact category name that best fits. No other text.
Here's the code:
[code content]
Model Votes. We ensured that our lcaderboard has coverage across all models, where each model received at least 2,000 votes, as shown in Figure 10.
Codestral
4,565
Qwen Coder
Gemini-1.5 Flash
3,853
GPT-4o mini
Language Models
Llama-3.1 70B
3,792
Llama-3.1 405B
Claude 3.5 Sonnet
2,707
Gemini-1.5 Pro
Deepseek V2.5
2,296
GPT-4o
2,111
1,000
2,000
3,000
4,000
5,000
Number of Votes
Figure 10. Number of votes for cach language model.
19

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
750
500
Totals
250
Oct 21
Nov 4
Nov 18
Dec 2
Dec 16
Dec 30
Jan 13
Jan 27
Figure 11. Number of votes over time.
Code Structure Detection. To detect the presence FiM, we check there exists suffix. If there is only the prefix, we label "completion-only" To detect if there are comments, we check if any of the previous lines start with common comment styles (e.g. # / /). We check for block comments in similar fashion using docstring styles (e.g. """
/** """,/** +).
Completion Bias. Users selected the first completion 86% of the time, revealing completion order bias. We investigated whether the bias was due to users instinctively pressing Tab for the first completion, as it requires simpler keystroke than Shift-Tab. Analysis of decision times revealed that users spent a median of seconds selecting the first completion, indicating this action was not automatic. However, users still took longer seconds) to select the second completion, suggesting they deliberated more between the two options.
First Completion
Second Completion
Decision Time (seconds)
0.5
Completion Similarity
Figure Completion similarity vs. decision time, grouped by selection the first or second code completion.
We hypothesized that the extended deliheration resulted from users comparing completion differences. To validate this, we evaluated code similarity between completion pairs using the Levenshtein ratio. The dataset was refined by removing outliers (identical or very dissimilar completions) and excluding comments minimize the impact of documentation differences. As shown in Figure 12, decision time increased with completion similarity for the second completion, indicating greater deliberation for highly similar completions. This trend was absent for the first completion, suggesting this extra deliberation did not occur for these cases.
Cost. We estimate that the total cost to collect our dataset approximately $5k USD.
C.1. Example Data
We provide examples of code contexts from each of the task categories. For readability, we select examples with shorter context lengths. Upon publication, we will also open-source more diverse examples (including those with significantly longer context lengths).
20

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Artificial Intelligence and Machine Learning
from main13 import knn, mip import pandas as pd
for pelass in [1, 2, 3]:
for fare in range (10, 200, :0):
my of pd. rata (f
"Pclass": [pclass] +3,
"Name": *34]*3,
"Sex": [0]*3,
"Age": [19]+3,
"sibSp": [0]*3,
"Parch": x3]*3,
"Fare": [fare]+3,
"Embarked"; ["s", "Q", "C"]
})
my_df pd.get_dummies (my_df,, (ollumms=("ESaaareee"" prefix==Embarked""
my_df "Embarked__s" my_df ""Ebbaredd_"") astype (int)
astype
my_df "Embarked_c"i my_df ["Embarked_c" [*] astype (int)
my df "Embarked_0") my df ""Embarked_0 0*) astype (int)
predictions =
"knn": knn. predict (my_df),
"mlp": mlp.predict (my_df)
ans_df DataFrar (index:(fare), columns= [1, 2, 3])
ans df at [fare, polass] predictions
print (ans_df)
Frontend Development and UI Design
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="OTF-$""
<meta nyie=oyye
ihitial-scalei..01
<titleeboccment</ttidlee
</head>
<body>
<script>
function getRandomNumber (min, max)
return Math. floor (Math. random () *)
</script>
</body>
</html>
21

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Algorithm Design and Problem Solving
import jtil.util.x;
public class hastmappipleemennaation
static class HashCode<K, V>1 //generics -> can any data type for
key and value.
private class Node
K key;
v value;
public Node (K key, value) {
this.key key;
chis.value value;
}
private int size; //n private LineedList-Node> buckets(); //N Jackets.lengee array of linkedlists
esuppresswarnings ( ("unchecked") public HashCode ()
this. = 0;
chis.buckets new LinkedList [4];
for (int i 0; buckets [i] new LinkedList<> ();
buckets.lengthy i+t) {
public void put (K key, V value) {
public static void main (String args (])
22

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Data Processing and File Operations
import os
from caa.inecchiin_fnntiiooo import path name " httppppelineeegnereeeeeeeeeeedee _
def upload_datte (image_path);
# iimage_pthh = os.listdir
[0]
full_path iasiiannamemmmaeeeeeeee
element
(full (aatt path)
def uplooad_igg___isonl Write data to JSON file fich_hhle_path h///mn_jionnsstorrdd//taaagggap json_file_path sson_fiieepathh.eepaaaee with open
jsoo")
'w') as file:
json. dump ("result"; element), file, indent=4)
print (f"Data successfully uploaded to (json file file_path)""
User Interaction and Input Handling
print ("Hello World") hamevar input ("Enter name ")
print ("Wel come namevar)
#Write python code to download and run deepseek model locally in my windows computer. I have python and pytorch installed in my computer.
#To download and run a DeepSeek model locally on your Windows computer, you can follow these steps:
23

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Game Development and Simulations
using Collectioos Ceneric;
using System.Collect
using UnityEngine;
public class Player MonoBehaviour
// Start is called before the first frame update public float speed 1f;
void Start
transform.posil new Vector3 (0, 0)
// Update called once per frame void (pdate
transform. Translate (Vector3. left speed Time. deltaTine )
Backend Development and APIs
async def discover_device (emp_user_no, access_token, repositories)
print (fdddccveeeeerddeeicc"e
async def check_device_.o health (request_type, payload, mesage_id id, rmp_user_no, repositories) print (f""checkdevvii whrechcc_deiicchhellttt
async def
D. Details on Model Ranking
Computing BT Coefficients. We estimate 3 by running a logistic regression:
i arg min BERM
Σ (XTT)
where CE represents the cross-entropy loss and is the sigmoid function. We use the sklearn package with penalty and no intercept term. We bootstrap the ranking calculation by sampling with replacement for 100 rounds to compute the 95% confidence interval. In our leaderboard, we use codestral an anchor model.
BT Ablations. Since confounding variables (e.g., length of the response or other stylistic formatting (Singhal et al., 2023)) may influence preference judgments, we also control for these variables in the BT model. Given a set of style features, which include model latency and completion length, we add style vector to the BT model Ž where= Z, E R a vector of S style features comprising the normalized difference between the feature values of both model responses. The extended
24

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
BT model includes the style coefficients 7 R$ and be written as:
8.9 arg min BERM MERS
where CE represents the cross-entropy loss and the sigmoid function. The resulting 3 represents model strengths adjusted for style effects, while quantifies the influence of style user preferences. |3 values used to create the ordered ranking of models on the leaderboard.
Since we observe completion bias (described in Appendix C), we perform an additional experiment to verify that we collected sufficient votes across all pairs. To do so, we upsampled and downsampled model pairs that had fewer and more comparisons, respectively, so that the difference between the most and least voted pair is within 10%. We find that the rankings remain identical to the ranking obtained using the full set of votes. This provides additional confidence in our reported leaderboard.
When comparing the original leaderboard (Table and the style-controlled version (Table 5), minimal changes to the overall "tiers" described in the main text. While observe some changes in the middle tier (e.g., Llama-3. 1-405b, Gemini-1.5-PP 1.5.ii--.--Fahh,, and Gemini-1.5 1.5-Pro swap places as well as GPT-4o and Llama-3.1-706 we do not observe significant changes between tiers.
Table B, values for each model bootstraped over 100 samples: their lower, rating, and upper bounds.
Model
Lower bound
3 estimate
Upper bound
deepscek-coder-fim
0.04
0.07
0.10
0.02
0,06
0.09
codestral-2405
-0.02
0.00
0.02
llama-3.14405b-instmee
-0.07
-0.04
-0.01
gemini-1.5-flash-0
-0.06
-0.04
-0.01
-0.08
-0.05
-0.02
gp1-4o-2022408-066
-0.09
-0.06
-0.03
llama-3. 1--ma-3.-77b--instrruut
-0.10
-0.07
-0.04
(wenn225-cooerr332-iisttttt
-0.16
-0.13
-0.10
gpL-4o-mmini-2024-07-11
-0.19
-0.15
-0.12
Table 5. {8 and 7 values for each model bootstraped over 100 samples: their lower, rating, and upper bounds.
Model
Lower
Rating
Upper
deepseek-coder-hng
0.05
0.08
0.11
chaube3-5-somne--2240022
0.03
0.06
0.09
codestral-2405
-0.02
-0.00
0.03
gemini-1.5-flash-00
-0.06
-0.03
-0.01
1.imaa33--40Sb-insttuuct
-0.07
-0.04
-0.00
gemini-1.5-pro-002
-0.09
-0.05
-0.01
-0.09
-0.06
-0.03
gpt-4o-2024-08-06
-0.09
-0.07
-0.04
(wen=22.--colrr-2-iinstttt
-0.16
-0.13
-0.10
spI-4o-mini-20244-77111
-0.18
-0.15
-0.12
Model
Lower bound
? estimate
Upper bound
Model latency
-0.33
-0.17
0.00
Response length
0.11
0.21
0.32
E. Additional Results
We provide additional details about win-rate analysis in Figure 13, 14, 15, and 16. We also showcase an experiment testing our prompting approach Table
25

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Model Loser
Claudo-3 3-5 S-vonnet 20240420
green cordon 32th inseruct
Uespeeeel coder fim
Bame- 1-05-- instrrr
gomine-11 flash 002
gemune-11 prrprr 002
Hat 2024 06
Itema-3. instrvec
gpt Bpt-4b 1os----------aa- 2024 477-18
0.3
codestral-2ee 2405
ceopseck-co code--odrr--eee film
-0.22
-0.04
0.07
-0.27
-0.10
-0.13
-0.01
0.03
-0.29
0.2
clrade 3-5 202200--022002200
0.22
0.06
0.29
0.24
-0.15
0.32
0.14
0.18
0.06
cocestral-2405
0.09
0.02
-0.03
0.1
-0.24
Model A: Winner
S-ppo-
2024--2222-----0 10024-08-00
-0.02
-2.00
-0.1
Home--.. 1-770b-mnetucch
0.01
-0.14
0.06
0.19
0.38
-0.01
0.04
0.31
0.21
speen- 2.5 sorder 22-iirntuuct
-0.03
-0.15
0.09
0.05
0.00
0.09
-0.17
-0.31
0.10
0.07
-02
got for 202---222-07-11-
0.29
-0.05
-0.21
-0.24
-0.05
-0.09
-0.04
-0.21
-0.07
-0.3
Difference Win Rates Based On Task
Figure 13. Win-rate difference based on Task: frontend/backend versus algorithmic design problems.
Table A controlled experiment with fixed model, DeepSeek Coder, where we vary whether we use the model's FiM capability or we Snip-It to post-process the model would with other models that do not have native FiM capability. While were not able still observe that & estimates are comparable between the
obtain a significant number of votes before deepseek-codee coder was deprecated, variants. This shows that our Snip-It approach can roughly recover FiM capabilities.
Model
Lower bound 3 estimate
Upper bound
deepseck-coder-ffin deepseek-coder
0.04
0.07
0.10
-0.04
0.06
0.15
26

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Model B: Laser
Cloude- S-tonez 301206620
Gmnen-2 5-ctater instrree
deoppeeee custer
cadentrak 2405
Ramin-3 4-05 Instrrctt
gentini-1 S-flash--e 002
gemini- 1.. 002
and App 2014 os
Home 3, 1-100 instreet
git mini- 2024 2024-07-1 07-18
despsocks coder fim -o
0.08
0.03
0.00
0.15
10.10
0.19
0.16
0.10
0.03
0.2
cauude S-oute-----aae 5--mmmm--307400000
-0.08
0.03
-0.04
0.11
-0.09
0.19
-0.16
-0.03
-0.06
cocestral-2405 -0.03
-0.05
0.05
0.00
0.02
-0.01
0.08
-0.01
Tama-a 3.1 1-405---imiinne instrrct
-9-9
-0.01
-0.13
0.05
Model Winner
gemini-L.S-iash-- S.5-iash--02
-0.17
-0.08
gamini-1.5
pot-4o-2024-08-0
-0.1
3.1-700binntrucc
-0.18
-0.20
-0.01
qwen-22 S-oodeoddddro www.--ooo--2---eieee wondder--b--nnttrure
-0.30
0.03
-0.15
-0.05
0.16
-0.16
-0.09
-0.03
-0.2
gpt 40 mini 2024-07-18 -0.03
0.06
0.01
0.10
0.00
0.07
0.03
-0.17
-0.06
-0.3
Difference Win Rates Based
Figure 14. Win-rate difference based on FiM: whether the task is FiM or not.
27

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Model B: Laser
Cloude- f-tonee 381246620
Gmeen-2 5-ctater instrred
deoppeeee cutter
Ramin-3 4-05 Instract
gentini11 S-flash--e 902
gemini- 1.. D02
snd Atp 2014 os
ilmms 3, 1-700 instreet
git mini- 2024 2024-07-1 07-18
cadentrak 2405
despsocks coder fim -osid
0.07
-0.03
0.26
0.00
0.04
0.13
0.07
0.17
0.12
0.2
cauude Sesdse------aae 3---0mmm--307400000
-0.07
-0.03
-0.05
-0.01
-0.03
0.12
-0.23
0.13
-0.12
cocestral-2405
0.03
0.03
0.30
0.04
-0.39
0.00
0.15
0.21
-0.08
Tama-3. 3.1 1-405---imitnee insirrctt
-0-96
-0.02
-0.12
0:08
Model Winner
gemini-L.S-iaah--0 O.5-iash-0022
-0.04
0.12
-0.04
gamini-1.Sppp
ppt-4o-2024-08-00
-0.1
3.1-700-instrucc
-0.09
-0.02
-0.03
0.00
qwen-22 S-.-S-odderrr 2.w.-/oddr------eeeee Innettrcc
-0.17
-0.13
-0.21
-0.04
-0.09
-0.36
-0.07
-0.05
-0.05
-0.2
gpt 40 mini 2024-07-18
-0.12
0.12
0.08
-0.05
0.01
0.02
0.02
-0.01
0.05
-0.3
Difference
Rates Based On Context Len
Figure 15, Win-rate difference based on context length: context length in top versus bottom 20 percentile.
28

---

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Model B: Laser
Cloude- f-tonee 381206200
Gmnen-2 5-ctater instrred
deoppeeee cutter
cadentral 2405
Ramin-3 3.1 4-05 Instrrct
gentini11 S-flash--e 002
gemini- 1...--ppe 002
Bast Atp 2024 0s
Home- 3, 1-100 instreet
git mini 2024 2024-07-11 07-18
despsocks coder fim -oaai
.0.05
-0.04
0.07
.0.03
00.10
0.10
-0.05
-0.12
0.03
0.2
cauude Soude------aae 5-55mmm--307400000
0.05
0.05
0.00
0.01
-0.01
0.04
-0.07
0.17
-0.3
cocestral-2405
0.04
-0.05
0.02
-0.06
0.03
-0.05
-0.02
0.07
0.04
Tuma- 3.1 1-405---imiinnn insirrctt
-0-96-
-0.01
-0.09
0.02
Model Winner
gemint-L.S--aa S-5-iash-002
0.00
0.07
gamini-1.5
0.04
pot-4o-2024-08-0
-0.1
3.1-700-instrucc
-0.05
-0.05
-0.04
-0.05
qwen-22 S-.-S-ooder-- 2...-ooder-3-2-eeeeeeeee Inn-nttcc
6.12
-0.17
-0.07
0.02
0.01
-0.03
-0.18
0.00
-0.2
gpt 4o mini 2024-07-18 -0.03
0.03
-0.04
0.13
0.05
0.24
0.11
0.08
-0.04
-0.3
Difference Rates Based On
Figure 16. Win-rate difference based on programming language (PL): Non- ton-python code versus Python code.
29


Reviews:
[{"review_id": "ARIK2F3ho1", "reviewer": "Reviewer_tbZQ", "summary": "Authors introduce EvalX a platform to compare the effectiveness of different LLMs for the use case of coding assistants. Their deployed platform has already collected over 11000 responses on comparisons between 10 different models. The model ranking presented from these results gives new insights on user preferences under different tasks that differ from the previous benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and appreciation of our work. We aim to address your comments below:\n\n*\\[Claim \\+ Evidence 1: focus on code completions\\]*\n\n* We agree that the paper already provides value to the research community and will be more precise about the scope of our work early on (e.g., we will revise L37 to state “... coding assistants, **specifically focusing on their ability to generate code completions**”).\n* While there are many ways to interact with AI programming assistants, code completions are one of the most frequent use cases, as found by [https://dl.acm.org/doi/10.1145/3597503.3608128](https://dl.acm.org/doi/10.1145/3597503.3608128).\n* Further, EvalX naturally leads to many interesting directions for research for the broader AI and software engineering community, which we discuss in Section 7\\. For example, EvalX can be extended to include more interaction modes beyond code completion. Since our submission, we have already added a prompt-based editing feature. The reviewer’s suggestion of long-term impact is another direction that can be studied using data collected from EvalX.\n\n*\\[Claim \\+ Evidence 2: preferences can be biased, compared to static benchmarks which can be nuanced\\]*\n\n* We agree, and believe our work is *complementary* to existing approaches and does not seek to replace them—both are necessary for a holistic view of LLM code evaluation. We discuss this in Related Work (L417), but will update the Introduction to be clearer.\n* We also speculate that building evaluations in the IDE may mitigate the reviewer’s concern for biases towards readability. Since our platform is embedded in a user’s **real development environment**, users are likely to prefer “useful” generations (readability being only one aspect of usefulness) as they would when writing code with Github Copilot.\n\n*\\[Experimental design 1: Does the model distribution (skewed towards latency) impact the quality of data? Impact of $\\tau$?\\]*\n\n* In the analysis conducted in the original submission, we took a few steps to mitigate this impact.\n  * First, when computing Bradley-Terry coefficients, we use an L2 regularization term to prevent overfitting and bias towards models that have received more votes.\n  * Second, we conduct a statistical bootstrap of the leaderboard so we can obtain confidence intervals around the estimated coefficients, improving reliability of our results.\n  * Finally, we ensure that we have reasonable coverage across all model pairs (e.g., $>150$ votes per pair), so votes are not too sparse across some pairs.\n* Inspired by the author's suggestion, we did additional simulations using our data. We upsampled and downsampled model pairs that had fewer and more comparisons respectively so that the difference between the most and least voted pair is within 10%. We find that the rankings remain **identical** to those we report in the original submission. This provides additional confidence that we collected sufficient votes across all pairs.\n* As $\\tau$ increases, our coverage becomes more evenly distributed (uniform at $\\tau \\rightarrow \\infty$).\n\n*\\[Suggestion 1: More detail about other benchmarks\\]*\n\nWhile we briefly cover metrics for each benchmark in Table 1, we will additionally provide a short summary of each benchmark in Section 4.2.\n\n*\\[Suggestion 2: “insights/takeaways for model training or prompting\\]*\n\nSince the final version allows an additional page, we will add a new Section 5.3, which will explicitly summarize the insights and takeaways for researchers building new coding assistants. Due to character limits in the response, please see our first response to Reviewer 3 (**1UGk**) for additional details. Additionally, if the reviewer has any suggestions, we would be more than happy to incorporate them\\!\n\n*Questions* (not repeated due to space):\n\n* We believe there are three main motivations: 1\\) free generations, 2\\) overall interest in participating in ML research, and 3\\) GitHub Copilot or alternatives may be unavailable in some countries.\n* Models are randomly selected (as discussed in L149).\n* Fidelity refers to the quality of the response, largely in terms of formatting. We will clarify the caption.\n* We observe this in our own data\\! Figure 6 shows that 65% of votes are on FiM tasks.\n* l is the maximum latency between models $i$ and $j$ at one point. $F\\_{max}(l; i,j)$ is the CDF of these latencies. $F\\_{max}(l;i,j) \\= P(max(L\\_i,L\\_j) ≤ l)$ where $L\\_i$ and $L\\_j$ are random variables for the latency of $i$ and $j$. We agree that $l\\_{max}$ is a better notation and will fix this.\n* The total cost is approximately $5k USD.\n* Correct, we always log model comparisons. Users are required to accept this before using the platform.\n* Each element of the matrix is the win-rate between two models over all their battles. The win-rate would be $\\in \\[0, 1\\]$. We will clarify this in the paper by saying $W ∈ ℝ^{(M×M)}$ with $W\\_{ij} ∈ \\[0,1\\] \\forall i,j$", "claims_and_evidence": "Authors advocate the need for a coding specialized benchmark beyond existing works like Chatbot Arena. While I agree with this claim, I am afraid their exclusive focus on code completion with the FIM/L2R setup fails to capture the entire spectrum of AI tools used in the IDE. Most leading developer AI assistants (Github Copilot, Cursor) today provide the LLM interface in multiple formats beyond standard L2R or FIM completion, particularly the chat functionality (where for instance users can understand a repo by including it as a context). I'm not certain of the relative usage of such features (code completion vs chat), and a discussion on this aspect can immensely benefit this paper. But I find the sole emphasis on code completion to be a limitation of this work, as this does not throw light on the effectiveness of a candidate model in the chat setting. In my assessment, this work has immense value for the research community, but it is crucial for the paper to accurately convey what this benchmark captures (code completion capabilities), and what it does not (chat style functionalities). This could perhaps be accomplished by specifically referring to the capabilities being assessed rather than claiming wholistic real-world evaluation of Code LLMs. This is only briefly expressed towards the end of the paper (Section 7) currently.\n\nSecondly, authors propose that LLM evaluation can be entirely based on human preference evaluation, but this may not necessarily be true. Say model A gives a more readable but less performant piece of code than model B, and the user selects model A's output under EvalX - that does not necessarily reflect the superiority of model A over B? User preference may be biased towards readability while ignoring aspects like efficiency (runtime and space), maintainability, and security vulnerability of generated code. While static benchmarks do not represent real world use cases well enough, they can be adapted to capture these attributes of LLM generated code, and they offer significant value in that regard, which is difficult to record via human preferences. EvalX's fine-grained evaluation (user feedback on specific completions) also ignores aspects like the long term impact of LLMs on developer productivity (e.g. by what factor is the time required to complete a project reduced with a specific model behind a coding assistant).", "methods_and_evaluation_criteria": "In my assessment the proposed methods and evaluation make sense for the problem at hand.", "theoretical_claims": "NA.", "experimental_designs_or_analyses": "- Given that users are less likely to vote when experiencing high latency, authors have optimized model pair sampling to reduce the latency observed with some regularization to stay close to uniform.\n    - The data is skewed towards faster models this way - does that not impact the quality of the resulting dataset? And the observations you can draw from it?\n\nRelated questions:\n   - How to quantify the impact of this model sampling scheme - What'd be the difference in response/ranking rate with your model sampling vs uniform sampling?\n    - How would tweaking $\\tau$ impact coverage and user responsiveness?", "relation_to_broader_scientific_literature": "Several prior works (Chatbot Arena, LiveBench, BigCodeBench) have focused on evaluation of LLMs in different contexts including for code. The proposed EvalX benchmark attempts to instead capture real world in-IDE use cases when comparing LLMs, which addresses significant gaps in prior evaluation that relied on a fixed set of problems and the LLM's ability to solve these or were only focused on chat based use-cases.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n- Very promising and convincing approach on comparing and evaluating LLMs for developer assistance that addresses severe limitations of prior work. The tasks involved in EvalX are by design realistic and represent several diverse aspects of real-world programming use-cases. \n- Authors have designed effective tooling for user preference collection while ensuring several biases are mitigated e.g. top-bottom random swapping; latency masking of the slower model.\n- Several ethical considerations have been implemented by the authors given the potentially sensitive nature of data they're collecting from users of this study (Section 3). I'd encourage them to practice utmost care in releasing this dataset.", "other_comments_or_suggestions": "- Other benchmarks could be described with a little more detail to help readers understand how EvalX fits into the broader context of LLM evaluation\n\n- Lack of insights/takeaways for model training or prompting\n    - Would you advocate specialized (perhaps smaller models) over general (large models) for developers?\n    - I believe the paper could immensely benefit from insights or recommendations for users of coding assistants and model builders (of LLMs powering AI assistants)\n\n- Minor:\n    - Chatbot Arena citation in Section 1\n    - Takeaways not clear in Section 1", "questions_for_authors": "- What would you assess as the primary user motivation to participate? Free generations from commercial LLMs in IDE?\n\n- Do users get to pick the candidate models A and B?\n\n- What does fidelity mean in the context of Fig 2's caption?\n\n- Section 2.3: Any data or citation to support the first sentence?\n\n- Notation could be clearer in Eqn 1 and 2:\n    - What does $l$ mean?\n    - It is perhaps easier for a reader if you use the notation $l_\\text{max}$ instead of using the subscript for $F$ - as the maximum operator is applied on the observed latencies of the 2 models.\n    - Did you mean $F$ to be the CDF of latency? $F(x) = P(X < x)$, I don't follow how the CDF of maximum latency would make sense in this context.\n\n- What is the cost incurred so far with the ~11k preferences collected by EvalX or whatever is the total number of completions offered to users?\n\n- When users opt out of data collection - do you still log model comparisons but not their pref/suff/completion code?\n\n- Line 346: How is each element of this matrix defined and what range can it take? I didn't follow how it can $\\in R$", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JfvQ5HNhbT", "reviewer": "Reviewer_tJ9Z", "summary": "The paper presents EvalX, a platform for evaluating coding capabilities of large language models (LLMs) in real-world settings. Unlike existing evaluations that rely on synthetic benchmarks or chat-based interactions, EvalX integrates directly into developers' VSCode environments to collect authentic user preferences on code completion pairs. \n\nKey contributions include: (1) a novel interface for comparing model outputs directly in the IDE, (2) a sampling strategy to reduce latency, (3) a prompting scheme to enable FiM code completion functionality, and (4) insights into user preferences across different programming contexts. The authors collected over 11k pairwise judgments across 10 models and found differences between their leaderboard and existing static benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful suggestions. We address your comments below:\n\n*\\[Claim \\+ Evidence 1: The position bias in the interface design is concerning. The preference data may reflect convenience rather than quality assessment, which may lower the data quality.\\]*\n\n* Since we randomize the ordering of the two generated responses (L149), the position bias affects **all** models equally. This means that the overall **quality** of the assessment is not impacted. However, although the **quality** of the assessment is not impacted, the **efficiency** of our platform is impacted. In short, since there’s a significant position bias, we need more votes to shrink the confidence intervals enough to draw reliable conclusions.  Recall, we use logistic regression to estimate Bradley-Terry coefficients and bootstrap the samples to build confidence intervals (which allow us to be able to tell whether models are statistically significantly different). Overall, despite the efficiency reduction from positional bias, EvalX collects a sufficient number of votes to yield tight confidence intervals on model comparisons.\n\n*\\[Claim \\+ Evidence 2:  While the authors claim supporting 103 programming languages, the actual distribution is skewed.\\]*\n\n* We agree with your concern but note that we report model performance *aggregating* across languages, rather than performance on an individual language.   \n* Your comment prompted us to do further analysis of the distribution across languages. We found that even accounting for examples with over 50+ votes, we still have 23 programming languages. This is significant compared to previous static benchmarks (e.g., those mentioned in Table 1). Please see the following table for a more thorough distribution of our programming languages:  \n* We propose to add this table and modify the writing. For example, in our data analysis (Section 5.1), we can explicitly discuss how the data is not evenly distributed over all 103 languages, but there is a core set of languages for which there are a substantial number of votes. We believe these changes will help reduce any misinterpretations of our results on multi-lingual programming and natural languages. \n\n| Vote Count | \\# Programming Languages |  \n|---------------|----------------------|  \n| 5 | 65 |  \n| 10 | 45 |  \n| 25 | 31 |  \n| 50 | 23 |  \n| 100 | 17 |\n\n*\\[Method 1: Is there any way to quantify the potential impact on the reliability of specific model pair comparisons?\\]*\n\n* In the analysis conducted in the original submission, we took a few steps to mitigate this impact.   \n  * First, when computing Bradley-Terry coefficients, we use an L2 regularization term to prevent overfitting and bias towards models that have received more votes.   \n  * Second, we conduct a statistical bootstrap of the leaderboard so we can obtain confidence intervals around the estimated coefficients, which improves the reliability of results.  \n  * Finally, we ensure that we have reasonable coverage across all model pairs (e.g., \\>150 votes per pair), so votes are not too sparse across some pairs.   \n* Inspired by the author's suggestion, we did additional simulations using our data. We upsampled model pairs that had fewer comparisons and downsampled model pairs that had more comparisons so that the difference between the most and least voted pair is within 10%. We find that the rankings remain **identical to those** we report in the original submission. This provides additional confidence that we collected sufficient votes across all pairs.   \n* Overall, we appreciate this suggestion and will highlight this more prominently as a point of consideration for future evaluation platforms.\n\nFinally, thank you for the helpful pointers to additional evaluation benchmarks\\! We agree it would be challenging to include in our comparisons, but we will definitely mention them in the related work and opportunities for extending EvalX.", "claims_and_evidence": "Most claims are reasonably supported, with several issues:\n- The position bias in the interface design is concerning - with 86% of users selecting the first completion (requiring just Tab vs. Shift+Tab). While the authors acknowledge this bias and analyze decision times (median 6s for first completion, 9s for second), they don't sufficiently address how this fundamental asymmetry might invalidate preference data. Lots of the preference data may reflect convenience rather than quality assessment, which may lower the data quality.\n\n- While the authors claim supporting 103 programming languages, the actual distribution is skewed. Python alone accounts for 6000+ samples, while many languages have minimal representation.", "methods_and_evaluation_criteria": "The methods employed are novel and address a real need in the field. The in-the-wild evaluation approach is valuable, and the Bradley-Terry model for ranking is appropriate. \n\n- While optimizing for lower latency improves user experience, the sampling strategy potentially undersamples specific model pairs, is there any way to quantify the potential impact on the reliability of specific model pair comparisons (e.g., will the lack of certain pairs)?", "theoretical_claims": "The paper appropriately focuses on empirical findings rather than theoretical claims. The mathematical formulation of the sampling strategy (equations 1 and 2) is clearly presented, and the Bradley-Terry model application is sound.", "experimental_designs_or_analyses": "The user study design shows impressive scale and ecological validity.", "relation_to_broader_scientific_literature": "The work makes a significant contribution by bridging static benchmarks and human preference evaluation platforms, providing super valuable human preference data (manual labelled). It extends evaluation methodologies from chat-based platforms like Chatbot Arena.", "essential_references_not_discussed": "I think this paper might also be related to some repo-level completion evaluation works, e.g.,:\n-  CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion (NeurIPS 2023)\n- RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems (ICLR 2024)\n- REPOEXEC: Evaluate Code Generation with a Repository-Level Executable Benchmark (seems still under review)\netc..\n\nBut it seems hard to directly compare but may worth mentioning", "other_strengths_and_weaknesses": "### Strengths:\n\n- The \"Snip-It\" method enabling non-FiM models to perform FiM tasks is an innovative contribution beyond the evaluation platform itself.\n- The analysis of different factors influencing preferences (Figure 7) provides valuable insights for future model development.\n- Extensive ablation studies.\n- Valuable dataset \n\n### Weakness:\n- see above", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TheDjy4gbz", "reviewer": "Reviewer_1UGk", "summary": "This paper discusses EvalX, a system deployed in-the-wild to gather human preferences regarding code. It constructs a leaderboard based on user preferences and identifies notable differences compared to existing static benchmarks and human preference leaderboards. By analyzing EvalX’s diverse and unique data distribution, this study derives new insights into user preferences for code.\n\n## Update after rebuttal\nThe contribution is ok for me. This paper is acceptable.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful suggestions. We address the comments below:\n\n*[Weakness 1: While the deliverables are well-suited for the community, the main methodology appears to offer limited contribution to this ML community. Despite these concerns, I am inclined to accept the paper.]*\n\nWe thank the reviewer for advocating for accepting our work. We believe there are multiple reasons why the EvalX is a timely and important contribution to the ML community. Since the final version allows an additional page, we propose to add a Section 5.3 to explicitly summarize the insights and takeaways for researchers building new coding assistants: \n\n- **Different models excel in different settings.** Claude 3.5 Sonnet performs better at frontend/backend tasks, while Gemini and Deepseek excel at longer contexts. In contrast, changing models for different programming languages seems unnecessary. More generally, a routing approach based on input code context is an interesting direction for future research.\n- **Models should be trained and evaluated on varying code structures.** We observed a variety of tasks and code structures in our platform (e.g., FiM, docstrings, and inline comments). Future benchmarks and training schemes should explicitly account for variations in code structures.\n- **Models should be trained on human preference for code.** Given the clear gap between our leaderboard and static benchmarks, models should be trained on human preferences (of which our dataset is one option). Recent approaches have considered this, but still have significant gaps (e.g., Qwen-2.5 Coder trains on LLM-as-a-Judge preferences as a proxy for human preferences). By releasing EvalX data, our work can help to address the need for human preference data in real-world coding contexts.\n\n*[Suggestion 1: I recommend rephrasing Section 2.3….clarify whether the offline datasets are used for tuning the models?]*\n- As discussed in L166 (right), we do not fine-tune models, but rather only use the dataset for evaluation and tuning our prompts (L195, left). \n- However, we agree that Section 2.3 can be further clarified! To this end, we will also significantly expand on existing details in Appendix A and release all corresponding code for these experiments. \n- Altogether, with changes to Section 2.3 writing, additional details in Appendix A, and full release of code, we believe this will reduce any confusion on our prompting methodology. \n\n*[Suggestion 2: explain FiM task relation to \"smaller models seem to outperform in other static benchmarks compared to our leaderboard\".]*\n- In Section 5.2, our analysis indicates that FiM may not be the main contributor to model performance. While direct comparisons between models with and without FiM training would be ideal, most providers don't disclose their training paradigms. Our ablation using the Deepseek API (Appendix E, Table 6) demonstrates that FiM as an input format doesn't significantly impact performance. We'll clarify this nuance in our revised text.\n\n*[Question 1: How do you determine whether a completion is a FiM task?]* \n- To answer this, we conducted further analysis on whether the suffix is related to the completion. We broadly categorized our suffixes as either 1) inline (in which case it is clearly related) or 2) on a new line. Suffixes on the next line could be in scope (i.e., in the same function or loop as the last line of the prefix) or out of scope. \n- As shown in the table below, the vast majority (~80%) of our suffixes are related to the completion. We will include this analysis in the Appendix.\n\n| Formatting | Scope         | Percentage |\n|------------|---------------|------------|\n| Newline    | Out of Scope  | 20.4%      |\n| Newline    | In Scope      | 22.2%      |\n| Inline     |  In Scope        | 57.3%      |\n\n*[Question 2: How do you categorize the domain of the completion, such as frontend or backend?]*\n\nWe follow a two-step process detailed in the Appendix (starting from L880), which we summarize here:\n- First, we ask a model (e.g., GPT-4o-mini) to summarize all code contexts into short one-sentence descriptions.\n- Next, we prompt a model (e.g., GPT-4o) to cluster all one-sentence descriptions.\n- Finally, we provide the full code context and ask the model to categorize the context given the aforementioned clusters. \nWe additionally note that two authors of the work verified that the categorizations were sensical before scaling. \n\n*[Question 3: Is it particularly challenging to further scale the collected data ... for a specific language?]*\n- Since we are collecting data in the wild, we cannot directly control what languages we collect. However, we strive to grow the user base of EvalX to reach broader audiences, which will lead to more data in additional programming and natural languages.", "claims_and_evidence": "Well supported.", "methods_and_evaluation_criteria": "Yes. It's reasonable.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The soundness is pretty good.", "relation_to_broader_scientific_literature": "Provide a dataset in the real world.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "I have some concerns regarding the scope of this paper. While the deliverables are well-suited for the community, the main methodology appears to offer limited contribution to this ML community. Despite these concerns, I am inclined to accept the paper.", "other_comments_or_suggestions": "I recommend rephrasing Section 2.3, as the organization of the entire subsection is difficult to follow. Additionally, sentences like \"we use offline datasets to improve chat models’ infilling capabilities\" are not easy to comprehend. Could you clarify whether the offline datasets are used for tuning the models?\n\nThe observation that \"smaller models seem to outperform in other static benchmarks compared to our leaderboard\" might be attributed to the FiM task involved in the evaluation. I suggest further clarification of the underlying causes.", "questions_for_authors": "How do you determine whether a completion is a FiM task? Although there might be a suffix, the completion may not be related to it. Classifying it as a FiM task affects the evaluation of model performance.\n\nHow do you categorize the domain of the completion, such as frontend or backend?\n\nIs it particularly challenging to further scale the collected data in terms of increasing the data size for a specific language?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a7cYxTZRK4", "reviewer": "Reviewer_mirs", "summary": "This paper presents EvalX, a platform for evaluating coding LLMs in real-world environments. Integrated into developers' IDEs, it collects user preferences on code completions. Unlike static benchmarks, EvalX provides real coding tasks and optimizes latency. Findings show model rankings differ from traditional evaluations, highlighting real-world coding insights.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful comments. We address your comments below:\n\n*[Weakness 1: The evaluation metric of the model is relatively simple and can only reflect user preferences.]*\n- There is an extensive list of existing literature that follows this paradigm (see the first paragraph of the related work section). Despite its simplicity, it has proven to be an effective way to have humans-in-the-loop to evaluate models\n- However, the data we collect enables other forms of evaluation. For example, by utilizing the snapshots from a user’s code trajectory, we can analyze the long-term impact of each code completion. This is a direction we are actively pursuing as future work.\n\n*[Weakness 2: Although the benchmark is multilingual, the uneven use of programming languages ​​by users may result in inaccurate benchmarks for less common languages.]*\n- We agree with your concern but note that we report model performance aggregating across languages, rather than performance on an individual language. \n- Your comment prompted us to do further analysis into the distribution across languages. We found that even accounting for examples with over 50+ samples, we still have 23 programming languages. This is significant compared to previous static benchmarks (e.g., those mentioned in Table 1). Please see the following table for a more thorough distribution of our programming languages:\n- We propose to add this table and modify the writing. For example, in our data analysis (Section 5.1), we can explicitly discuss how the data is not evenly distributed over all 103 languages, but there is a core set of languages for which there are a substantial number of votes. We believe these changes will help reduce any misinterpretations of our results on multi-lingual programming languages. \n\n| Vote Count | # Programming Languages |\n|---------------|----------------------|\n| 5 | 65 |\n| 10 | 45 |\n| 25 | 31 |\n| 50 | 23 |\n| 100 | 17 |\n\n\n*[Question 1: Can this method include evaluations of some open-source models? How do you think you should balance user experience with incorporating more models into the evaluation?]*\n- Yes, EvalX can include open-source models! In fact, we reported results in our submission on 4 open-source models from multiple organizations (e.g., Llama 70b [1], Llama 405b [2], Qwen32b Coder[3], Codestral [4]). Since EvalX is an ongoing data collection effort, we continue to add open-source models to our platform.\n- However, you do raise a great point that we will clarify in our revision. We don’t anticipate being able to evaluate all models using EvalX because we do not want to impact user experience negatively. One way to operationalize this is to select models that perform well on existing benchmarks, indicating they will be usable in practical settings. This is what we did when selecting models for this work.\n\n[1] https://huggingface.co/meta-llama/Llama-3.1-70B.  \n[2] https://huggingface.co/meta-llama/Llama-3.1-405B  \n[3] https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct  \n[4] https://huggingface.co/mistralai/Codestral-22B-v0.1", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes. By collecting user preferences, the author evaluated the models.", "relation_to_broader_scientific_literature": "This work is related to the evaluation of code generation for large models.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strengths**\n1. This work proposes a novel method for llm code evaluation, which collects user preferences for models through a vscode extension.\n2. Compared with static code evaluation benchmarks, this method can better reflect users' preferences for llm in real scenarios.\n\n**Weaknesses**\n1. The evaluation metric of the model is relatively simple and can only reflect user preferences.\n2. Although the benchmark is multilingual, the uneven use of programming languages ​​by users may result in inaccurate benchmarks for less common languages.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Can this method include evaluations of some open-source models? Doing so seems to negatively impact the user experience. How do you think you should balance user experience with incorporating more models into the evaluation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a new platform (EvalX) for comparing coding assistants. The EvalX platforms has collected 11K pairwise comparisons from 10 models by integrating an interface into the VSCode IDE, and it proposes a sampling strategy to reduce latency for practical use. This work is undoubtably a great contribution to the research community as understanding the preferences from users in a production environment is crucial in developing next generation of programming tools. Also the ethical considerations that are implemented by the authors are also potentially impactful for follow-up works in collecting user data for coding tasks. Concerns are raise on limited scope, as only code completion is assessed, but code completion is also the majority use case of coding assistants, thus the concern is relatively minor. But I would suggest the authors to add more discussions on the impact on soundness resulted from the skewed distribution of the programming languages, as well as main insights and takeaways derived from the evaluations for the selection of coding assistants for the users, as well as for the model developers.
Decision: Accept (poster)