EditBench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits
Authors: Wayne Chi
Valerie Chen
Ryan Shar
Aditya Mittal
Jenny Liang
Wei-Lin Chiang
Anastasios Nikolas Angelopoulos
Ion Stoica
Graham Neubig
Ameet Talwalkar
Chris Donahue
Venue: ICLR 2026 Oral (2026)
Keywords: code
real-world
llm
code edit
edit
Primary area: datasets and benchmarks
TL;DR: We propose a new benchmark for evaluating an LLM's ability to perform code edits. Our data is gathered from in-the-wild code edits, leading to more realistic problems.

Abstract: Instructed code editing, where LLMs directly modify a developer's existing code based on a user instruction, is becoming a widely used interaction mode in AI coding assistants. However, few benchmarks directly evaluate this capability and current datasets often rely on artificial sources. We introduce EditBench, a benchmark for evaluating LLM code editing capabilities  grounded in real-world usage, i.e.,~user instructions and code contexts collected in the wild.  EditBench comprises of 545 problems, multiple natural and programming languages, and a diverse set of real-world use cases, ranging from resolving errors to adding features. EditBench introduces context-dependent problems that require the model to understand code context, highlighted code, and cursor position in addition to the user instruction. We evaluate 40 diverse LLMs and observe that EditBench is a challenging set of problems where only 3 models score over 60\%. We find that model performance varies across different categories of user instructions. Further, we find that varying levels of contextual information greatly affect task success rate, with performance varying up to 11\%, indicating the importance of evaluating with realistic context.

Published as a conference paper ICLR 2026
EDITBENCE: EVALUATING LLM ABILITIES TO PERFORM REAL-WORLD INSTRUCTED CODE EDITS
Wayne Chil,+
Valerie Chen 1.+
Ryan Shar
Aditya M
Jenny Liang'
Wei-Lin Chiang
Anastasios Nikolas Angelopoulos Chris Donahue if
Ion Stoica?
Graham Neubig'
Ameet Talwalkar
Carnegie Mellon University
UC Berkeley, Arena
ABSTRACT
Instructed code editing, where LLMs directly modify developer's existing code based on a user instruction, is becoming a widely used interaction mode in Al coding assistants. However, few benchmarks directly evaluate this capability and current datasets often rely on artificial sources. We introduce Edi t.eechh, a benchmark for evaluating LLM code editing capabilities grounded in real-world usage, i.e., user instructions and code contexts collected in the wild. Edit Edit Bench comprises of 540 problems, multiple natural and programming languages, and diverse set of real-world use cases, ranging from resolving errors to adding features. Edithench introduces context-dependent problems that require the model to understand code context, highlighted code, and cursor position in addition to the user instruction. We evaluate 40 diverse LLMs and observe that Edi Bench is challenging set of problems where only model scores over 60%. We find that model performance varies across different categories of user instructions. Further, we find that different levels of contextual information greatly affect task success rate, with performance varying up to 8%, indicating the importance of evaluating with realistic context.
Github Repo
Leaderboard
1 INTRODUCTION
Software developers increasingly write code with Al assistants such as Github Copilot (Gihhb 2022) Cursor (Cursor 2023), and Continue (Continue Dev 2025) using a variety of modes of interaction. Instructed code editing, where developers use natural language to request the assistant to edit a highlighted section of code, has emerged as a prominent interaction mode alongside autocomplete suggestions and chat (Nam jet al. 2025). Due to the flexibility provided through natural language instructions, use cases for edits are diverse and range from code improvements given detailed user instructions to bug fixes provided only an eтror trace (Cassano et al., 2023b). Because of this, instructed code edits pose challenging set of problems that existing LLMS must tackle to support developers.
Despite the emergence of this new interaction modality, we lack benchmarks to capture real-world edit behav- ior. Code generation benchmarks typically evaluate L.MM capabilities on generating code from scratch (Chen) et al. 2021; Austin et al.] 2021; Jain et al. 2024, White et al., 2024p. While there are few edit-related datasets (e.g., CanItEdit (Cassano et al. 2023b), Aider polyglot (Gauthier, 2025h), the sources of data are
* Equal contribution.
t Equal senior author.

---

Published as a conference paper ICLR 2026
Language Model
+
Test Cases
Instruction
Pass
Test
fix the flask app and show the current...
queries_all, all users
nouter (users
def pet_userst
X
gets_ ciss_ccurent_date _ate
Context
V
users users_route____ roustoeeeettttee__eeeeeeees exists
Code
return [sansfytusers) users
health_rove rsiite
Highlighted Code
db.sesston.
cereloo-ooerylierraattttt users))
mo_dict_erne
Cursor Position
Task Success
Desp.routad (healthh)
Figure I: EditBench tests LLMs' real-world editing capabilities. We propose EditBench, an evalua- tion real user instructions and code snippets collected in-the-wild. is the first benchmark for instructed code edits that requires models to ingest the user instruction, current code, highlighted code, and cursor position solve problems.
not reflective of most real-world software development, relying either simple, annotator-written problems or Leetcode and educational style problems that do not capture diverse, real-world software development challenges. Recent work has begun collecting human preferences to interactively evaluate models Chatbot Arena (Chiang et al.] (024)) evaluates LLM capabilities for chat and contains a coding subset, while Copilot Arena (Chi al. (025) evaluates LLM capabilities perform code completions-hhghigghtin
growing awareness of the need for grounding evaluations with in-the-wild data. However, "arena-style" evaluations are costly, requiring significant number of human votes rank new model.
We introduce Edi Bench, a benchmark for evaluating LLM code editing capabilities built on real-world edit contexts and instructions (Figure D. We source our problems by developing VS Code extension that mimics existing instructed code editing tools from GitHub Copilot and Cursor. As developers use the extension, we gather live, in-the-wild dataset containing user wser-written instructions, associated code context, and user votes between pairs of model responses. We recruited nearly 500 users to provide these data points. EditBench differentiates from previous edit-related benchmarks in several ways:
Diverse user instructions and context. Since BditBench is constructed from data collected from programmers performing day-to-day coding tasks, users specify formats. For example, a bug fix can be requested as "fix this" accompanied with highlighted code, a direct dump of the trace, natural language description of the erroneous behavior. Ed i Bencch tests for these varied user instructions instead of the more templated approaches (e.g., fix a specific function in a well-defined way) in previous benchmarks.
goals with diverse content and
Context dependent problems. Real instructed code edits often feature ambiguous user instructions that require contextual clues parse the underlying user intent. In addition to the user instruction, in EditBench we also capture the code file to edit, the highlighted region of code, and the user's ss current cursor position. Code context length can be significant (e.g.. 2 20k characters). requiring the model to properly use the comments, highlighted code, and other contextual clues to determine the correct solution. We are the first benchmark to include this combination of features for instructed code edits.
Multiple natural and programming languages. While most previous coding benchmarks consist of only English problems, EditBench consists of natural languages (English, Spanish, Russian, Chinese, Portuguese) and 2 programming languages (Python and Javascript). Since our code is gathered in-the-wild, any natural language variations occur both the user instruction and code itself.
We evaluate 40 open-weight and closed models on EditBench and find that the best model, claude-scnnet- (Anthropic) 2023), achieves a pass@l of 66.67%. Closed-source models tend
2

---

Published as a conference paper ICLR 2026
outperform open-weight models, with deepseek-chate two open-weight models in the top 10. We observe that both the inclusion of additional context (e.g., high- lighted code and eursor position) and the type of edit category (e.g., optimization versus bug fixing tasks) drastically affects performance, Finally, we find that EditBench is only weakly correlated with existing edit benchmarks like Aider Polyglot (Gauthier 2025), suggesting that our real-world data captures a unique set of difficult edit tasks. Our results show that Editßench is challenging even for state-of-the-art models and reveals new insights into model capabilities, emphasizing the importance of benchmarking LLMs on realistic data.
and
being the only
2 RELATED WORK
2021 and MBPP (Austin et al. (202T), largely focusing on interview-style programming problems have been the most commonly used to
Coding Benchmarks. Static benchmarks, e.g., HumanEval (Chen et al.
evaluate coding capabilities (Lu et al. 2021; Nijkamp al. [023] Zhu al. [022 Wang et al. 2023 Liu
let al.
2023 Jimenez et al. 2023b; Khan et al., [2023] Yan et al. 2023) Cassano et al., [2023a] Muennighofi
ct al.
2023 Dinh et al. 2023] Yang et al., 2024b), measured using pass@k. Additionally, some recent
work
focuses on creating live benchmarks that reduce contamination risks (Jain et al. 2024, Whit White et al.
2024
Increasingly, people are interested in code editing with LLMs, focusing on bug fixing (Zhag et
20236
Moon et al., [2233] Shinn et al., [2023] Chen et al., 2023] Olausson et al.] [023 Jin et al. 22023]
Joshi et [ Wei et al.
2023 Li et al. (0022). a specific subset of code editing:
code 2022, Fried et al.) 2023 Yee & Guha 2023] Roziere et al. [2023] Guo et al.
completion (Bavarian et al.
2024a Zhang et al. 2023a). an inference strategy that requires specific insert locations; and intrinsic code
editing (Li et al. 2023 Gupta et al.) 2023), which involves editing code without specified instruction, exerting the model's ability to intrinsically ascertain the desired code changes. CodeEditorBench (Guo et al. 2024b) evaluates code editing using competitive programming problems and CanItEdit (Cassano al. 2023b expands on this to create varied prompts and diverse topics.
Grounding Evaluation in Real- World Data. A limitation of the aforementioned benchmarks is that the source of their tasks is not from real-world user data. Copilot Arena (Chi et al. 2025) evaluates code completions with real-world data and highlights how the distribution of data from benchmarks differs from real-world data in terms of the type of task, context length, and more. However, these in-the-wild evaluations require immense scale build a leaderboard and evaluate new models (c.g.. Chatbot Arena (Chiang et al. 2024) has millions of votes). The primary benchmark that creates problems from real-world sources is SWE-Bench (Jimenez et al., (223a) and related extensions including SWE-Bench Multimodal (Yang et al. 2024a) and Multi-SWE-Beed
(Zan et al., 2025). However, these benchmarks focus on fixing issues that require agentic workflows ge.g., editing multiple files) and are limited to a handful of repositories or problems written in one natural language. Our work, Edit tiittee Bench, complements this growing set of benchmarks by providing a benchmark for instructed code edits that is realistic (i.e., collected from real users in real workflows) and diverse (i.c., contains many different natural languages and task categories).
3
BENCHMARK CONSTRUCTION
3.1 DATA COLLECTION.
We develop open-source VSCode extension with instructed code editing as a core feature to support the collection of code edit data. Gathering data via a real coding extension (Izadi et al. 22024] Chi et al.) (025) allows for more realistic instructions and tasks when compared to coding competition platforms. For each code edit, the user highlights a code-snippet and writes a short task description (Figure 2 Participants are not compensated for using the extension, as in traditional user study, but instead receive free access
3

---

Published as a conference paper ICLR 2026
Highlight Code and Write Instruction
User Reviens Code Edit
preees, dine carst Geance, net, torget reet)
pracess www.strryyeeeureco rest, to rettt meceeiiii
Creace tarpet rast directary of it saeex"
tet suddirecte in Sauce
seaitirs Ie for a Enterrtparee
73
xodereeterrs ae in serce
Fouud Eoen suddiral
26
Drees (laninaacina)) suadinectarien
sandbor
patr
patr_
Verdine los
Optimioe
Precess sabdirocarry
Enter your prompt code generation (Preas Enerr confirm
uudir audlici
Figure 2: We develop an open-source VSCode extension collect real-world edits.
state-of-theear models. Given the sensitive nature of programming, we established clear privacy controls to give users the ability to restrict our access to their data. Depending privacy settings, we collect the user's instruction, code context (including the highlighted code segment, the cursor location, prefix, and suffix) at the time of the request, and model responses. Additionally, we log whether the user accepted the edit. Our data collection process was reviewed and approved by our institution's 's IRB. Additional details about our data collection policy provided in Appendix A
3.2 PROBLEM CURATION.
Across 458 users, we collected 2672 responses (i.e., the user accepted an edit). However, not all of these responses were interesting, challenging, or even feasible to turn into testable problems. We narrow our problem set in the following ways. First, we focus on questions written in Python and Javascript, which combined comprise of the majority of our responses at just over 1700 problems. Second, we exclude problems that are too similar to one -iibherrssmeimees a user might try similar prompts on the same code context to see how different models edit. Lastly, we remove any trivial (e-g., add single parameter), stylistic (e.g., add a comment), or ambiguous problems. We provide concrete examples of removed problems in Appendix C This filtering process left us with around 470 problems which we found both interesting and challenging. Given that not all problems are feasible to create test harnesses for, succeeded in creating 109 unique problems for BditBench-core EditBench. In order to equally distribute the natural languages in the problem set, also translate each problem to the other languages found in our problem set to form Edi tBench-complete followed similar method prescribed by HumanEval-XI. (Peng et al. 2024) and translate the comments in each problem using GPT-4o to create a total of 540 problems. To validate the translations, we had native speakers evaluate subset of the translated tasks. primarily in Chinese and Spanish. In addition to GPT-4o, we experimented with several other models (GPT-4o-nano, GPT-4o-mini and Google Translate, but found GPT-4o to provide the best quality with no noticeable concerns with any of the translations.
There are five languages Enggish, Russian, Chinese, Polish, and Spanish -
To do so, we
3.3 TEST HARNESS CREATION.
The data from our extension provides us with realistic human instructions and code, but does not contain test cases, making the raw data ill-suited for a benchmark. We create test harnesses composed of the environment setup, which includes preparing configurations, virtual environments, mock files, and test cases that define expected inputs and outputs.
To write our tests, we assemble team of five experienced programmers who have expertise in both natural and programming languages present in the real-world edit data. The team, recruited through academic networks, included researchers and students from various fields who write code extensively. The annotators
4

---

Published as a conference paper ICLR 2026
Table 1: Comparing EditBench to other edit-related benchmarks. We compare Edi Bench with similar benchmarks (CanltEdit (Cassano et al. 2023b), EditEval (Hu et al. 2023), Aider Polyglot) in terms of the problem source, user instruction (# NI. refers to the number of natural languages), code context (# PL. refers the number of programming languages, HI refers to whether
can highlight subset of code), and associated test cases. Standard deviation is indicated by =. Edi SditBee Bench is the only benchmark built from in-the-will problems and exhibits considerable variation in both instruction and code context length.
Benchmark
Problem
Instruction
Code Context
# Problems
Source
NL
Length
PL
Length
HL
CanItEdit (Cassano et al. EditEval (Hu et al. 2023
2023b
105
Annotator
1
140 +: 105
3
1309 = 1116
No
194
Annotator
1
99.9 +: 49.3
1
258 = 185
No
Aider Polygiol Gauthier
2025
225
Coding Exercises
1
606 +: 885
5
6184 = 6452
No
EditBench
540
In-the-wi an-the-wild
5
238 +: 738
2
5642
7567
Yes
none
pardas
scabom
requests
beem
cotonw
streamit
stleam
metpletils
time
datetime
statsmodelss
typing
xmi
torch
alogram
colpallenggire
colpali_ engine
tkinter
asyndo
Shufil
pathiib
random
torchvision
pickle
ntb
transformers
44 more
numpyy
json
Figure 3: Distribution of libraries in EditBench for Python problems. Edi Eench contains 74 unique imports compared to 25 (CanltEdiit, 15 (Polyglot), and 16 (EditEval) from other benchmarks. See Appendix for other languages and other benchmarks.
were instructed to create test harnesses that adhere to the user's intent and are generalizable to different potential implementations. While the user instruction and code file are perhaps the most important pieces of information, they by themselves can often be too ambiguous. The highlighted code segment and cursor locations provide crucial contextual clues to prescribe user intent. Annotators were asked to design problems given all of this information, and if problem was still too ambiguous, we asked the annotators remove the problem. To support the annotation process, we generated some example solutions using GPT-4o and Sonnet 3.7 (chosen to balance cost and quality) to give insight into possible solutions. Additionally, annotators were also asked to screen for and remove any Personal Identifiable Information (PII). Finally, all refined test cases were assigned a second annotator the team to do second review with the same procedure.
Originally, we attempted to use a coding agent (e.g., Claude Code) to construct test cases, but found that the agent often struggled with test case generation itself, frequently resorting to undesirable tests such as directly pattern- mettennntttchng with the source code, despite explicit instructions avoid this behavior. However, despite the complexities involved in environment setup, especially for languages such as Javascript, we found the agent was consistently able to set up the correct packages and environments. As result, we used the agent to setup the test harness environment. We provided setup files (e.g., a conftest. . py file in Python and a jest-confar js file for Javascript) to help support the agent and standardize outputs.
4 BENCHMARK STATISTICS
EditBench consists of 540 problems that span 5 natural languages (English, Spanish, Russian, Chinese, Portuguese) and 2 programming languages (Python and Javaseript). Edi tench features diverse set of problems with considerable variation in instruction and code context lengths (Table Based on the import that Edit Bench captures 74 different unique imports, demonstrating
library usage (Figure
5

---

Published as conference paper ICLR 2026
Table 2: Comparing user instructions written in IDE to the instructions written by human annotators. We provide examples across different task categories, comparing with two edit-related datasets (CanItE- dit (Cassano et al. 2023b) and EditEval (Hu et al., 2023)). We truncate some instructions for brevity and provide full examples in Appendix [ In general, we find that real-world prompts much less specified and require models to leverage the provided context, compared to existing benchmark prompts.
EditBench (Ours)
CanltEdit Cassano et
2023b
EditEval (Ilu 2023)
Feature Addition
take the globe countries layer from below >1/ this and add it to the existing globe
Add method returna the estimated the appzepriate losation this house, calculated by...
Add a function
that
'filter.o Filleerodd.nn odd.numeers' filter odd numbera using lambda function.
Feature Modification
do not use style, use python atyle
Flip the correlation function given to calculate the sovariance instead using the Corr (X, Var and (Y) The new function should...
Modify the function to correctly determine the season based on monch and day, for
considering season changes. Raise error when...
Resolve Errors
Runt imcBrror: Cannot close running ng event loop ays:l: Runt imeWarning: corcutine "Applicat Lon, shatdown' was
Fix combination.us imited. that it returns the right result. The function combination unl unlimitd rep should...
Fix the bug in sum nom_even.aand even eeedeeeeeniiee index' make it return the aum of even numbers at even indices.
Optimize Code
optimize the computation by better batching the latter pazt
Opt imiie the bm25 algorithm by avoiding frequency selculdriens.
Optimize the function to find longest common subsequence for the given two sequences using dyramic programming
much more diversity (at least three times) than existing benchmarks. From analysis Edi tBncch problems, we find the following characteristics:
Real user instructions are diverse and messy. When inspecting real-world data, find that users write varied instructions across many problem categories. While many of these categories are similar to existing benchmarks, find that user instructions much more informal and less well-specified compared the umnotator-writter instructions in existing benchmarks (Table Interestingly, even the way user would write an instruction within category varies in terms of descriptiveness. For example, to resolve errors, users may briefly describe the erroncous behavior using natural language or directly paste in the terminal error traces. Further, unlike prior benchmarks where user instructions are only written English, we find users write instructions in multiple languages, including Russian, Chinese, and Spanish (see T for additional comparison user instructions).
Real-world code contexts span many applications and context lengths. We observe that users work on variety of applications, including frontend/backend, machine learning, and algorithmic problems. Additionally, the context lengths are much longer than those evaluated in prior benchmarks (Table 12]. We also look at the distribution code-to-edit token lengths, as computed by the number of highlighted tokens, and find that most people highlighting targeted portions of code for edits. The median is 138 tokens, while the full file is typically closer to 4.5k tokens. The code contexts that we collect are primarily in Python (43%), with the Savasrrip//yppecrript next most common programming languages being Javascript/ Typescript (21 %), PHP (18%), and HTML (7%).
6

---

Published as a conference paper at ICLR 2026
We focus on problems written in Python and Javascript, which together comprise the majority of in-the-wild instructed edits collected.
We identify four common clusters of functional edits. By analyzing in-the- wiid user instructions in EditBench, we derive four different categories that describe functional real-world edits: feature addition, feature modification, bug fixing, and optimization. We find the distribution across these categories as 43% additions, 27% modifications, 22% fixes, and 8% optimizations. Table provides examples of each category. our later analysis, we compare how well models are able to perform these different problem categories.
5 EVALUATION
We now use Edi t.ench to evaluate models and identify trends in code editing capabilities across models. We also compare t Bench results to existing benchmarks. We overview our choice of LLMs, evaluation metrics, and prompts to perform code edits, with additional details in Appendix D
Model choices. We select 40 LLM spanning multiple model families, sizes, and training schemes (e.g..
reasoning and non non-reasoning models). We use 9 models from the GPT family (OpenAl 2025). models
from Qwen (Hui et al.] (2024), 5 models from Llama (Meta 2025), 4 models from Mistral (Mistral 2025 3
models from Sonnet (Anthropic 2023).3 3 models from Gemma (Team 2025b). 2 models from Grok (Grok 2025), 2 models from Deepseek (DeepSeek-Al et al., 2024), 2 models from Gemini (Google DeepMind 2025). model from Kimi (Team 2025c)), and I model from the GLM family (Team] (022a)). For full list of models, see Table For GPT reasoning models (gpt- o3-mini, gpt-o4 -mir.i, gpt-pppppp-- 5), we also vary reasoning effort. We set temperature to O when possible to reduce non-deterministic outputs.
Evaluation Metrics. Following prior work (Kulal et al. 2019 Chen et al. 2021 we report pass@l, where 1 code sample is generated per problem and problem is considered solved if it passes all unit tests. To facilitate analysis on the types of problems that current models excel or struggle with, we also partitioned our dataset into two subsets of easy and hard difficulty, in addition to reporting the Full results. We categorized problems that were solved by i or fewer models as Hard and the remainder as Easy (Gauthier) 2025). To obtain roughly even split between problems, we selected k 20. We find that easy versus hard problems are roughly evenly distributed across problem categories.
Code Editing Methods. In all our prompts, the model is given the user instruction and main code context and requested to edit the entire file by regenerating the entire code context. We also evaluate models when given varying levels of contextual information (e.g., highlighted code and cursor position). We find that models perform best when given highlighted code, but not cursor position; hence, we run all of our main experiments with highlighted code given only. All prompts are provided in Appendix D
5.1 DISCUSSION OF RESULTS
We present our primary results in Figure 4 and highlight the key takeaways below. Appendix E provides additional results and discussions.
EditBench is a challenging benchmark, even for current state-of-the--rt models. Only out of 40 models (cLaude-sonnet-2 achieves more than a 60% pass@l. Further, EditBench captures questions of varying difficulty, reflecting the diversity of challenges in real-world code edits. As such, we find a sharp contrast between the easy and hard questions, where the average gap across models is 59.3% (standard deviation of 10.6%). Given the large gap between easy and hard problems, we explore what types of prompts are present in hard problems compared to the general datasct. Overall, we see that hard instructions tend to have shorter instructions (by nearly 5 times) but slightly longer highlighted code. This means that the model cannot simply rely on following the user's 's instructions alone but rather needs to reason about multiple pieces of information. We provide an example in Appendix E
7

---

Published as a conference paper ICLR 2026
Actess Seeed Opm
20
40
nout
Figure 4: We evaluate 40 LLMs on EditBench. We report the passe 1 of each model; only out of 40 models have a pass@1 1. greater than 60%. In general, closed-source models outperform open models.
Table 3:
Additional context affects performance. Highlighted code is crucial to performance, improving task success rate across out 7 models when included in the prompt. Surprisingly, adding cursor position leads to mixed results. Models chosen are the best model in the top 7 model families.
Pass
Model Name
Code Only
+Highlight
+Cursor Only
+Highlight +Cursor
c clauue-sonnet-4
62.41
64.81 (42.4)
63.15 (40.74)
64.26 (+1.85)
deepseek-chat-v33
51.48
54.26 (42.78)
53.15 (+1.67)
52.78 (+1.30)
semini-2.5-f1es
52.59
52.96 (40.37)
52.41 (-0.18)
56.30 (+3.71)
glm-4.6
52.96
56.48 (+3.52)
52.22 (.0.4)
44.81 (8.15)
03-mini
60.00
56.85
(3.15)
59.26 (.0.4)
55.19 (4481)
kimi-k2-099
54.63
56.48 (+1.55
52.22 (-241)
58.15
(43.52)
qwen3-coder
56.48
53.89
(-2.59)
56.48 (40.00)
53.89
(-2.59)
Model performance is affected by additional contextual information. To evaluate how additional contex- tual information (highlighted code and cursor position) affects model performance, we run an ablation with the 7 top models in different model families (Table 33. When adding highlighted code to the prompt, the task success rate increases for out of the models. On the other hand, additionally adding the cursor position leads to mixed performance when compared to only adding the highlighted code. We notice that trends are generally consistent; the two models that do not benefit from including highlighted code in context- o3-mini and twen3-coder: do not benefit from including cursor position either. These findings show the importance of evaluating models on editing tasks that require integrating multiple pieces of information.
Gap between closed and open models. Comparing the colors in Figure 4 very readily shows that open models tend to lag behind closed models. Out of the 40 models we evaluate, only out of the top 15 are open models, and the bottom 15 are all open models. Of the open models, we find that g1m-4. 6 performs the
8

---

Published as a conference paper ICLR 2026
3.3-70b instruct
qwen cnder-add
meen3-coder
9pt
gpt
0.7
0.7
0.7
0.7
37
07
0.7
0.6-
0.6
0.6
0.6
3.6
0.
0.6
0.5
0.5
0.5
0.5
35
05-
0.5
0.4-
0.4
0.4
0.4
9.4
04
0.4
0.3
03
0.3
0.3
0.3
03
0.3
02
02
0.2
0.2
9.2
02
0.
103000 mod
Ficers ere
0p0 Mix
Figure 5: Comparing top-performing open-weight and closed models. To illustrate individual LLM differences, we compare 7 models and find passšl varies greatly depending on the problem category. Additionally, different models perform best at different categories.
best with pass@l of 56.48% with kimi- -2 and ceepseek-chat -v3. not far behind. Surprisingly, gpt-5 with default reasoning (medium effort) lags behind gpt --nt-5 When inspecting test cases where gpt-5 failed, find that struggles with simple tasks like formatting code indentation properly and catching edge cases, despite being a strong reasoning model.
Models excel in different problem categories. When we divide questions into categories that test different oditing-related skills, we find that performance varies. Overall, we find that models perform best on bug fixing problems (average of 52.2%), which may be most akin tasks found in prior benchmarks like SWE- Bench (Jimenez et al. 2023a). In contrast, models tend to struggle with optimization and feature addition (44.6% and 39.6%, respectively). Still, we find that claude sonnet-4 ranks first in every category except optimization. Furthermore, we find that some models have particularly large gaps between categories (Fig- 5). For example, qwen3-coder- making feature modifications.
's top category is fixing bugs while claude-sonnet
5.2 COMPARISON TO EXISTING BENCHMARKS
We compare our results with two maintained leaderboards: performance on Aider Polyglot (Gauthier 2025 ) which has been used in prior model releases as metric of model editing capabilities, and ranking on the coding subset of Chatbot Arena (Chiang et al. 2024), which has been widely used to capture human preferences. We have 17 and 30 shared models, respectively. We observe a weak, positive correlation with both Polyglot (Pearson correlation coefficient r = 0.24, = 0.06) and Chatbot Arena (r = 0.11, = 0.01).
We believe our observations are due to the following factors. The first is code-centric input and output. Input/outputs in Chatbot Arena are often written purely in natural language, so the majority of coding- related questions in Chatbot Arena do not contain code (Chi al. 2025); this is unlike Bench and Polyglot, both of which require code for every problem. Second, there a difference in interaction modality, Edi tBench and Polyglot test model's ability perform instructed code edits, where there is freeform input (the user instruction) and structured output (the resulting code), while Chatbot Arena evaluates a model's 'ss ability to chat, where there is both freeform inputs and outputs. Also, the inclusion of additional code context (e.g., highlighted code) may affect correlation to Polyglot. Finally, correlation may be affected by the inclusion of real-world user intent. Polyglot's problems are entirely based on coding exercises from educational-style problems that lack the organic user intent present in Chatbot Arena and Edi Bench.
6 CONCLUSION, LIMITATIONS, AND FUTURE WORK
As instructed code edits become more widely adopted in real-world IDEs, there is a need to benchmark LLM capabilities these types of problems. We develop a VSCode extension to collect real-world instructed

---

Published as a conference paper ICLR 2026
code edits, which include user instructions and code contexts. We transform this in-the-wild edit data into Beithench, a set of high-quality test harnesses that evaluate LLM's ability to perform diverse tasks. Evaluations on models show that Bdichench is challenging even for current state-of-the- models and provides insights into how performance varies when considering different code context information and types of edits. Overall, to adequately support developers using LLM-powered tools, our findings demonstrate the need for future models to be trained on real-world interaction modes and evaluated across a broad spectrum of problem categories, languages, code contexts, and user intents.
Limitations and Future Work. While attempted to make Edi tBench diverse possible, there are still additions from which it would benefit. For example, as we collect more data using our extension, we will increase the number of examples have for the existing languages and expand other common programming languages. Additionally, despite improvements over existing benchmarks, it is unclear to what extent our problems encapsulate all real-world cases. We plan continue updating the Edi tencch leaderboard as new models are released and exploring automatic workflows to more seamlessly translate real-world data to benchmark problems.
ACKNOWLEDGMENTS
This work was supported in part by the National Science Foundation grants IIS1705121, IIS 181888017, IIS2046613, IIS21 12471, and funding from Datadog. Any opinions, findings and conclusions or recommen- dations expressed in this material are those of the author(s) and do not necessarily reflect the views of any of these funding agencies.
REFERENCES
Anthropic. Meet claude, 2023. BRRL:rttss//w..aatrriic...oo
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.077 07733, 2021.
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207. 122.5 2022.
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl-e: a scalable and polygiot approach benchmarking neural code generation. IEEE Transactions on Software Engineering, 2023a.
Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Jacob Ginesin, Edward Berman, George Chakhnashvili, Anton Lozhkov, Carolyn Jane Anderson, et al. Can it edit? evaluating the ability of large language models to follow code editing instructions. arXiv preprint arXiv:2312.12450 2023b.
Mark Chen, Jerry Tworek, Heewoo Jun Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arX1v:2304.05/28, 2023.
10

---

Published as conference paper ICLR 2026
Wayne Chi, Valerie Chen, Anastasios Nikolas Angelopoulos, Wei-Lin Chiang, Aditya Mittal, Naman Jain, Tianjun Zhang, lon Stoica, Chris Donahue, and Ameet Talwalkar. Copilot arena: A platform for code Ilm evaluation in the wild. arXiv preprint (rxii:25022.0322.. 2025.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, al. Chatbot arena: An open platform for evaluating Iims by human preference. arXiv preprins arXiv:24033.44132, 2024.
Inc.
Continue Dev. Continue: Open-source ai code assistant, ht cont inue 2025. Accessed: 2025-05-08.
Cursor. Cursor: The ai-first code editor, 2023. URL httpss//causstt.hhffaatttt 2023-12-03.
Accessed:
DeepSeek-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, and Chengda Lu et al. Deepseek- v3 technical report. ArXiv preprint, abs/2412.19437, 2024. URL /ttp:tts://aax..02//////// 2212
19437
Tuan Dinh, Jinman Zhao, Samson Tan, Renato Negrinho, Leonard Lausen, Sheng Zha, and George Karypis. Large language models of code fail at completing code with potential bugs. Advances in Neural Information Processing Systems, 36, 2023.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: generative model for code infilling and synthesis. The Eleventh International Conference on Learning Representationss 2023. URL https: httts:://ppenreviee
net/ for.moormmmiom d 1MMET.
Paul Gauthier. Aider polyglot coding benchmark. 2025. Accessed: 2025-05-08.
Github, Github copilot your ai pair programmer, 2022. URL httltpss//gitubbb.om/fateeeee copilot
Google DeepMind. Gemini 2.5: Our newest gemini model with thinking. https:://blo..oooll4 technology, wowpggggggggglooooooooooooogoooe deepmind/ wtwpe//eeeeiiiennnneeeeeeeeeeeee -oodel-hhinkigg wawpann--d-tttttaa-aaaaeeeeaee March 2025.
Grok.
Grok
code
fast
model
card.
https://aata.x.al
2005 - 08 - 26 (rookoo--ao code fast 1 model-cee pmodel-card..o card. pdf, August 2025.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Decpseck-coder, When the large language model meets programming the rise of code intelligence, 2024a.
Jiawei Guo, Ziming Li, Xueling Liu, Kaijing Ma, Tianyu Zheng, Zhouliang Yu, Ding Pan, Yizhi LI, Ruibo Liu. Yue Wang, Shuyue Guo. Xingwei Qu, Xiang Yue, Ge Zhang, Wenhu Chen, and Jie Fu. Codeeditorbench: Evaluating code editing capability of large language models, 2024b. URL https
//arxiv. abs/ 2404, 03543
Priyanshu Gupta, Avishree Khare, Yasharth Bajpai, Saikat Chakraborty, Sumit Gulwani, Aditya Kanade, Arjun Radhakrishna, Gustavo Soares, and Ashish Tiwari. Grace: Language models meet code edits. In Proceedings of the 31st ACM Joint European Sofwware Engineering Conference and Symposium on the Foundations of Software Engineering, 2023. URL http:::///.....r///.........
3616253

---

Published as a conference paper ICLR 2026
Qisheng Hu, Kaixin Li, Xu Zhao, Yuxi Xie, Tiedong Liu, Hui Chen, Qizhe Xie, and Junxian Hc. Instructcoder: Empowering language models for code editing, arXiv preprint arXiv:2310.20322 2023.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical report. ArXiv preprint, abs/2409. 12186, 2024. URL https
7/arxiv. wwt.vvi.og///ba... 2409. 12186
Maliheh Izadi, Jonathan Katzy, Tim van Dam, Mare Otten, Razvan Mihai Popescu, and Arie van Deursen. Language models for code completion: A practical evaluation, 2024. URL abs/2400 97
org/
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07972 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint 10.02:2310.06770, 2023a.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2023b.
Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy, Inferfix: End-to-end end program repair with llms. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023. URL
https::/doo http:///ooooog///.// 1145/115566 /661 643 061386199
Harshit Joshi, José Cambronero Sanchez, Sumit Gulwani, Vu Le, Ivan Radičck, and Gust Verbruggen. Repair is nearly generation: Multilingual program repair with 1lms. In Proceedings of the Phirty-Seventl AAAI Conference on Artificial Intelligence and Thirty-Fift Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium Educational Advances in Artificial Intelligence, 2023. URL
1609//a............../.am v3714. 0177144.7 25642
Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. arXiv preprint wrXiv:2303.03000 2023.
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. Spoc: Search-based pseudocode to code. Advances Neural Information Processing Systems, 32, 2019.
Jia Li, Ge Li, Zhuo Li, Zhi Jin, Xing Hu, Kechi Zhang, and Zhiyi Fu. Codeeditor: Learning to edit source code with pre-trained models. ACM Transactions on Software Engineering and Methodology, 2023.
Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan. Automating code review activities by large-scale pre- training. In Proceedings the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ESEC/ESE 2022, pp. 1035-1047, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 97814503941 30. http:::/dol..ogg/0.11115115 0tt.://1..111//555551000 354988
doi: 10. 00.1144//544255.3544444
URL
https://don www.:/oo..rr......
12

---

Published as a conference paper ICLR 2026
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2023.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie LIU. CodeXGLUE: machine learning benchmark dataset for code understanding and generation. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Truck (Round 1), 2021.
URL https: /ttp///peeeeeeeeeeeeeeeee ew o
Meta. The Ilama 4 herd: The beginning of a new era of natively multimodal ai innovation. ht / ai meta. com/b b og/ htt/lllma-- m-mu l modaal weneeellgennen April 2025. Introducing Llama Scout, Ilama 4 Maverick, and a preview of Llama 4 Behemoth, which represent Meta's first wpen-w natively multimodal models with unprecedented context length support built using a mixture-of-eeppets (MoE) architecture.
Mistral. Upgrading agentic coding capabilities with the new devstral models. https::/misst mirtaaaall laii news htt//ddvvtrrlaa221 2507] June 2025. Introducing Devstral Medium, as well as an upgrade to Devstral Small.
Seungjun Moon, Yongho Song, Hyungjoo Chae, Dongjin Kang, Taeyoon Kwon, Kai Tzu-iunn Ong, Seung- won Hwang, and Jinyoung Yeo. Coffee: Boost your code llms by fixing bugs with feedback. arXiv preprint 231.v22311.07215, 2023.
Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. The Twelfth International Conference on Learning Representations, 2023.
Niels Mündler, Mark Niklas Müller, Jingxuan He, and Martin Vechev. Swt-bench: Testing and validating real-world bug-fixes with code agents, 2025. UBL.Lttpp:://rrxivoorr//aass
12952
Daye Nam, Ahmed Omran, Ambar Murillo, Saksham Thakur, Abner Araujo, Marcel Blistein, Alexander Frömmgen, Vincent Hellendoorn, and Satish Chandra. Prompting llms for code editing: Struggles and 2025.
remedies. arXiv preprint
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL hattps://opp.reett net Sttt/oooumll@=saac.ee...
Theo Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. Demystifying gpt self-repair for code generation. arXiv preprint arXiv:236600896, 2023.
OpenAl. Introducing openai o3 and 04-mini, April 2025. URL https:// cow.index. Introducing-on o3 and 04 1oimmmii// Accessed: 2025-05-15.
Qiwei Peng, Yekun Chai, and Xuhong Li. Humaneval-xl: A multilingual code generation benchmark for cross-lingual natural language generalization. arXiv preprint 2402 2024.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint a30iv:2308. /2950, 2023.
13

---

Published as a conference paper ICLR 2026
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. /tt.atttpp///geeeeeiiieee.///ooo
5 Team. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025a. URL https://arxiv orq atq/abs/25083 05477
Gemma Team.
Gemma 3 technical report, 2025b. URL htl.jttpp.//aaav..orr//////22...
Kimi Team. Kimi k2: Open agentic intelligence, 2025c. URL https://axxi..rgg/
20534
Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, et al. Recode: Robustness evaluation of code generation models. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 18818-13844 2023.
Yuxiang Wei, Chunqiu Steven Xia, and Lingming Zhang. Copiloting the copilots: Fusing large language models with completion engines for automated program repair. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023. URL bttps::///oi. org.
wttp:///////....0........1. 00000/.115/3888 643 36162766 6271
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- -Zvv, Neel Jain, Khalid Saifullah, Siddartha Naidu, al. Livebench: A challenging, contamination- Ilm benchmark. arXiv preprint arXiv:2406. 19314, 2024.
Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Shuiguang Deng, al. Codescope: An execution-based multilingual multitask multidimensional benchmark for evaluating Imms on code understanding and generation. arXiv preprint arXiv:231 1.08588, 2023.
John Yang, Carlos E Jimenez, Alex I. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, et al. Swe-bench multimodal: Do ai systems generalize to visual software domains? arXiv preprint arXiv:2410.0388 2024a.
John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36, 2024b.
Ming-Ho Yee and Arjun Guha. Do Machine Learning Models Produce TypeScript Types that Type Check? In Proceedings of the 37th European Conference Object-Oriented Programming, 2023. URL
https://dol. wtt./.ooorrloo... http:/11..8855000 48550/ awtpv.rxiiv...... www.xii..30.... 12163
Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et al. Multi-swe-bench: A multilingual benchmark for issue resolving. arXiv preprint arXiv:2504.0m 02605. 2025.
Jiyang Zhang, Sheena Panthaplackel. Pengyu Nie, Junyi Jessy Li, and Milos Gligoric. Coditt5: Pretraining for source code and natural language editing. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ASE '22, New York, NY, USA, 2023a. Association for Computing Machinery. ISBN 978 450394758. doi: 10. 14115/55511449.5565555 1/c6l..or./.0..1155555133323333
URL https
14

---

Published as conference paper ICLR 2026
Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. Self-edit: Fault-aware code editor for code generation. In Proceedings of the sl Annual Meeting of the Association for Computational Linguistics ( (Volume I: Longg Papers), 2023b. htt.pp:://aaaaachoooooyyo...
acl-lonng .5
Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipimeni, and Chandan K. Reddy, Xlcost: A benchmark dataset for cross-lingual code intelligence, 2022. URL https: pttpps://ariivi org /abs/ 2206. 08474
15

---

Published as conference paper ICLR 2026
A DATA COLLECTION DETAILS
A.1 SYSTEM DETAILS
We adapt the prompt used template from Continue (Continue Dev 2025
The has requested section of code in file be rewritten.
This is the prefix of the file: """ [language) (prefix)
This is the suffix of the file: "* (1anguagge) (suffix)
This is the code to rewrite: "" ("language))
(code_to_cdit)
You are expert programmer. You will rewrite the above code to do the following:
Keep in mind indentations. Output only code block with the rewritten code:
A.2 GENERAL INSTRUCTIONS
Step 1: Install the extension and restart Visual Studio Code after installation. If installed successfully, you will see EditBenchExt show up on the bottom right corner of your window and the check mark changes to a spinning circle when completion is being generated, Note, if you are using any other completion provider (e.g. Github Copilot), you must disable them when using EditBenchExt..
Step 2: EdittBnnchExt currently supports main feature: read autocomplete and in line editing (beta) below to understand how to use each one. Since we show paired responses, the way you use them are slightly different than your standard AI coding tools!
Step 3: This step is optional. If applicable, you can change what data is saved by EditBenchExt by following the instructions in "Privacy Settings".
Step 4: Create a username by clicking the EditBenchExt icon on the sidebar; detailed instructions are also "Create account".. Your username will be used for future leaderboard to compare individual preferences.
16

---

Published as conference paper ICLR 2026
A.3 PRIVACY INSTRUCTIONS
Privacy Settings. Your privacy is important to us. Please read carefully to determine which settings are most appropriate for you. To generate completions, the code in your current file is sent to our servers and sent to various API providers. This cannot be changed.
Data Collection. By default, we collect your code for research purposes. You can opt-out of this. If you working code containing sensitive information, recommend that you opt out of data collection. To opt-out of data collection, please change codetrivacySettings to Debug. We will only log your code for debugging. To disable logging entirely, please change codePrivacySettiinn any bugs you encounter will be non-reproducable on our end. You can find these settings by searching for EditBenchEEx in your vscode settings clicking the gear button of the EditBenchExt extension Extension Settings.
Private. Opting-out means
Removing your data. If you would like have the option in the future for delete any of your data, you must create an account on EditBenchExt following instructions described in "Create an account." To remove your data, you can email any the EditBenchExt maintainers with your username.
Data Release. Prior to releasing any collected code snippets to enable future research efforts, we will run PII detector and remove any identified entities further ensure no personal information is released.
17

---

Published as conference paper ICLR 2026
B EDITBENCHEXT INSTRUCTED EDITS DATA ANALYSIS
We analyze the in-the-wild data collected through EditBenchExt. We visualize the distribution of languages that users code (Figure natural languages that users write instructions in (Figure 7 length of instructions (Figure . length of highlighted code (Figure ( and length of code context (Figure IOP. We find that across user votes, 50.8% voted for the left response, 34.6% voted for the right response, and 14.6% voted for neither. This means that 85.4% of the time, at least one of the responses
accepted.
We also provide additional examples of user instructions across different task categories Table 5 and the full instructions from Table 2 in Table 4 An example of highlighted user code is given in Figure
Distribution file types
python
javastript html
typescriptreatt
typescript
shellscript
markdown
javascristreact
plaintext
yaml
File Type
sveite
css
latex
java
go
ahk2
csharp
dart
jsanc
bat
powershell
lua
rust
200
400
600
800
1000
1200
1400
Figure 6: Distribution of file types over instructed edit users in EditBenchExt. The majority of users are working on Python code.
18

---

Published as conference paper ICLR 2026
Distribution of Languages
English
Russian
French
Romanian
Italian
Chinese (Simplified)
Welsh
Catalan
Portuguese
Langunge
Spanish
Japanese
Slovenian
Danish
Bulgarian
Estenian
Norwegian
Indonesian
Korean
Swedish
Dutch
250
500
750
1000
1250
1500
1750
Count
Figure 7: Distribution natural languages in user instructions for instructed edits in EditBenchExt. The majority of users write instructions in English.
19

---

Published as conference paper ICLR 2026
Distribution of Code to Edit Taken Length
1000
Frequency
800
600
200
0
500
1000
1500
2000
2500
Code to Edit Taken Length
Figure 8: Distribution of highlighted code (also referred to as code to edit) token lengths. Users do not always highlight code. However, still know their cursor placement.
20

---

Published as conference paper ICLR 2026
Distribution of User Input Length
200
150
Requency
50
50
100
150 Length
200
250
Figure Distribution of the number characters user instructions.
21

---

Published as conference paper ICLR 2026
Distribution of Srefix+ Token Length
1750
Prqueecy
1000
750
500
250
250
500
750
1000
1250
1500
1750
2000
Length
Figure 10: Distribution of context length (defined by the prefix and suffix token length) of code files.
22

---

Published as conference paper ICLR 2026
Table 4: Full examples of user instructions across different task categories, comparing with two edit-related
datasets (CanItEdit (Cassano al. 2023b) and EditEval (Hu al. 2023)).
EditBench (proposed)
CanItEdit
Cassano et
2023b
EvitEval et 2023
Feature Addition
take the globe countries layor from below 11/ this and add it ta the existing globe
Add method lestimate, locat ion' that returna the estimated the sppropriate lucation for this house, calculater culated by getting the average location of the top 5 most similar houses in terms ustimated price.
Add functicn "iiltes.dddinumees" to filter odd numbers using lambda function.
Feature Modification
do not use B style, use python style
Flip the correlation function given to calculate the covariance instead using the Corr Y), Var and Var (?). function (orr Y), Var (X) and Var that order.
Modify the function to correctly determine the based month and day, considering edge cases for season changes. when invalid month is provided.
errer
Resolve Errors
Runt meRrror: Cannot c' ose a running event lcep sys:l: Runt imeWarning: coroutine 'Applicon iooiiiatiinn..oo ieaiiiinn.huttoonnn
Pix combinat ionJul rep ()
Fix the given fanction to correctly identify whether string zepresents valid floating-poing number or not, including handling edge cases such as scientific notation (e.g., '1a-4",, positive and neqative signs, and leading/trailing whitespace. Ensure the function is robust and handles exceptions appropriately.
so that it returns the right result. The function combination.unll www.uniimieddrreeee.. rep should returning the combination of n-z+1 and by calling combination arquments.
awalted sya:l: Runt imeWarning: coroutine 'Application imitialize was
those
never awaited
Optimize Code
optimize the computation by better batching the latter part
Optimize the bm25 algorithm by avoiding frequency calculations.
Optimize the fuaction to find the longest common subsequence for the given two sequences using dynamic progremming
23

---

Published as conference paper ICLR 2026
Table 5: Additional examples of user instructions across different task categories, comparing with two edit-related datasets (CanItEdit (Cassano et al. 2023b) and EditEval (Hu et al. 2023
EditBench (proposed)
CanItEdit Cassano et al. 2023b
Editlival (Hu et al. 2023
Feature Addition
example usage
Add mothod called 'header' which returns the header of cav file as a list.
a chock None prevent possible mull reference exceptions 'editorial reviews' function.
Feature Modification
mod1 Fy the amap so the displayed values are the as the zext displayed on the map.
Hodify the 'Quiz' class allow the skip a questien uaing skel..ssip.uusttinn and record the rumber of question ions that were skipped in seelf.skippes.
Modi fy the funct to return the word with the number of occurrences in the given strings. multipla words with the same maximum occurreaces, retarn all of them list sarted alphabetically.
are
Resolve Errors
theta alpha gradient ValueError: non non-broaden boo-bbroadastable output operand doean't match the broadcast shape (2,
Pix the mathods 'Course' * that they never throw errors. Rven when faansse.sssde 0', Instead they should return 'Hone'. Additicnslly, the words "Fur', "will', or 'map' anywhere code, You should accomplish this uaing higher order
Fix the function correctly find the single element in sorted array where every element appears exactly twice.
shape (2,
do
Optimize Code
run these in parallel
Optimize the AT to find the beat move in less steps.
Optimize the given function to find the firat position of an element in socsed array.
24

---

Published as conference paper ICLR 2026
Example EditBench brnginal_ood,,ppo
From langchain_gpende Import ChaLOpenAZ
from langchain. proompt import PrompeTemplate
from langchain, cheing import LiMchain
langchain_commane httpghain_.mmmnnnt..eeeeieeeeeee from os import geteny
import BM25Retriever
4 soma importa spacing
load_doteav()
st,title st.title ("CardioRaS")
4 load in PUP for
"retrievar* not in st.sesaion_atate:
st.text ("Loading EDE...")
prog_bar st.pragresa
pdr_reades Peddee (open ("Mosa and Adama 10e Vol 2.pdr", 1xb1))
chunks []
for page_num in range (60, 6001:
chunks. Saunks.sppend (page_num 80) / (600 60)
groo_bazrpproeee
(page_rumm .xtxratt_txxt
put chunks into vectoz stoze
retriever B#25Retriever. wew.verr.oottttteeee (churks, metadatas= ("page__un p in range (EO, 600) preprocess canrrocer__uuccewwrr.... (ann=woer_ttkknniee
at.sessioo_state ("rattterrrr retriever
Lext (*Loaded PDF")
"messages" noz in st.seasion_state:
st.seesion_etate ("meste(mmesagees)
"pole": "assistant", "contens"; *11, I'a chatbot who has read the Moss Adams cardiology textbock. ноw help you?"
st. for.formmchttt input", cleer__o__uubbtssrrro)) a,b st.oolum ([4,1]
user_ _eer_inpet = a.taxt_inputt
tabel="questions""
placeholder="Watt is the incidence of congenital heart disease?", bbbel__iiibiilitt"===olpppeee
b. borrm_aubbmls wtw__aabni=_btoonnnooe buttonn""Seee"" wwa_oottannerrwidhhhrrro
Ior i, enumera.e wet..ssesion_sseeeeeeeee.e [essioon_sate_mmeeaaess)..
message ("conteentt ie_user=mss[==lee"
"user", key=str(E))
user_input st.aession_stede [ta_atteeepassoos"
wt..ession_ssate.nee seaate.mmeeages... ("role""" *uses", "content": user_input)
message (sser_ings)))) is_user=irue, sesttastssttttttssioo.....
11)
11m ChatOperAI
base base_ur]= http"tttt////ppooooreee.tss httpp////r.///////////// streaming=True)
retriever at.seasion_stttee (tatt___ttttteretteeeeee"e"
docs - setriever.get_.eeevaatttt
(user_ingus
DIVIDER n_*,10
context (eViIERR.oii((fPMM d.metadota ('pago_num d.pago_contend for d in docs])
prompt PromptTemplate:
vauu___erialee= ("context", "questier"),
template=n**Yo are a helpful AT asaistent who has read the Moas Adams Cardiology textbook,
following just you don'= know. Context: context
queation. you don't answer,
Question: questior "*e
print (prompt)
25
L.Mchain (1lm=1im, promppt
response chain. (context guestion-user_ingus
at saasionn statel ('massages") append ("role"; *aasistant" "content": responsel
measage (response, key=str (enetr((le (len (st. cenesston_tttteeeeeeeee
1)
Example code file and highlighted section. The instruction for this file: "Can you edit this work with streaming responses?"
Figure 11:

---

Published as conference paper ICLR 2026
C EDITBENCH DETAILS
Data Curation and Programming Languages. We started with 999 Python and 234 Javascript files. We curated (Phase 1) down to 370 Python and 100 Javascript files. We then successfully tested and annotated 104 Python and 9 JavaScript problems. React represents its own ecosystem in Javascript; 5 out of 9 of our problems are based on React.
Library Distribution. EditBench contains 74 unique imports for Python (Figure 3). We also calculated distributions for CanltEdit (25 unique imports), Polyglot (15 unique imports), and EditEval (16 unique imports) (Figure 12
collections
copy
heapa
random string
scpy
aet
numpy
autograd
(a) CanItEdit library distribution
stfing tertooss
operator future
Joon
functoars
inspect
random
collections
(b) Polyglot library distribution
collections
dataclasses
sys logging
typing
watorms
numpy
uild
unilb
pathilb
math
heapa
(c) EditEval library distribution
Figure 12: Library distributions for comparison benchmarks: CanltEdit (25 unique imports), Polyglot (15 unique imports), and EditEval (16 unique imports).
Problem Filtering. When filtering in-the-wild data, discarded problems that were too easy and too ambiguous. Examples of problems that are too easy:
Given instruction "decrease the speed" and the highlighted code snippet that clearly includes self master....... after after (30, self w.f..aame_i game-loopp) Adjust speed here (milliseconds) . is obvious that the change trivial, as it just involves increasing the value of the hard-coded value.
Given the instruction "add api key" and the highlighted code snippet chat_model ChatOllame (model poodeel""iama" 2", pese_urr="http" "bsseuuribttt::/llooe http:::///oaallht:1111300.1 It is clear that the change would simply involve adding an api key parameter.
Examples of problems that ambiguous:
26

---

Published as conference paper ICLR 2026
Given the instruction "find and solve problems" and code context consisting of dozens of lines of Python code to instantiate an ML training pipeline with no obvious issue. From the annotator's s perspective, it unclear what problem the user was intending the LLM fix.
Given the instruction "The code does not seem implement all the logic please extend make all logic work." and a short highlighted snippet (e.g., list_avail aallereesolut ions (yt) ) ) From the annotator's perspective, it is unclear what "logic" needs be implemented and the highlighted code provides insufficient context.
27

---

Published as conference paper ICLR 2026
D EVALUATION SET-UP
Prompts. We evaluated models using three prompting strategies. The Whole (i.e., +Highlight) prompt tasks the model with regenerating the entire code file (Figure (3). The Cursor Position (i.e., +Highlight, +Cursor) prompt is the same the Whole prompt with the user's cursor position added (Figure [4) (i.c..
Code Only). The No Highlight prompt is the same as the Whole prompt but information about user's highlights removed (Figure 15) Note that the prompt in Section A is slightly different due to the settings (e.g., cost, response time, etc.)
Model Access We used the OpenAl API to query the GPT models, the Anthropic API for the Claude models, and OpenRouter for access all other models. The full list of official model names and links to the providers is in T
Model Parameters. For every model provider, the default settings were used. The gpt-os co-minni gpt -onniinn and gpt-5 models used the default medium effort for reasoning while gpt-o3 13mini (high), gpt-o4-mini (high), and gpt-5 (high) used the high reasoning effort setting.
Evaluation Environment. To isolate our testing environment, we ran all our evaluations inside of a Docker container. We used the Ubuntu 22.04 image for our container. The Dockerfile for building our container will be provided with the release of our benchmark.
Generate a new implementation of the following code based on the user instruction:
The Original code (to be modified):
" "[lang] (original code)
The user instruction is:
(instruction)
And they highlighted this section to be changed: "' [lang]
(highlighted_code))
Please only change the highlighted section and leave the rest of the code unchanged. Please output the entire code file. Respond only in a code block beginning with ". (lang).
Figure 13: Whole prompt given to models
28

---

Published as conference paper ICLR 2026
Table Each model in our experiments with their official names and provider links
Model
Model Size
Proprictary
to Provider
gpt 40-mini
Unknown
True
https//plattorm aepfoor.openaiii.....o..o
gpt
Unknown
True
https:/platfformopos oop.oo.mooooiii....o.oo.ooo...ie com//ocs/moooel//gtt--
qpt---nano
Unknown
True
https:/pllttforr atform. wew...ii........................ Som///ee//mmdell/pt..--...
gpt 5 mini
Unknown
True
https//platform.co wnfforoooonnii.....o com/Goes/mooell//gtt
gpt-5
Unknown
True
http::///atffmoooo/o oeemm.iiii.c...mmm caccaaicco//ee com/docs/nodels/gge
gpt mini
Unknown
True
https https:/pllttiin wttformmmmm..mmmmommmmmomoooo openai. com/@acs/moodel/oo-mmiii
spt-od-mini
Unknown
True
hupp://platfoom www....peii.oommm com//ocs//noddll/gtttooooo
opt-o8s-206
20h
False
https:/pllttiie wttpotormm....m...omomooo openai. com/ com/face//mmddll//gtoooss///
gpt EEC-o98-12
1206
False
https https///patfrroopo.o copiooooooppiii...ooo com/ /om.dooo///oooll/totosss---oss
sonnet-3.5
Unknown
Tiue
htpps//doss..o anthropic.ccooo hwtpr/oppc....o/////eeee www//e//doss/aboou chaub//modeesovvervieew
sonnet-3.7
Unknown
True
https///dcss.. anthropic. www//n//toss/aahuuli whw.eeeeieeeeeeeeeeeeeeeeeeee
Unknown
Tiue
hitps//doce wwt.ttooi..commmeee www//////oc///bbe//////ooo/////e claud//modelssooervieee
g1m-4.5
355b
False
hiiss hhaps///pemrouutto ailz wwwzaaii/allmm----- ai/glm-4.5
gemma- -a-ma--n--4b-it-
8b
False
http:://openrouterr grgoogglgggoooooomm. 3n en--4b-it
gemma-3-12b-1 3-12b-11
12b
False
https://openroutett
weg.g//mm/./----. 3-1bbit
gemma-3-27b-it
27h
False
htpp::/oppemrruuerr.a go wlwgooolllggmngggn 27b
qemini-2.5-flas.
Unknown
True
https://openrouueerr
2.5-ulushh
gemini-2.5-pro
Unknown
Thue
https://opennrouteraaaiii wtwp://wtttetttioooooooooooooooo wew.gggggmgggggggiiiiin.oo.. legeminiiii-
pro
grok-4-fast
Unknown
Tiue
https://opeemrouter...tt ai/ fast.free
grok code -ast--
Unknown
True
ximi k2
False
https http:://peemmoucccca/omoomoaaoas
32B
False
72B
False http:://ppeauoouccc.iooov//eeeee 2.572-i--tn---
False http:://eeeooooouucccooeeeeeeeee
qwen-3-4b
qwon-3-8h
8B
False https://qeemoouuee..uqqqee//eaeu
qwen-3-14b
14B
False http:///opauruuucc.iiqqeeeeeeeee
Gwen-3-30b-ubb
30B
(wen-3-coder-ffe
Unknown
True
qwen-3-cuder
405B
False
deepseek-v3-chat
deepseek-r1
Unknown
False
1lema-O-raverick
Unknown
False
https///peannuuee..iiimm aimeta htw.://wnnnnallaaaaaaaaaaaaaeeee
11ema-4-scout
Unknown
http:://qpenrouter.aaii Haaaa/laaaa scout
12ama-3.1-405B
405B
False
http:///peerroutee.cc aimeta lww./laaaaaaaamaaaaaaa 1-405b
11.ma-3.3-703
70B
False
anne 3aam://lama-33-7700-nnnunuu
3.3-
11ama-3.3-8b
False
mistralai devatral small
24B
False
mistralaai-dovatraa-meel
Unknown Unknown
True
mistralal-codestan
True
mistral-small3322-11 instruct
24b
False
29

---

Published as conference paper ICLR 2026
Generate a new implementation of the following code based on the user instruction:
The Original code (to be modified):
" [[lang] (original.code)
The user's cursor position (line number: column number) is at {cursor pos}
The user instruction is:
(instruction)
And they highlighted this section to be changed:
" "[lang]
[highlighted.com }
Please only change the highlighted section and leave the rest of the code unchanged.
Please output the entire code file.
Respond only in a code block beginning with (lang].
Figure Cursor Position prompt given to models
Generate new implementation of the following code based on the user instruction:
The Original code (to be modified):
" "(lang] (original_code)
The user instruction is: (instruction]
Please output the entire code file.
Respond only in code block beginning with "* [lang].
Figure 15: No Highlight prompt given to models
30

---

Published as a conference paper ICLR 2026
E ADDITIONAL EVALUATION RESULTS
Effect of context length. We also conduct additional analysis by binning performance into short, medium, and long. Perhaps unsurprisingly, we see that models tend to better on shorter context length problems. general, the worse model is overall, we also that it has much larger gap between the best and worst bin (c.g, gemma- -n---n-i--iiii- it has a 34.2% gap and opt-oss- 1 20b a 33.6% gap).
Table 7: Effect of context length on average pass
Context Bin
Average Pass
Short (i.e., < 1 chars)
71.03 + 7.60
Medium (i.e., 1k-3k chars)
62.09 + 8.56
Long (i.e., > 3k chars)
59.94 + 10.43
Instruction and Highlight Length analysis. Given the large gap between easy and hard problems, we explore what types of prompts present in hard problems compared to the general dataset. As shown below, we see that hard instructions tend to have shorter instructions (by nearly times) but slightly longer highlighted code.
Table 8: Comparing instruction and highlight length for easy versus hard questions.
Instruction Length (chars)
Highlight Length (chars)
Easy Questions
351.21 + 1018.87
942.30 + 1275.35
Hard Questions
75.09 + 107.20
881.45 + 1275.23
Code Context Dependent Example. Additional code context is often crucial to understanding and solving a problem. This can be because the code context is simply too long because the user instruction is too ambiguous. Let us take problem 45 in EditBench as an example.
In this example, the user instruction is to 'remove', which could mean the removal of the class, the function, or the implementation of the remove functions. However, when observing the problem we can consider the following from the rest of the code context:
1. There is highlighted code segment. This means it is impossible for the intent be removal as the only available operation is to add code.
2. The remove vertex and remove edge functions appear multiple times the code. However, the function implementations implemented incorrectly in the original code.
Thus, the other interpretations of 'remove" make little sense given the entire context of the problem. The correct answer can be inferred from the rest of the context, but would be difficult to understand from the instruction alone.
31

---

Published as conference paper ICLR 2026
F FURTHER DISCUSSION
F.1 LIMITATIONS
addition to the limitations in Section 6 we discuss several more below:
Limited Programming and Natural Languages. Although EditBench contains problems in both Python and Javascript as well as several non-English languages, the amount of Javascript is limited. We aim to continue collecting more data and building test harnesses and problems for more programming and natural languages.
Contamination. One major challenge with releasing benchmarks is that future models may accidentally (or intentionally) be trained on the benchmark itself. We have taken pre-emptive measures to prevent this by ensuring the dataset documentation contains instructions to prevent any accidental scraping of our data. Following recent benchmarking efforts (White al. 2024 Jain et al.) 2024), we will also aim make pipeline more automatic. Combined with the continuous stream of data from EditBenchExt, new problems can be continuously released, preventing data contamination. We discuss this more detail in Section F.2
F.2 FUTURE WORK
In addition to increasing the number of examples for the existing languages and expanding to other common programming languages, we plan to continue updating the EditBench leaderboard as new models are released.
Automatic Test Harness Generation When we evaluated our fully-agentic pipeline on our model generations, all models achieved passêl 1 of 0%. This indicates that these test cases were either broken too constrained to be usable in the benchmark. Given that prior research indicates models are at least somewhat capable of generating well-specified test cases (Mündler al.) 2025), suspect that models still unable fully understand the intent behind in-the-wild user instructions. Given that we have a continuous stream of data from EditBenchExt, resolving this will be key to enabling fully automatic test harness generation for EditBench. In general, we also believe that improving an agent's "e ability to generate test harnesses constitutes an interesting avenue for future research.
F.3 BROADER IMPACT.
This paper presents work whose goal is to advance the field of Machine Learning. Due to the ethical and user privacy considerations involved with storing and releasing user code data, we take a conservative approach to data release. Despite giving users full control over their privacy, have least two annotators who provide additional screening for Personally Identifiable Information (PII) on each problem during our data curation and release process. We will continue screen for PII release more problems.
32


Reviews:
[{"review_id": "ItIFVf52t1", "reviewer": "Reviewer_yRbL", "summary": "This paper introduces EditBench, a novel benchmark designed to evaluate Large Language Models' (LLMs) capabilities in instructed code editing. Unlike existing benchmarks that rely on synthetic or competitive programming problems, EditBench's key contribution is its data, which is sourced from real-world developer workflows. Data was collected via a custom VS Code extension from nearly 500 users, capturing user instructions and code context in the wild. The benchmark comprises 545 problems spanning diverse real-world use cases, from bug fixes to feature additions. EditBench is the first to systematically require models to leverage rich contextual cues—including the user instruction, the entire code file, highlighted code snippets, and cursor position—to solve problems, thus more accurately simulating a developer's environment. It features diversity across 5 natural languages (including Chinese, English, etc.) and Python/Javascript programming languages. The authors evaluated 40 different LLMs, finding that EditBench is highly challenging; even the best-performing model (claude-sonnet-4) achieved only 66.67% pass@1. The study further demonstrates the significant impact of contextual information and task categories on model performance.", "questions": "Q1: The paper notes that models perform best when given highlighted code but not the cursor position, and that context can affect performance by up to 11%. Could the authors provide a more detailed ablation study in the appendix or rebuttal, clearly listing performance metrics for all four combinations: (a) No extra context; (b) Highlighted code only; (c) Cursor position only; (d) Highlighted code and cursor position. This would clarify why the cursor position appears to have a detrimental or unhelpful effect and quantify the exact utility of each cue.\n\nQ2: Given that non-English problems in EditBench-complete were generated via GPT-4o translation, what specific quality control steps were taken to verify the naturalness and preservation of the original user intent in these translations? Were the non-English problems (e.g., Chinese, Russian, etc.) reviewed by native speakers to ensure quality?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1: The primary strength is the in-the-wild data collection methodology, which captures non-templated, natural user instructions and context from a real IDE environment, reflecting authentic developer challenges. \n\nS2: The use of a rigorous process involving human experts and secondary review to create the test suite addresses the inherent difficulty of creating a verifiable benchmark from raw user behavioral data, ensuring the high quality and trustworthiness of the results.\n\nS3: The design explicitly tests the models' ability to integrate contextual cues (file content, highlights, cursor), which is shown to be crucial for performance, thus providing valuable insights for future LLM development.", "weaknesses": "W1: EditBench currently supports only Python and Javascript. For a benchmark aiming for real-world representation, the lack of support for other major industry languages like Java, C++, Go, C#, and TypeScript limits its broad applicability.\n\nW2: The extended version, EditBench-complete, relies on GPT-4o for translating non-English instructions and comments. While this increases linguistic diversity, the paper lacks detailed evidence of human quality validation for the LLM-translated content. This raises a concern about potential loss of naturalness or subtle nuance of user intent in non-English problems, potentially diminishing the \"in-the-wild\" authenticity of this subset of data.\n\nW3: The paper mentions code context lengths can be long (average file length 4.5k tokens) but provides minimal dedicated analysis on model performance differences concerning context length. Given that LLM performance on long-context tasks is a major research area, a specialized analysis contrasting performance across different context length bins (e.g., short, medium, long) would have been highly valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ODQ31YVLRk", "reviewer": "Reviewer_CbSq", "summary": "This paper introduces a new benchmark, EditBench, for evaluating LLM code editing capabilities grounded in real-world usage, i.e., user instructions and code contexts collected in the wild. EditBench comprises of 545 problems, multiple natural and programming languages, and a diverse set of real-world use cases, ranging from resolving errors to adding features. 40 diverse LLMs have been evaluated and only 3 models score over 60% on EditBench, indicating it is a challenge set of problems. Also, empirical results and analyses show that, varying levels of contextual information greatly affect task success rate.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a new benchmark EditBench for evaluating LLM code editing capabilities grounded in real-world usage.\n2. Only 3 models score over 60\\% on EditBench, indicating it is a challenge set of problems.\n3. The presentation is good and the paper is easy to follow.", "weaknesses": "1. No codebase is released or provided to confirm the reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pf4nwAEX7K", "reviewer": "Reviewer_aKBn", "summary": "The paper presents a benchmark for code editing, where code editing\nmeans working with a VSCode extension and using a sidebar LLM chat window\nto edit code. The paper recruited 500+ participants to use a VSCode extension\nthat was developed by the authors. They received  appropriate IRB\napproval and take care to manage PII.\n\nThe paper then puts in the effort to turn the raw data into a benchmark\nsuite with test cases. The benchmark is quite challenging and much more\nrealistic than prior work in this space.", "questions": "- I don't quite understand Appendix A.1. This looks like the system prompt\n  for editing. What does \"collecting user votes\" means? Actually, section D\n  makes it clear that this is not the prompt template for the model, so I\n  am quite confused.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I think this paper sets a new bar for coding benchmarks. Almost every other\nbenchmark that I can think of for any coding tasks (and not just code editing)\nuses artificial tasks and prompts. The only related work I can think of that\nuses natural prompts is the StudentEval benchmark (Babe et al, 2023), but the\ntasks in that are quite trivial compared to what is presented here.\n\nI have gotten quite tired of evaluating LLMs on made up tasks that I know are\ntrivial compared to the kinds of prompts that get used in the wild. This paper\nhelps change that. I would personally use this benchmark in my own work. It\nis easy to see that other researchers would also find it very useful.", "weaknesses": "- The only weakness I can think of is that the context provided to the\n  models seems to be just the current editor window. Tools like Cursor\n  are well beyond this and supply significantly more context. However, \n  I think this simplification is the right one for a benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KeyzTfMpvx", "reviewer": "Reviewer_SuE5", "summary": "The paper presents EditBench, a benchmark designed to evaluate large language models (LLMs) on real-world instructed code editing, where LLMs are guided by natural language to modify existing code. To address the lack of realistic interactions, the authors built a VS Code extension that collects in-the-wild user instructions, highlighted code, and cursor positions from nearly 500 developers. After filtering and manual annotation, they curated 545 problems across five natural languages (English, Spanish, Russian, Chinese, Portuguese) and two programming languages (Python, JavaScript), each paired with test harnesses for automated evaluation. Evaluating on 40 LLMs show that EditBench is highly challenging where only 3 models (Claude-Sonnet-4, GPT-o3-mini-high, Claude-3.5-Sonnet) exceed 60% pass@1. Including highlighted code improves performance by up to 7%, while cursor position lowers it.", "questions": "1. What specific criteria were used to determine which problems were considered “interesting and challenging”?\n2. How did the authors define and measure problem difficulty when splitting EditBench into easy and hard groups?\n3. What is the estimated upper bound of model performance when following the instructions in EditBench, and was any human or expert-based evaluation used to assess the clarity and followability of these instructions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. EditBench is collected directly from real-world interactions, which makes it more representative of practical coding workflows.\n2. EditBench incorporates diverse user instructions, code context, highlighted code, and cursor position.\n3. Multilingual benchmark across five natural and two programming languages broadens applicability.\n4. Comprehensive evaluation with 40 LLMs spanning reasoning and non-reasoning models.", "weaknesses": "1. The authors keep stating that they include \"both interesting and challenging (line 158)\" or removing \"ambiguous (line 178)\" problems, but there is a lack of explanation what problem refers to bad or good problem.\n2. In the evaluation, the authors split EditBench into easy and hard groups, but they do not provide clear criteria explaining how this division was made.\n3. Following up on the previous concern, the authors claim that “\\emph{hard instructions tend to have shorter instructions (by nearly 5 times) but slightly longer highlighted code}.” It is unclear whether the shorter instructions are compared to those in easy problems or to the length of the highlighted code itself. For either perspective, there is a lack of in-depth analysis to figure out the root cause.\n4. From the example shown in Table 5, the instructions collected by EditBench appear less clear than the annotator-provided prompts in other benchmarks. This may explain why all LLMs perform poorly with such instructions. It would be helpful to know what the upper bound of performance is when following these instructions.\n5. It is notable that older reasoning models such as o3 and o3-mini-high outperform GPT-5. A deeper analysis of this result would be valuable to understand why newer models lag behind in this setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)