Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions
Authors: Federico Cassano
Luisa Li
Akul Sethi
Noah Shinn
Abby Brennan-Jones
Jacob Ginesin
Edward Berman
George Chakhnashvili
Anton Lozhkov
Carolyn Jane Anderson
Arjun Guha
Venue: COLM (2024)
Keywords: benchmark, programming, llm, code editing
TL;DR: CanItEdit evaluates the instructional code editing capabilities of large language models, reveals a performance gap between open and closed models, and demonstrates that fine-tuning with a new dataset improves performance of open models.

Abstract: A significant amount of research is focused on developing and evaluating
large language models for a variety of code synthesis tasks. These include
synthesizing code from natural language, synthesizing tests from
code, and synthesizing explanations of code. In contrast, the behavior of 
instructional code editing with LLMs is understudied.
These are tasks in which the model is provided a block of code and an instruction to modify the code. 
The editing instruction may ask for a feature to be added or removed, describe a bug and ask
for a fix, or ask for a different kind of solution.
We introduce a carefully crafted benchmark of code editing tasks and use it
to evaluate several cutting edge LLMs. Our evaluation exposes a significant gap
between the capabilities of state-of-the-art open and closed models. For
example, even GPT-3.5-Turbo is better than the best open model at
code editing tasks. We also introduce a new, carefully curated, permissively licensed training dataset of code editing tasks
coupled with natural language instructions.
Using this training dataset, we show that we can fine-tune open Code LLMs to significantly
improve their code editing capabilities,
closing the gap between open and closed models.
All code, data, and models are available at https://github.com/nuprl/CanItEdit.

Published as a conference paper at COLM 2024
Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions
Federico Cassano
Luisa Li
Akul Sethi Northeastern University
Northeastern University cassano. fenor theastern.cc. edu
Northeastern University
Noah Shinn Northeastern University
Abby Brennan-Jones
Jacob Ginesin Northeastern University
Wellesley College
Edward Berman Northeastern University
George Chakhnashvili Northeastern University
Anton Lozhkov HuggingFace
Carolyn Jane Anderson Wellesley College
Arjun Guha
Northeastern University and Roblox a. guha@northeastern. edu
Abstract
A significant amount of research is focused on developing and evaluating large language models for variety of code synthesis tasks. These include synthesizing code from natural language, synthesizing tests from code, and synthesizing explanations of code. In contrast, the behavior of instructional code editing with LLMs is understudied. These are tasks in which the model is provided a block of code and an instruction to modify the code. The editing instruction may ask for feature to be added or removed, describe a bug and ask for fix, or ask for a different kind of solution. We introduce carefully crafted benchmark of code editing tasks and use it evaluate several cutting edge LLMs. Our evaluation exposes a significant gap between the capabilities of state-of-the-art open and closed models. For example, even GPT-3.5-Turbo is better than the best open model at code editing tasks. We also introduce new, carefully curated, permissively licensed training dataset of code editing tasks coupled with natural language instructions. Using this training dataset, we show that we can fine-tune open Code LLMs to significantly improve their code editing capabilities, closing the gap between open and closed models. All code, data, and models are available at https: / /ttrr/gttttt gith....c. com/nuprr/CanIttEo
1 Introduction
Large language models of code (Code LLMs) are becoming an essential tool for software engineering practice and research. There has been significant research on synthesizing code from natural language instructions, but comparatively less attention has been given to code editing tasks. However, LLM users expect models to be capable of editing code. For example, the LMsys dataset of in-the-wild conversations with chatbots (Zheng et al., 2023) has 4,188 conversations containing code, and 831 (19%) of these involve code editing, where the user prompts the model to update code based on natural language instructions (Appendix E). In general, code editing with an LLM encompasses activities like feature addition or removal, bug fixing, and code refactoring (Zhang et al., 2023b; Moon et al., 2023; Shinn et al., 2023; Chen et al., 2023; Olausson et al., 2023; Jin et al., 2023).
The ability to edit code is also essential for a model to be useful for an Al-focused code editor such Cursor (Cursor, 2023), Copilot Chat (Copilot, 2023), or ChatGPT Advanced

---

Published as a conference paper at COLM 2024
Instruction: Edit the C4 class and its methods to represent the C8 group instead.
-class (4 Module):
+class C8(nn. Module):
"""Represents the cyclic group C4,
"""epresentss the cyclic group
where each element represents discrete rotation. """
def (self:: super () -niitit_n
def size(self):
"""Outputs the size of this group.
return 4
return 8
def elements(self):
"""Returns all the elements of this group""*
return torch. tee. np. pi/2, np.pi, 3*np.pi/2]] pi/2])
d np.pi
return torch. tensor ((0., d, d+2, d+3, d+4, d+5, d+6, d+7])
Figure An abbreviated example code editing task from the CANItEDIT dataset (Figure 9 presents the full example). The model is tasked with editing the C4 group to represent C8 instead. The model expected to infer the after code segment from the instruction and the before code segment, shown in the inferred code diff.
Data Analysis (ADA) (OpenAI, 2023a). Cursor and Copilot Chat facilitate edits with human- written instructions. In contrast, ADA uses both human-written instructions and model- generated reflections (Shinn et al., 2023; Fan et al., 2023; Phung et al., 2023) extend and edit code. This approach represents step towards Al-driven code assistance. In both scenarios, instructional code editing is employed, which we define as a function M(c, I) - d', where the original code, I is the instruction, and c' is the modified code. Figure illustrates this process, showing how the model edits a code segment from a given instruction.
Model-generated reflections and human- instructions both describe desired code changes. However, they differ in the level detail: reflections, usually more detailed, are generated by model with access to the code, offering richer context and potentially strategic plan for code modifications. In contrast, human-written instructions are typically shorter and less detailed but may express the true user's intent more clearly. We refer to these as descriptive and lazy instructions, respectively. We thoroughly analyze examples of such instructions in Appendix E.
In this work, we introduce CANITEDIT, a novel dataset comprising 105 hand-crafted in- structional code editing problems, featuring both descriptive and lazy instructions and extensive hidden test suite. Designed to assess model's proficiency in handling diverse code editing scenarios, CANITEDIT serves as a benchmark for evaluating state-of-the-art Code LLMs in instructional code editing. Our evaluation focuses on measuring the accuracy of given model's ability write correct code modifications without introducing unneces- sary code. We conduct comprehensive assessments of closed and open models, revealing significant performance disparities between the leading closed and open models ($5). To help address this gap, we propose training dataset and methodology for code editing. Our findings demonstrate that fine-tum thn-tunnng open Code LLMs on this dataset can significantly enhance code editing performance (54).
To summarize, we make four main contributions: (1) We introduce CANITEDIT, a dataset of instructional code editing problems, designed to evaluate the code editing capabilities
2

---

Published as a conference paper at COLM 2024
of large language models ($3). (2) We propose novel metric, ExcessCode, for quantifying the typical volume of unused code produced by a model when generating the correct code edits ($5.1). (3) We perform thorough evaluation of the latest Code LLMs in the context of code editing, providing insights into their current capabilities ($5). (4) Finally, we present specially tailored training dataset for code editing, along with an effective training methodology, demonstrating significantly enhanced code editing performance through fine-tuning models of varying sizes (64).
2 Related Work
Instruction-folloiing Language Models. Correctly prompting an LLM crucial for it to perform a desired task. There are multiple methods for instruction tuning LLMs to better adhere to natural language instructions. One method involves employing human annotators to create sample instructions and provide feedback on numerous model outputs (Ouyang et al., 2022; Köpf et al., 2023). This method is costly and demands substantial resources. An alternative, cost-effective method is to use LLM to self-instruct, generating instructions from a smaller set of human-written seed instructions (Wang et al., 2023). These methods have been applied to generate datasets for instruction-tuning Code LLMs (Chaudhary, 2023; Luo et al., 2023). Specific to code generation, another strategy to instruction-tun is to use commit messages as instructions (Muennighoff et al., 2023). In this paper, we use commit messages as instructions for code editing. Our results demonstrate that while instruction-tunee models can edit code, they are not as effective as models that we explicitly train for this task ($5).
an LLM
Code Generation Benchmarks. Several benchmarks exist that test a model's s code generation ability. HumanEval and MBPP are two prominent benchmarks for evaluating LLMs in Pythón programming (Chen et al., 2021; Austin et al., 2021). MultiPL-E expands these benchmarks to 18+ additional programming languages (Cassano et al., 2023b). These benchmarks assess model-generated candidate completions against a series of human- authored unit tests. EvalPlus (Liu et al., 2023) utilizes mutation testing to expand the test suites of the Python benchmarks. EvoEval (Xia et al., 2024) addresses limitations in existing code generation benchmarks by evolving them into diverse domains. It reveals significant performance drops in LLMs compared to standard benchmarks, highlighting the need for more comprehensive evaluation methods. All of these benchmarks utilize the pass@k metric, which measures the likelihood of the model generating a completion that passes all of the tests in k tries; we also adopt this metric in our evaluation ($5.1). However, these benchmarks are limited to the evaluation of a model's s ability to generate a single function from a natural language description and do not assess code editing capabilities. HumanEvallacch ack (Muennighoff et al., 2023) is a benchmark designed for evaluating LLMs across various single-function and bug fixing. Specifically, HumanEvalFix, bug-fixing variant of HumanEvalPackk, is extensively used for assessing the models' capabilities in code refinement (Moon et al., 2023; Muennighoff et al., 2023). However, the instruction is fixed for every problem. SWE- Bench (Jimenez et al., 2023) evaluates LLMSs across varied programming tasks including planning, retrieval, and code editing. Our work concentrates specifically on code editing tasks, aiming to more precisely guide model development. Unlike SWE-Bench, which sources its problems from GitHub PRs and issues, our benchmark is handcrafted, reducing contamination risks as seen with models like StarCoder and StarCoder2, which are trained extensively on GitHub data (Li et al., 2023b; Lozhkov et al., 2024).
function code generation tasks, such as synthesis, code explanation,
Code Editing Using Large Language Models. Previous studies on code editing with LLMs have predominantly focused on bug fixing (Zhang et al., 2023b; Moon et al., 2023; Shinn et al., 2023; Chen et al., 2023; Olausson et al., 2023; Jin et al., 2023; Joshi et al., 2023; Wei et al., 2023; Li et al., 2022), a specific subset of code editing; fillin-the-middle code completion (Bavarian et al., 2022; Fried et al., 2023; Yee & Guha, 2023; Roziere et 2023; Guo et al., 2024a; Zhang et al., 2023a), an inference strategy that requires specific insert locations; and intrinsic code editing (Li et al., 2023a; Gupta et al., 2023), which involves editing code without a specified instruction, exerting the model's s ability to intrinsically ascertain the desired code changes. Recently, LLMSs have progressed in code editing guided by natural
3

---

Published as a conference paper at COLM 2024
CanltEdit Dataset Statistics
Total Tasks
105 /35/35/35
Total Problems
210 (70/70/70
Topics
Data Structures Algorithmss
39
Language Processing
21
Mathematics
25
Data Science
10
Miscellaneous
10
Problems With Library Usage
22
Code Segment
Mean 1: Std. Dev.
Mean Lines (Before) After)
42.5 33.9 49.8 36.6
Levenshtein Distance
302.1 1 339.6
Combined Mean Lines
92.3 + 69.9
Combined Mean Tokens
865.3 639.7
Combined Max Tokens
3,583
Instruction
Mean + Std. Dev.
Mean Tokens (Descriptive) Lazy)
81.7 50.4 35.6 30.6
Table Dataset statistics for CANITEDIT.
language without specific edit locations (Hu et al., 2023; Li et al., 2023b; Muennighoff al., 2023). However, this advancement lacks benchmark evaluations to effectively measure the models' code editing skills. Notably, StarCoder (Li et al., 2023b), the first LLM trained on an extensive dataset of commits using the format <before><commit message><after>> we have shown enhanced code editing capabilities ($5). Before this study, StarCoder's 's practical code editing performance had not been assessed. StarCoder2 has replaced commits with pull requests and issues, which typically include more natural language (Lozhkov al., 2024). CodeEditorBench (Guo et al., 2024b) evaluates code editing using competitive programming problems. Unlike CANITEDIT, it uses fixed prompts and lacks diverse topics and dual instruction sets. The recent introduction of InstructCoder (Hu et al., 2023), model explicitly trained and evaluated for instructional code editing, marks significant step towards code editing with LLMs. However, its evaluation involved GPT-4-generated (OpenAI, 2023b) and numan-provided labels, which raises issues regarding reproducibility and comparability in future research. Moreover, the model has not been publicly released, prohibiting us from evaluating it on our benchmark.
3 The CANITEDIT Dataset
Benchmark Overview CANITEDIT dataset comprising 105 meticulously constructed Python code editing challenges. Each problem includes the input code segment (before), the expected code segment (after), the two types of natural language instructions (descriptive and lazy), and a hidden test suite. These challenges span a broad spectrum of computer science domains, such as data structures, algorithms, mathematics, language processing, and game programming, requiring knowledge of popular external Python libraries like NumPy, Pandas, TyTorc,, and others. Table 1 presents general dataset slatistics for CANITEDIT.
Following Swanson (1976) and follow-up work (Levin & Yehudai, 2017), we classify code editing tasks into three distinct categories based on their primary goal: a corrective edit fixes errors, perfective edit enhances existing features, and adaptive edit meets new require- ments. We have 35 problems per category for an even distribution across the different types of code changes. The dual instruction formats test the models' ability to execute tasks based on the provided context: descriptive instructions provide comprehensive details for explicit guidance, while lazy instructions offer minimal direction, challenging the model to infer

---

Published as a conference paper at COLM 2024
the required actions. Both instructions should lead to equivalent after segment. Descrip- tive instructions serve to replicate situations where users provide specific specifications or another model outlines plan. In contrast, lazy instructions resemble typical user queries for LLMs in code generation. As each problem has two distinct instructions, the dataset effectively contains 210 problems. We showcase examples from CANITEDIT in Appendix C.
Dataset Creation To manually construct CANITEDIT, we assembled a team of eight ex- perienced Python programmers, each with different domain expertise, and appointed one as the lead. The team, recruited through academic networks, included researchers and engineers from various fields who use Python extensively. Our objective was to fill each change category (corrective, perfective, and adaptive) with 35 problems, for a total of 105 problems. The change category is determined by the type of edit required to transform the 'efore' code segment into an 'after' code segment that passes the hidden test suite.
Before starting, we provided the team with verbal and written guidance, a standard template, and example problem. They were instructed to begin by writing brief description of the problem and the applied changes for review and refinement by the lead. Next, they were tasked to write the 'before' code segment and hidden test suite, followed by the 'after' code segment, along with both lazy and descriptive instructions. The lead initially reviewed all problems in development, and they were additionally reviewed by the entire team in weekly meetings. Upon completion of a problem, the lead generated sample completions to ensure that the failures and successes were reasonable and consistent with the problem's intent.
The team dedicated significant effort developing comprehensive test suites for each problem, incorporating a variety of testing techniques such as unit tests, property-based testing, mocking, fuzzing, and integration tests. These suites were designed rigorously evaluate whether the 'after' segment met the problem requirements while ensuring the 'before' code did not. To confirm the completeness and correctness of the test suites, we created an automated verification pipeline that ensured 100% line coverage and that the suite passed all tests with the 'after' code while failing at least one with the 'before' code. The team also manually reviewed the tests to ensure correctness and completeness.
Annotators were encouraged to create problems of varying difficulty and were assigned specific topics and change types. The lead provided example problems and guidance writing unambiguous instructions to ensure fair model evaluation. All instructions and several model completions were reviewed by the project lead to maintain consistency and clarity across the dataset.
4 Fine-Tuning
We describe our approach to fine-tuning Code LLMs for code editing tasks, focusing on the DeepSeekCoder--Bar
family (Guo et al., 2024a), a variant of CodeLlama (Roziere et al., 2023) trained on 2 trillion tokens of GitHub code and natural language, using StarCoder's 's filtering rules (Li et al., 2023b). These models, top-performing in code generation and open-access under permissive license, show robust performance on CANITEDIT without specific training for instructional tasks (55).
For our ablation studies, we focus on the model with 6.7 billion parameters, which offers an ideal balance between size and performance. This allows us to extrapolate results to larger models with more parameters without the need for extensive training. Following the most performant training strategy identified, we also fine-tune the 1.3b and 33b models to evaluate the impact of model size on code editing performance. Our fine-tuned models are referred to EDITCODER. These models have been full-parameter fine-tuned by calculating the loss on only the 'after' code segment. Appendix B provides further details and experiments on the training process.
We experiment with two training datasets we gathered: EditPackFT and Commits2023FT, which we describe below. Table 2 presents the statistics for these datasets.
5

---

Published as a conference paper at COLM 2024
Dataset Statistics
EditPackFT
Commits2023FT
Total Commits
22,602
24,129
Unique Initial Verbs
184
199
Code Segments (Mean + Std. Dev.)
Lines of Code
29.2 1: 13.7
119.3 1 75.9
Levenshtein Distance
197.1 + 260.6
406.6 + 631.2
Commit Messages (Mean Std. Dev.)
Tokens
10.1 4.6
23.1 + 35.2
Table 2: Training dataset statistics for EditPackFT and Commits2023FT
concaterated
1000
concatenated
1000
after
750
after
500
20
200
500
(a) EditPackFT
(b) Commits2023FT
Figure The distribution of the number of lines in the 'before' and 'after' code segments in the EditPackFT and Co1MIts2023FT datasets. The 99th percentile is removed for clarity.
EditPackFT We created the EditPackFT dataset by further filtering the Python split of the ComFitPackFT dataset (Muennighoff et al., 2023). CommitPack is an extensive dataset comprising 4TB of permissively licensed commits from 2016 GitHub snapshot across various programming languages. CommitPackFT is subset of CommitPack, filtered to be amenable for enstruction-tunng Code LLMs. The primary criterion for CommitPackFT's selection involved retaining commits whose messages begin with an imperative verb, mirroring the typical structure of natural language instructions. We apply a series of additional filtering steps, which make the dataset more suitable for code editing. We remove any item that pass any of the following predicates:
1. The presence of empty 'before' 'after' code segment, disregarding whitespace.
2. No change detected in the 'before' and 'after' code segments.
3. The inclusion of the words TODO, FIXME, or BUG in the 'after' code segment, which signals an incomplete commit.
4. Incorrect parsing of the 'after' code using the Python ast module.
Originally, the dataset contained 56,025 commits, and after applying the filtering steps, we are left with 22,602. As shown by Figure 2a and Table the mean number of lines in the 'before' and 'after' code segments is 29.2. The mean Levenshtein distance between the 'before' and 'after' code segments is 197.1 characters, indicating that the changes are small relative to CANITEDIT'S 302.1 characters. We also analyze the distribution of the commit message lengths, and find that the mean token count 10.1, which is quite low compared to CANITEDIT'S 81.7 tokens for the descriptive and 35.6 tokens for the lazy instructions. We further analyzed the original CommitPackFT dataset to ensure that these findings weren't artifacts of our filtering strategy: the full dataset has similar statistics.
Commits2023 To address the limitations of EditPackFT, we developed the CommIts2023FT dataset. This dataset is filtered from 416,792 Python file changes gathered from commits in permissively licensed GitHub repositories. We name the unfiltered dataset Commits2023, Our objective is to create dataset akin to CommitPackFT, but with more recent data and

---

Published as a conference paper at COLM 2024
more diverse example length distribution. We employed the same filters this dataset as used for EditPackFT, and also applied the initial filters from CommitPackFT, which includes only commits with messages that start with imperative verb followed by at least noun. Additionally, we only retain one file from multi- file commits to avoid exact duplicate commit mes-
sages in our dataset. After this filtering process, obtain a dataset comprising 24,129 Python file changes. Figure 2b presents a broader distribution of the number of lines in the 'before' and 'after' code segments, with an average of 119.3. We also observe a much larger change distribution, with mean Levenshtein distance of 406.6, signaling that this not only a dataset with larger code segments, but also contains more varied changes. Figure il- lustrates a sunburst plot of the most frequent initial verbs in the commit messages of Commits2023FT, along with their corresponding root nouns. This set of verbs is slightly more varied than those in Edit- PackFT, featuring 199 unique verbs in comparison to 184. Furthermore, the token count distribution of the commit messages is twice as high and much more varied than that of EditPackFT, with a mean of 23.1 and standard deviation of 35.2.
bug
support
test
asue
Add
test
Fix
Upddan
esst
Ued
Figure 3: Sunburst plot of the top 20 most frequent initial verbs, with their corresponding top 10 root nouns, in the commit messages of Com- mits2023FT.
Ablation Datasets For ablation analysis, we generated two additional datasets: Com- mits2023Raw25k and Commils2023Ft EdittPPckF Commits2023Raw25. consists of random selection of 25,000 commits from Commits2023, We use this dataset to assess the impact of the filtering process on the final dataset. Fommitt2223FTEEdittPPa the combined dataset of 2om3Its2023FT and EditPackFT. We find that the combination of C0m3Its2023FT and EditPackFT yields the best results by a significant margin ($5.2), and thus we train our final models on this dataset. We believe that these results are due to the increased amount of data and the expanded length distributions.
represents
Evaluation
In this section, we evaluate the performance of various open and closed models on the CANITEDIT benchmark, as well our fine-tuned models.
Evaluation Tools and Hyperparameters We run the open-access models using Hugging- Face Transformers (Wolf et al., 2020) and vLLM (Kwon et al., 2023). We use the following hyperparameters for all inference experiments: 2048 maximum new tokens, temperature 0.2, and top-p sampling cutoff of 0.95. Following Cassano et al. (2023b), we sample 20 completions for each problem. We run all tests in Docker container to mitigate the risk of malicious code execution.
Models Evaluated We evaluate several state-of-the-art models of varying sizes, fine-tuning some of them to build EDITCODER. We group the models into two categories: open, models which we have access to their weights, closed, models which we do not. We prompt each model with their recommended prompt template. The specific templates used appear in Appendix A.6. The full list of models and their sizes appears Table 3.
5.1 Evaluation Metrics
We employed two metrics to assess model performance: pass@k assesses functional correct- ness, and ExcessCode assesses conciseness and precision of code edits.
pass@k is the likelihood that at least one successful edit was made from k attempts, assessed by the test suite. In this section, we show results only for pass@1, and
7

---

Published as a conference paper at COLM 2024
Model
Descriptive
Lazy
Name
Size
pnss
ExcessCode
pass@1
ExcessCode
Closed Models
GPT-4
63.33
0.15 + 0.09
51.95
0.14 + 0.10
GPT-335-Turbo
48.14
0.47 0.34
42.71
0.00 0.00
Open Models
CodeLlama-Instruuc
70b
45.05
0.28 + 0.15
37.52
0.02 = 0.02
Mixtral-Instruct
8x7b
30.10
0.40 0.16
24.90
0.01 0.01
EDITCODER
33b
55.90
0.33 0.21
42.33
0.27 0.24
DeepSeekCoder-Istruuctt
33b
49.78
0.36 + 0.24
38.94
0.51 + 0.34
DeepSeekCoder-Base
33b
47.71
0.53 1 0.24
34.71
0.62 0.41
CadeIllam-Instruct
34b
30.63
0.33 0.21
24.15
0.18 0.14
StarCoder2
15b
41.95
0.36 + 0.20
31.48
0.04 + 0.04
StarCoder
15b
37.10
0.56 + 0.28
27.62
0.42 = 0.34
OctoCoder
15b
34.43
0.12 - 0.07
25.95
0.07 0.07
13b
26.90
0.90 0.68
16.89
0.42 0.41
EDITCODER
6.7b
48.33
0.36 + 0.17
39.29
0.32 + 0.25
DeepSeekCoder-Inttruuc
6.7b
41.03
0.13 + 0.06
31.65
0.22 = 0.12
DeepSeekCoder-B
6.7b
32.62
1.01 0.42
27.76
1.25 0.98
Ladeellam-Innstuuu
7b
32.83
0.31 0.15
23.49
0.36 0.26
EDITCODER
1.3b
26.67
0.14 0.09
21.43
0.20 + 0.12
DeepSeekCCoder-Intruuc
1.3b
26.22
0.32 0.18
17.27
0.32 0.13
DeepSeekCCoder-Bas 1.3b
17.90
0.69 0.42
11.76
2.79 2.29
Table 3: Evaluation results of close and open-access models on CANITEDIT. We report the pass@1 and ExcessCode metrics for both the descriptive and lazy prompts as well the size of the model if available.
evaluation results for pass@10 and pass@100 with higher temperatures can be found in Appendix A.2.
ExcessCode evaluates the presence of unnecessary code changes, as indicated by the fraction of changed lines not covered by the test suite. We calculate this metric by averaging the mean line coverage for passing completions across all problems, omitting those with successful completions. The Python code used to calculate this metric is found at Appendix A.1. We additionally report the standard error of the mean for this metric.
5.2 Results with Existing Models
We draw several conclusions from the full results in Table 3.
Closed source models outperform open source models. Our evaluation indicates a sig- nificant performance disparity between open and closed models. GPT-4, despite not be- ing primarily trained on code-related tasks but general instruction-followingg surpasses DeepSeekCoder-Instruss 33b the leading open source model - -by an average of 13% in pass@1 for both descriptive and lazy tasks. DeepSeekCoder-Inne open model exceeding any closed model's performance, surpassing GPT-3.5-Turbo for descriptive prompts.
stands out as the only
DeepSeeKCoder-Instruk utilizes an undisclosed instruction-tuning dataset, therefore di- rect comparisons with other open models may not entirely fair. In contrast, Mixtral- Instruct (Jiang et al., 2024), comparable to OpenAl models its general instruction-followin training focus, significantly lags in performance against both closed models and open mod- els specialized in code generation tasks. Lastly, Codellaam--Instrutt//00 ranked second in
8

---

Published as a conference paper at COLM 2024
instruction-followie capabilities, was developed using dataset of examples generated by Llama 70b with a larger focus on code generation tasks. This may explain its superior performance compared to Mixtral-Instrucc.
Descriptive prompts yield better performance than lazy prompts. Descriptive prompts result in a 8.68 absolute increase on average in pass@1 compared to lazy prompts, which may be the result of more detailed information and additional pointers in descriptive problems. Lazy instructions generally lead to lower ExcessCode for larger models. This may be indicative of lazy instructions introducing less noise in the task, as there are less tokens to attend in the prompt given.
Larger models perform better than smaller models. Model size correlates positively with pass@l, and negatively with ExcessCode, indicating that larger models are more adept at precise edits to code. This pattern is most clearly seen in the evaluation results of DeepSeekCoder-Bar StarCoderBase, and StarCoder2, where a steady increase in perfor- mance is seen along with an increase in model size (Appendix A.5).
Models pre-trained on commits are better at code editing. is pre-trained on GitHub commits, while StarCoder2 focuses on GitHub issues. StarCoder outperforms similar-sized DeepSeek and Codel models our benchmark, despite their superior code generation capabilities. OctoCoder, StarCoder-basse model fine-tuned on instructions (Muennighoff et al., 2023), shows lower pass@1 performance, suggesting instruction fine-tuning on commit-based models may reduce code editing efficacy. Con- versely, StarCoder2, exchanging commits with issues in its training, improves in editing tasks with larger models (Appendix A.5). This may be attributed to extended training and modern architecture rather than data source shift.
Among open models, StarCoder
5.3 Results after Fine-Tuning on Commits
Training Dataset
Descriptive
Lazy
Name Commiis2D23FT-FFddPPPddFTT
#Tokens
#Items
pass@1
EC
pass@1
EC
74M
46,274
48.33
0.36
39.29
0.32
Commits2023FT
62M
24,129
46.57
0.42
37.19
0.22
Commits2023Raw254
62M
25,000
42.57
0.41
35.14
0.17
EditPackFT
12M
22,602
47.14
0.40
36.05
0.07
Table 4: Ablation results of training DeepSeeCCoder-677-BBee on different datasets and evaluating on CANITEDIT. We show the total number of tokens and items in each dataset, as well as the pass@1 and ExcessCode (EC) metrics for both the descriptive and lazy prompts. The reported sizes of the datasets are after deduplication.
In addition to evaluating existing open models, we also fine-tuned pre-trained DeepSeek models (54) to build EDITCODER, which we now evaluate.
Optimal Dataset: Commits2033FT+Ediiaad
In finding the best training dataset for SeepSlekkCodee--6.-b-Baese the base model for EDITCODER, various ablation datasets were tested. Results in Table 4 show Commiss003FTT-Riiiiack
is the top performer for both descriptive and lazy instructions. The dataset's larger size and diverse data types, including varied commits, edits, and instructions, likely contribute to its superior performance.
Fine-tuning on open commits can significantly improve code editing performance. EDIT- CODER-33b surpasses all open models in pass@1 for both descriptive and lazy instructions types, showing an overall 10.7% increase in pass@1 and notable decrease in ExcessCode Base-33b. Additionally, we see a substantial increase in pass@l for every iteration of EDIT over its corresponding base model, with the largest improvement being a 45.1% increase at 6.7b.
compared to its base model, DeepSeekCoders Badeeeeaee---BBee3bbbb
Both EDITCODER-33b and EDITCoDER-6.7b outperform GPT-3.5-Turbo in pass@1 for de- scriptive instructions, with EDITCODER-SS also matching GPT-335-Turte in for lazy ones. In higher temperature scenarios (Appendix A.2), EDITCODER-333 beats GPT-3.5-Turbo in

---

Published as a conference paper at COLM 2024
both instruction types for pass@10 and pass@100, and even surpasses GPT-4 in pass@ 100 for descriptive instructions. Analysis in Appendix A.3 shows EDITCODER excels in corrective changes but is less effective in perfective changes. Further analysis in Appendix A.4 shows outperforms all models, including GPT-4, in simple single-function
that EDITCODER bug fixes, and retains synthesis capabilities. This demonstrates the effectiveness of targeted fine-tuning on code editing datasets, addressing the distinct needs of instructional code editing compared to general code generation.
Conclusion
We present CANITEDIT, benchmark designed to assess the instructional code editing skills of Code LLMs. It includes 105 hand-written code editing problems, each accompanied by dual natural language instructions: a "lazy" instruction that a human may write, and " "descriptive" instruction that may be generated by an agent revising code in loop. Each problem has a comprehensive test suite. We evaluate contemporary state-of-the-ard Code LLMs and reveal a significant gap between closed and open models. We also demonstrate that fine-tuning with custom dataset and training methodology can significantly improve code editing capabilities across various model sizes. Our work provides a foundation for evaluating future enhancements in instructional code editing for Code LLMs, offering valuable tools and insights for Al-based software development research and practice.
Limitations We evaluated LLMs in reproducing the entire 'after' code segment, which may not be the most token-efficient method. A potentially more efficient strategy would involve generating a list of specific changes to applied to the 'before' code segment. Furthermore, our study does not explore varying prompt formats. Instead, we have adopted a format consistent with that used by other models (Li al., 2023b). Another limitation is the size of our final training dataset, which is relatively modest. We have not investigated the potential benefits of utilizing larger datasets, which could notably enhance performance, particularly with larger models. Our work only targets Python. Similar results may be possible for other high-resource programming languages, but low-resource languages may require additional effort (Cassano et al., 2023a). We identify these areas as opportunities for future work.
Acknowledgements
This work was partially supported by the United States National Science Foundation (Awards SHF-2052696 and SES-2326174). Federico Cassano was affiliated with Roblox for significant part of his work on this project. We thank Northeastern Research Computing and Joydeep Biswas for providing computing resources.
References
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. ar preprint ar xrXim:2108.07732 2021.
Ned Batchelder and Contributors Coverage.py. Coverage.py: The code coverage tool for Python. URL https:: //gp:::///ithie thub. http:////eeebab///eeeeeeeeee /tt.tp////gttth...//////atrrrree
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arxiv preprint ar arXio:220..12255 2022.
Andrei Z. Broder.
Identifying and filtering near-duplicate documents. In Combinatorial Pattern Matching, 2000.
Federico Cassano, John Gouwar, Francesca Lucchetti, Claire Schlesinger, Carolyn Jane Anderson, Michael Greenberg, Abhinav Jangda, and Arjun Guha. Knowledge transfer
10

---

Published as a conference paper at COLM 2024
from high-resource to low-resource programming languages for code llms, arxio preprint arXi::2308.0900 00895, 2023a.
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costee Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation. IEEE Transactions on Software Engineering (TSE), 2023b.
Sahil Chaudhary, Code alpaca: An instruction-follooiing llama model for code generation. https://gt thub. com/ woom/sahil2aa sahi128011 14/ c/ceeaaaaaaa, 2023,
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. Xiv preprint ar arXi:21077.03774 2021.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. Xrv preprint arXio:2304.05125 2023.
GitHub Copilot. Github copilot your pair programmer, 2023, URL http:://///gt com/ features/copilot.
Cursor. Cursor: The ai-first code editor, 2023. URL http://cuursrrssfffetttu Accessed: 2023-12-03.
Tri Dao. Flashattention-2:: Faster attention with better parallelism and work partitioning. ar Xiv preprint ar arXiv:230.008691 2023.
Z. Fan, X. Gao, M. Mirchev, A. Roychoudhury, and S, Tan. Automated repair of programs from large language models. In 2023 IEÉE/ACM 45th International Conference on Soft- ware Engineering (ICSE), 2023. URL https:/ batps://dont ieeecooputersociety org/10. 1109/ ICSE48619.
Daniel Fried, Armen Agnajanya,,,,,,, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL https: / httpvviiwwweeffruureeeeee hww.frrmmmii-hhw--bbbeeee...
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder. When the large language model meets programming - the rise of code intelligence, 2024a.
Jiawei Guo, Ziming Li, Xueling Liu, Kaijing Ma, Tianyu Zheng, Zhouliang Yu, Ding Pan, Yizhi LI, Ruibo Liu, Yue Wang, Shuyue Guo, Xingwei Qu, Xiang Yue, Ge Zhang, Wenhu Chen, and Jie Fu. Codeeditorbench: Evaluating code editing capability large language models, 2024b. URL
2404. 2404.03543..
Priyanshu Gupta, Avishree Khare, Yasharth Bajpai, Saikat Chakraborty, Sumit Gulwani, Áditya Kanade, Arjun Radhakrishna, Gustavo Soares, and Ashish Tiwari. Grace: Lan- guage models meet code edits. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023. URL https://doi org/10 10. /145/3611111 1643. 3616253.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn't always right. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7038-7051, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational org/
Linguistics. doi: 111638/v12200 http://1/211eeellmmmaaaa emnnl-mmin..566 564. URL www.lemmainm..... 564.
Qisheng Hu, Kaixin Li, Xu Zhao, Yuxi Xie, Tiedong Liu, Hui Chen, Qizhe Xie, and Junx- ian He. Instructcoder: Empowering language models for code editing. Xiv preprint arXit:2310 2023.
11

---

Published as a conference paper at COLM 2024
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? Xiv preprint ar Xt::2310.00677, 2023.
Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. Inferfix: End-to-end program repair with llms. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023. URL https:://oi.. htt:///oo.org////// org/10. 1 / 1455/61160 1..11.443.11111
Harshit Joshi, José Cambronero Sanchez, Sumit Gulwani, Vu Le, Ivan Radiček, and Gust Verbruggen. Repair is nearly generation: Multilingual program repair with lims. In Proceedings of the T'hitty-eventth AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, 2023. URL https://doi. org/10. (aa9//aaa.. v37i4. 25642.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Rich árd Nagyfi, et al. Openassistant conversations -emeraattiiisdmmmraiiiinnng large language model alignment. Xiv preprint arXiv:2304.07327, 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In ACM SIGOPS Symposium on Operating Systems Principles (SOSP), 2023.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022. URL https: // www.aaollaatttoooo.o.ooo . wrg.2022.c2. acl-long. 577.
Jure Leskovec, Anand Rajaraman, and Jeffrey David Ullman. Finding Similar Items, pp. 68-122. Cambridge University Press, 2 edition, 2014. doi: 10.1017 /CB/////01111999400100
Stanislav Levin and Amiram Yehudai. Boosting automatic commit classification into mainte- nance activities by utilizing source code changes. In Proceedings of the 13th International Con- ference on Predictive Models and Data Analytics in Software Engineering, PROMISE, pp. 97-106, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450353052. doi:
URL https://doi. org/16 /0. 1145/3127000 3127016.
Jia Li, Ge Li, Zhuo Li, Zhi Jin, Xing Hu, Kechi Zhang, and Zhiyi Fu. Codeeditor: Learning to edit source code with pre-trained models. ACM Transactions on Software Engineering and Methodology, 2023a.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muermighoff, Denis Kocetkov, Cheng- hao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenú Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muh- tasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy,, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Ku- nakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer
12

---

Published as a conference paper at COLM 2024
Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. StarCoder: May the source be with you! ar Xiv preprint ar arXit:2305.06661, 2023b.
Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan. Automating code review activities by large-scale pre-training. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, PP. 1035-1047, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450394130. doi: 10.1145 10.1145/35402555 14145//554025035490811 URL https: /tpp:://oo... org/10 10. 1145/3540250.. 3549081.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. arXiv preprint ar XrX:::2305.01210, 2023.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakksh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generation, 2024.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arviv preprint (XXi::33060086666 2023.
Seungjun Moon, Yongho Song, Hyungjoo Chae, Donginn Kang, Taeyoon Kwon, Kai Tzu- iunn Ong, Seung-won Hwang, and Jinyoung Yeo. Coffee: Boost your code 1lms by fixing bugs with feedback. ar Xiv preprint ar Xxx::2311.072115 2023.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: In- struction tuning code large language models. In NeurlPS 2023 Workshop on Instruction Tun- ing and Instruction Following, 2023. URL https://openrevieee net net/forummiiiii nett
Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar- Lezama. Demystifying gpt self-repair for code generation. ar preprint arXit:2306.09880 09896, 2023.
OpenAl, Introducing chatgpt enterprise, 2023a. URL https://openai..om// purroducinncchatppp-eeetrrrrpppp Accessed: 2023-12-03.
OpenAL. Gpt-4 technical report, 2023b.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to fol- low instructions with human feedback. In Advances in Neural Information Processing Systems, 2022. URL
weurdiiniinnnnuussseseeeee newrips. cwtpuriiisccppppppeeeeee files/paper /22pp////222fflee/ Coneeeeeeeeeeeeeeeeeeeeeeeee .df..
13

---

Published as a conference paper at COLM 2024
Tung Phung, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. Generating high tegh-precision feedback for programming syntax errors using large language models. Xiv preprint ar arXiv:2302.04666. 2023.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2020.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. ar arviv preprint Xro::2330 12950, 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu
Yao. Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openrevieer net/ forum?id=vAEIhf
E. Burton Swanson. The dimensions of maintenance. In Proceedings of the 2nd International Conference on Software Engineering, ICSE '76, pp. 492-497, Washington, DC, USA, 1976. IEEE Computer Society Press.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023. URL https: hatas////lcaaaholoo/oooo wrrg//rgg// acl-long, 754.
Yuxiang Wei, Chunqiu Steven Xia, and Lingming Zhang. Copiloting the copilots: Fusing large language models with completion engines for automated program repair. In Pro- ceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023. URL https: htttt:::/ddooorrgg orgoi..org//00 (tt..i..rrg/111.11///1166664 1145/3611633. 3616271.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020. URL https://aclanttooogy.oorr org/ memnl.-eemo.6.
Chunqiu Steven Xia, Yinlin Deng, and Lingming Zhang. Top leaderboard ranking top coding proficiency, always? evoeval: Evolving coding benchmarks via IIm, 2024. URL https://arxiv.orgg http://www.orrabb/////aa/ 19114.
Ming-Ho Yee and Arjun Guha. Do Machine Learning Models Produce TypeScript Types that Type Check? In Proceedings of the 37th European Conference on Object-Oriented Programmingg 2023. URL https: //ooi. org/10 48550/ arxiv. 2302.12163.
Jiyang Zhang, Sheena Panthaplackel, Pengyu Nie, Junyi Jessy Li, and Milos Gligoric. Coditt5: Pretraining for source code and natural language editing. In Proceedings of the 37th TEEF/ACM International Conference on Automated Software Engineering, ASE '22, New York, NY, USA, 2023a. Association for Computing Machinery. ISBN 9781450394758. doi: URL https: //doi http::///oooooo//////// 1145/3551 349. 3556955.
Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. Self-edit: Fault-aware code editor for code generation. In Proceedings of the 61st Anmual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), 2023b. URL https: http:::://aalathologgyyy .rg gerg/2222..rm acl-long. 45.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. Lmsss-chat- A large-scale real-world Il conversation dataset. arx preprint arXia:2309.11998 2023.
14

---

Published as a conference paper at COLM 2024
A Additional Evaluation Details
In this section, we provide several additional details about evaluation. We provide the following additional details:
1. A Python implementation for computing the ExcessCode metric in A.1.
2. Results of evaluation at higher sampling parameters in A.2.
3. A deeper analysis of our results per change type in A.3.
4. An evaluation of EDITCODER HumanEvall'ack in A.4.
5. A deeper comparison of the first and second versions of StarCoder in A.5.
6. Each prompt format used in our evaluation in A.6.
OpenAI Model Versions For our evaluation we use the following versions of the OpenAl models, which at the time of writing were the latest stable versions:
GPT-4: gpt-4-0613
GPT-3.5-Turbo: gpt-3.5 300-33--tuu t-rrbbo-0125
A.1
Computing ExcessCode
excess_code (before: str, after: str, lines_missing int):
2
3
Compute the ExcessCode score for a single code edit completion
4
Args:
5
before: The original code segment.
6
after: The modified code segment.
7
lines_missing: The number of lines with missing code coverage.
Returns:
9
The computed ExcessCode score.
10
11
import difflib
12
differ difflib. Lifffer
13
before_lines before. splitlines()
14
after_lines after splitlines()
15
lines_changed = (differ (bemprrebbeeoeelliinm after_lines) return lines_missing / lines_changed
16
Listing 1: A Python implementation for computing the ExcessCode metric
We provide simple Python implementation for computing the ExcessCode metric in List- ing 1. The function takes in as input the original code segment, the modified code segment, and the number of lines with missing code coverage. The lines with missing code coverage are computed using a code coverage tool, in our case, Coverage.py (Batchelder & Contribu- tors to Coverage.py), For Coverage.py, the number of lines with missing code coverage can be obtained by running the command coverage report -m.
A.2 Results At Higher Sampling Parameters
In this section, we evaluate the performance of various models under higher sampling parameters compared to those used in our main evaluation ($5).
Standard Sampling Parameters In $5, we assessed several models on CANITEDIT using standard parameters: temperature of 0.2, top-p of 0.95, and 20 samples. These parameters, often used code generation tasks (Chen et al., 2021; Cassano et al., 2023b;a; Muennighoff et al., 2023; Li et al., 2023b; Lozhkov et al., 2024), balance the selection of higher probability
15

---

Published as a conference paper at COLM 2024
Model
Metrics
Name
Size
pnss@1
pass@10
pass@100
ExcessCode
Descriptive
GPTT4
63.67
73.67
80.00
0.18 + 0.09
GPT-3.5-Turbo
47.88
61.67
71.43
0.33 + 0.21
EDITCODER
33b
54.36
73.15
81.90
0.81 1 0.38
CodeLlama-Instruct
34b
28.50
52.00
64.76
0.38 + 0.16
StarCoder2
15b
40.06
62.12
71.43
0.47 + 0.19
StarCoder
15b
33.31
59.80
70.48
0.76 0.25
Codellama-Instruut
13b
25.01
50.04
62.86
0.47 1 0.29
EDITCODER
6.7b
46.31
60.24
70.48
0.42 + 0.23
Codellama-Instruut
7b
29.93
52.86
65.71
0.76 + 0.39
EDITCODER
1.3b
26.29
40.22
47.62
0.50 0.23
Lazy
GPT-4
52.55
64.56
71.43
0.12 1 0.09
GPT-3.5-Turbo
40.93
53.33
60.95
0.29 + 0.27
EDITCODER
33b
40.34
58.63
68.57
0.46 + 0.20
CodeLlama-Instruut
34b
21.10
42.69
56.19
0.19 + 0.08
StarCoder2
15b
30.23
48.82
59.05
0.17 = 0.07
StarCoder
15b
24.75
50.23
65.71
0.68 0.25
CodeLlama-Instrucc
13b
16.67
38.52
58,10
0.67 + 0.44
EDITCODER
6.7b
37.74
51.16
57.14
0.33 1 0.15
CodeLlama-Instruct
7b
20.85
41.10
54.29
0.11 - 0.06
EDITCODER 1.3b
20.20
32.70
39.05
1.47 1.20
Table 5: Evaluation results of models on CANITEDIT at higher sampling parameters. We report the pass@1, 10, and pass@100 metrics for both the descriptive and lazy prompts, as well as the ExcessCode metric. The size of the model is reported available.
tokens while allowing for sampling of lower probability tokens at conservative levels, addressing surface form competition (Holtzman et al., 2021), making it typically more effective than greedy decoding (Chen et al., 2021).
Higher Sampling Parameters For this evaluation, we increased the sampling parameters to assess the models' robustness under more diverse generation conditions. Following Chen et al. (2021), we adopted a temperature of 0.8, top-p p of 0.9, and 100 samples. These aggressive parameters allow the model explore wider range of possibilities, useful when multiple completion attempts are possible. Due to the higher computational costs, we limited our evaluation subset of models compared the main evaluation.
Metrics For this evaluation, we expand our metrics to include pass@10 and pass@100, alongside the standard pass@1 and ExcessCode. pass@10 and pass@100 offer deeper insights into model performance by evaluating the success rate across the top 10 and 100 completions, respectively. These metrics are crucial for understanding how models perform in scenarios that permit multiple attempts, such as when users are provided with range of completions to select from or when an external verifier is used to determine the best completion.
A.2.1 Results
The results of our evaluation at higher sampling parameters are shown in Table We draw several conclusions from the results.
16

---

Published as a conference paper at COLM 2024
pass@1 decreases for open models. Closed models maintain consistent pass@1 performance under higher sampling parameters. In contrast, open source models generally exhibit decline, showing 2-3% reduction in pass@1 performance compared to the main evaluation (Table 3).
Multiple trials benefit open source models. Open source models significantly improve with multiple trials, showing larger gains in pass@10 and pass@100 compared to closed models, which also improve but to lesser degree. Specifically, EDITCODER-33b outperforms all models, including GPT-4, in pass@ for descriptive instructions and matches closely in pass@10. However, EDITCODER-331 lags behind GPT-4 lazy instruction scenarios across all metrics and tends to generate more excess code for both prompt types. We expect that the performance of EDITCODER on lazy instructions will improve with more data and larger pre-trained models.
Lazy instructions benefit more than descriptive instructions from multiple trials. The performance disparity between descriptive and lazy instructions persists, even under higher sampling parameters and multiple trials, as seen in pass@10 and pass@100. Despite this, the rate of improvement from multiple trials is greater for lazy instructions, with increases of 57.76% in pass@10 and 22.57% in pass@100 surpassing the gains for descriptive instructions, which are 48.18% and 17.23% respectively. This indicates more pronounced benefit from multiple attempts in scenarios involving less structured prompts.
Significant increase in ExcessCode. The average ExcessCode metrics for both descriptive and lazy instructions, at 0.507 and 0.449 respectively, have increased from the main evaluation's 's averages of 0.392 and 0.235. 1 This is expected, as higher sampling parameters tend to yield broader range of completions, consequently resulting in an increase in superfluous code.
A.3
Results Per Change Type
Model
Corrective
Adaptive
Perfective
Name
Size
p@1
ExcessCode
p@1
ExcessCode
p@1
ExcessCode
GPT-4
62.21
0.05 = 0.03
57.29
0.31 = 0.19
53.43
0.08 + 0.06
GPT-3.5-Turbo
47.93
0.00 + 0.00
42.29
0.17 + 0.12
46.07
0.60 + 0.54
EDITCODER
33b
56.86
0.02 + 0.02
51.21
0.77 + 0.42
39.29
0.05 + 0.04
EDITCODER 6.7b
48.64
0.00 0.00
42.71
0.43
0.21
40.07
0.66 1 0.42
EDITCODER 1.3b
26.36
0.11 + 0.10
23.21
0.14 + 0.10
22.57
0.26 + 0.18
Table 6: Results of OpenAl models and EDITCODER on CANITEDIT per change type. We report the pass@1 and ExcessCode metrics for each change type, as well the size of the model if available. Results for lazy and descriptive prompts are aggregated across all problems. pass@1 is abbreviated to p@1. Closed models are the rows highlighted in gray.
In this section analyze the results of EDITCODER against the OpenAl models on CAN- ITÉDIT per change type. We hope to gain insights into the strengths and weaknesses of these models for different types of code changes. Table 6 shows the results of our analysis, we aggregate the results for lazy and descriptive prompts across all problems, this is done for conciseness and to minimize the noise of our results, each pass@1 and ExcessCode metric is calculated across 70 problems per change type, instead of 35. We utilize the same sampling parameters as in our main evaluation. Our key findings include:
GPT-4 outperforms other models in functional correctness across all types of changes.
Corrective changes typically incur minimal excess code, with some models achiev- ing perfect scores in this area. Adaptive changes, intuitively, tend to introduce the most excess code.
These values are calculated by averaging over the results from the models in this table, and not the entire set of models in the main evaluation.
17

---

Published as a conference paper at COLM 2024
Our EDITCODER-33b model surpasses GPT-3.5-Turbo in both corrective and adap- tive changes, while EDITCODER-6.7 shows comparable performance to GPT-3.5- Turbo in these categories. However, for perfective changes, both EDITCODER-33b and EDITCODER-6.70 underperform compared to GPT-3.5-Turbo. This suggests potential improvement in training data for perfective changes. As demonstrated in Figure 3, verbs associated with perfective changes, such as reffctor or improve, appear less frequently than those related corrective or adaptive changes, such as fix or add, respectively. Artificially balancing the dataset with more examples of perfective changes could potentially enhance EDITCODER'S performance. We leave this area for future work.
According to dataset, the most challenging changes are perfective, followed by adaptive, with corrective being the simplest.
A.4 Evaluation HumanEvalPack
Model
passo1
Name
Size
Fix
Synthesize
GPT-4
47.0F
86.6*
EDITCODER
33b
53.0
63.5
DeepSeekCoddrr-Isstuut
33b
47.5t
79.2
CodeLlama-Instrucd
34b
36.5
43.8
StarCoder2
15b
48.6*
51.6
StarCoderBase
15b
25.6
33.6
OctoCoder
15b
30.4
35.1
CodeLlama-Instruct
13b
19.4+
23.7
EDITCODER
6.7b
46.6
52.9
DeepSsekCoder-Instruut
6.7b
44.9t
76.2
EDITCODER 1.3b
24.3
28.3
DeepSeekCoddrrInsstuuc
1.3b
9.1
62.4
Table 7: Results of models the Python subset of HumanEvalFix and HumanEvalSyn- thesize. t indicates that the result originates from Lozhkov et al. (2024), while $ indicates that the result comes from Muennighol et al. (2023). The rest of the results from our evaluation following the same methodology as in Muennighoff et al. (2023).
To draw similarities and differences between CANITEDIT and HumanEvalPack, in this sec- tion evaluate models on the Python subset of HumanEvalFix and HumanEvalSynthessiz Results are available in Table
Benchmark Overview HvlakEvalPack is a benchmark comprised of 164 single-function problems aimed at evaluating both code generation and code editing. The problems designed to not require domain-specific knowledge or familiarity with popular external libraries. HumanEvalFix contains only corrective code changes, while HumaanEvalSytthe. size purely focuses on code generation, tasking the model to generate function from its signature and docstring.
Results In this benchmark, EDITCODER-33b outperforms even GPT-4 in fixing bugs, while maintaining competitive synthesis capabilities against models trained on general instructional data, with EDITCODER-6.77 outperforming even larger models like CodeLlama- Instruct-34b. Additionally, we find large disparity between the performance of models on HumanEvalFix and HuumanEyalSynthesize for DeepSeekCoder- 1oree--ott----tutt113333 which performs significantly worse in fixing bugs than in synthesizing functions. These results demonstrate that HumanEvalPack and CANITEDIr complement each other: the former focuses on function algorithmic and puzzle-like problems, while the latter emphasizes
18

---

Published as a conference paper at COLM 2024
code editing tasks requiring broader knowledge of software engineering concepts in wide range of domains.
Example Problems To illustrate typical problems in HumanEvalPack, selected exam- ples from HumannvalSynthesszee and HumanEvalFix. Listing presents a HumanEval- Synthesize problem where the model must complete function based on its signature and docstring. Listing 3 demonstrates an incorrect function implementation from HumanEval- Fix along with it's ground truth unit test suite, where the model's task is correct the implementation. Unlike in CANITEDIT, the model must infer the correct implementation solely from the faulty code and the test suite, without explicit instructions. One could argue that this falls under intrinsic code editing, rather than instructional code editing, since the model is given any instructions about the intent of the function, making this benchmark more suitable for evaluating works such as
Li et al. (2023a) and Gupta et al. (2023).
Write a Python function has__close elements (numbers: List[float]] threshold: float) -> bool to solve the following problem: Check if in given list of numbers, are any two numbers closer to each other than given threshold.
>>> has_close_elementie ([1.0,, 2.0, 3.01. 0.5)
False
>>> pas_cclose_eemmentteee
2.8, 0.0, 4.0, 5.0, 2.01, 0.3)
True
Listing 2:
The prompt of problem in HumanEvalSynthesize. The task for the model is to complete the function.
def inique_digits((x
odd__iigit_elemnnts []
for j. in exumerate if all (int(c) 2 == for с in str (i)):
4
odd_diigit_elements htt.:eeemeetsssppeenncece.
odd_digiit_eeements append (j)
return sorted(ood_digitteeeemnsss
def check (unique_digits): 10
assert unique_digits 33, 1422, 13) [1, 15, 33]
assert unique_digits<(1122 323, 1422, 10]) []
12
2033, 111, 151]) [111, 151] assert unique_digits ((135, 103, 31]) = [31, 135]
assert
13
14
15
check (unique_digits)
17
Fix bugs in mnique_igitts
Listing 3: The prompt of a problem HumanEvalFix. The implementation is incorrect, and the task for the model is to re-implement the function correctly.
Correlation with CANITEDIT We hypothesize that the performance of models on CAN- ITEDIT is correlated with their performance on general code generation tasks. Intuitively, if model exhibits strong capabilities in code generation tasks, it is likely perform well on code editing tasks as well. We quantify the correlation between the performance of models on CANITEDIT and HumanEvalPack by calculating Pearson correlation coefficient between the pass@1 metric of models CANITEDIT and HumanEvalPackk Synthesize, utilizing models on which we have results for both benchmarks. We find a correlation of 0.661, indicating moderate positive correlation between the two benchmarks.
A.5 Comparison of StarCoder Models
StarCoder Models The first version of StarCoder models were pre-trained on several gigabytes of GitHub commits, discussed in (Li et al., 2023b). In contrast, the second
19

---

Published as a conference paper at COLM 2024
Model
Descriptive
Lazy ExcessCode
Name
Size
pass@1
ExcessCode
pass@1
StarCoder2
15b
41.95
0.36 +: 0.20
31.48
0.04 + 0.04
StarCoder
15b
37.10
0.56 + 0.28
27.62
0.42 + 0.34
StarCoderBase
15b
35.33
1.55 + 0.89
27.05
0.85 + 0.55
StarCoderBase
7b
32.90
0.43 + 0.17
21.95
0.49 = 0.37
StarCoder2
7b
25.10
1.47 + 0.78
13.76
1.81 + 1.22
StarCoder2
3b
15.95
0.91 1 0.39
13.33
1.09 1 0.98
StarCoderBase
3b
14.81
1.22 + 0.52
9.90
1.17 + 0.76
StarCoderBase
1b
4.90
0.99 + 0.85
5,48
0.00 + 0.00
Table 8: Evaluation results of StarCoder models on CANITEDIT.
version, StarCoder2, did not include commit data in its training process. However, it was trained GitHub issues, which provides it instruction following capabilities. Issue data encompasses a broader scope than commits, including discussions, bug reports, and feature requests. With prompt-engineering as demonstrated in Lozhkov et al. (2024). Furthermore, in most benchmarks evaluated in Lozhkov al. (2024), StarCoder2 surpasses its previous version, StarCoderBass, code generation tasks.
StarCoder2 models be used for code editing tasks,
The original StarCoder model, StarCoderBase-155) was additionally trained on Python code from GitHub. StarCoderBase models are available in four sizes: 15b, 7b, 3b, and 1b. On the other hand, StarCoder2 has not undergone further training on additional Python code and is available in three sizes: 15b, 7b, and 3b.
Evaluation We evaluate sizes of StarCoder and StarCoder2 models on CANITEDIT and present the results in Table 8. We find that StarCoder2 outperforms StarCoder in the 15b and 3b sizes, but not the 7b size, where StarCoderBase-T1 significantly outperforms StarCoder2-7b. Additionally, for the 7b and 3b sizes, StarCoder2 models tend to generate more excess code than StarCoderBase models. We attribute the performance improvements in StarCoder2 to the broader training data and architectural enhancements, as discussed in Lozhkov et al. (2024), rather than to the superiority of issue data over commit data for code editing tasks. However, we also believe that for utilizing StarCoder2 models directly for code editing tasks, the issue prompt format is a viable alternative to the commit format previously utilized by StarCoderBase models. We provide the prompt format we utilized for StarCoder2 models in Figure 5.
A.6 Prompt Templates Used in Evaluation
We evaluate all of our models on CANITEDIT using the same evaluation pipeline. However, for each model, we may utilize different prompts to generate the completions. These prompts are most aligned to how the model was trained, and are intended to maximize the model's performance on the task, while keeping the prompts similar as possible across models. Figure 4 shows the prompts used for each model. For a fair comparison, evaluate all models not trained commits explicit code editing tasks using basic 1-shot prompt, showing the model how to add a sub function to a code segment with a add function, and changing the variable names from and b to x and y.
Furthermore, given the natural language characteristics of GitHub issue data, significant prompt-engineering was required to facilitate code editing tasks for StarCoder2 models. The specific prompt format used for StarCoder2 models is provided separately in Figure 5.
20

---

Published as a conference paper at COLM 2024
<user>
You are PythonEditGPT. You will be provided the original code snippet and an instruction that specifies the changes you
make.
You will produce changed code, based on the original code and the instruction given. Only produce the code, do not include any additional prose.
<system>
You are tyt You will be provided the original code snippet and an instruction the changes you need to make.
specifies
## Code Before ppy def (a, b): return b
You will produce the changed code, based the original code and the instruction given. Only produce the code, do not include any additional prose. <user>
## Instruction
## Code Before py def add(a, b):
Add a "sub" function that subtracts two numbers. Also write docstrings for both functions and change a,b to x,y.
return b
<assistant>
## Code After py def axd(x, y):
## Instruction
Add "sub" function that subtracts two numbers. Also write docstrings for both functions and change a, to x,y.
"""Adds two numbers."
return x +
def sub(x, y):
<assistant>
"""Subtracts numbers." "MH
## Code After 'py
return x-y
def add(x, y):
<user>
"""Adds two numbers."
You tGP.. You will be provided the original code snippet an instruction that specifies the changes you need to make.
return y
def sub(x, y):
"""Subtracts two numbers." "**
return
You will produce the changed code, based on the original code and the instruction given. Only produce the code, not include any additional prose.
<user>
Code Before
ppy (before)
## Code Before
pp (before}
Instruction
(instruction)
## Instruction (instruction)
(b) Conversation template utilized for all chat models with a 'system' prompt. The prompt is then adapted to the specific model chat format. This is the prompt utilized for: GPT-4, G-1--.5- Turbo, Codel lama-Instruct, and DeepSeekCoder- Instruct models.
(a) Conversation template utilized for all chat models without 'system' prompt. This is the prompt utilized for OctoCoder.
<commit __beeer>>> (before)
## Code Before:
(before) ## Instruction: (instruction]
<commi t__ms²>
(instruction) <commit_afters
## Code After:
(c) Prompt utilized for StarCoder and StarCoder- Base models of all sizes. StarCoder models are trained commits this format (Li et al., 2023b).
(d) Prompt utilized for our fine-tuned EDITCODER models. DeepSeekCoder-Bee Base-Base models this prompt with the add and sub 1-shot example.
Figure 4: Prompts for each model evaluated on CANITEDIT. The (before) identifier is replaced with the 'before' code segment, and (instruction) is replaced with the instruction. Text wrapped in <...> is used to represent special tokens that utilized by the models.
21

---

Published as a conference paper at COLM 2024
<issue_starr t>username_0 have program in Python that I'd like to change.
Here is the code for the program: .. ppy
def add(a, b): return + b
The change I'd like to make is:
Add a "sub" function that subtracts two numbers. Also write docstrings for both functions and change a, to x,y.
Please someone help
you also provide the full code with the change? <issue_ccmmenttune commentzsername..1 Sure, no problem. will be able to help. I am an expert in edi editing Python code.
Here is the full code with the change:
ppy
def add(x, y):
"AAds two numbers. """"X"
return
def sub(x, y): ""\"\"Subtracts two numbers. "\"""" return
Upvotes: 200<issue_ cotessee_oommnnuuuue toseenne Thank you so much! have another program in Python that I'd like to change.
Here is the code for the program: "pppy (before)
The change I'd like to make is: (instruction)
Please someone help me. Can you also provide the full code with the change? Upvotes: 180kissue_ccmee commentzusername_.. Sure, no problem. I will be able to help. am expert in edi editing Python code.
Here is the full code with the change:
[after)
Figure Prompt utilized for StarCoder2 models. StarCoder2 models are trained on Gub issue data, which makes this prompt format amenable to code editing tasks (Lozhkov et al., 2024).
22

---

set Published as a conference paper at COLM 2024
B
Training Details
In this section provide details the training process for EDITCODER and ablation results on the loss masking technique used in training.
B.1 Training Tools and Configuration
For training all of our EDITCODER models, utilize fine-tuning pipeline based on the HuggingFace Transformers library (Wolf et al., 2020). Additionally, we utilize DeepSpeed ZeRO (Rajbhandari al., 2020) to efficiently shard the model across multiple GPUs. Due to memory constraints, we offload the optimizer to the CPU for the 33b model. We also use FlashAttention 2 (Dao, 2023) to speed up training on large context window sizes. All of our models are trained on a single machine equipped with 8 NVIDIA H100 (80GB) HGX GPUs. The effective micro-batch size set 32 (4 gradient accumulation steps, with a single batch per GPU). We utilize the A optimizer with a learning rate of 2 x 10-5, 5, a linear decay scheduler, and 10 warmup steps. These parameters were chosen based on previous work on fine-tuning for code generation tasks (Cassano et al., 2023a), it is likely that we could get superior results by running hyperparameter search. To facilitate reproducibility, the random seed to 42 for all experiments.
Prior to training, we shuffled the dataset randomly and deduplicated it following the method outlined by Li et al. (2023b). This process combines MinHash (Broder, 2000) and Locality Sensitive Hashing (LSH) (Leskovec et al., 2014). We format the training data as prompt, with the 'before' code segment followed by the 'instruction' and the 'after' code segment, and mask the loss calculation to only consider the 'after' code segment. All models underwent training for epochs, with packed context window of 8192 tokens, including padding for the remaining tokens. We select the model from the epoch with the highest performance on held-out validation set. The number of epochs chosen for each EDITCODER is the following:
EDITCoDER-1.3b: 8
EDIrCoDER-6.7:: 4
EDITCODER-333: 2
As shown by the number of epochs, we found that larger models overfit to the data more quickly, suggesting that we could achieve better results with a larger dataset.
B.2 Effect of Loss Masking
In our training pipeline, mask the loss calculation to only consider the 'after' code segment. The intuition behind this is that we don't need the model to learn how to reproduce the 'before' and 'instruction' segments, as these are always going to be provided as input to the model at inference time. The exact prompt format we use for training is shown in Figure 4d.
We wish to verify that this loss masking is beneficial for our task. To assess our hypothesis, we train two DeepSeekCoder-6.1 -edee...---Bseeee models on EditPackFT one with loss masking and one without, calculating the loss on tokens. We then evaluate both models on CANITEDIT, and report the pass@1 and ExcessCode metrics in Figure 6. The reported result with loss masking is the same as the one reported in Table 4. We find that the model trained with loss masking outperforms the model trained without it, and leads to decrease in ExcessCode and its standard error. Furthermore, we plot the training loss curves for both models in Figure 7. We observe that the model trained with the loss masking technique is more stable and converges faster than the model trained without it.
Deduplication, achieved by concatenating the 'before' and 'after' code segments, helps mitigate overfitting to specific training examples (I (Lee et al., 2022).
"We chose EditPackFT for this experiment as it is the smallest dataset we use for training, allowing us quickly compare the two methods.
23

---

Published as a conference paper at COLM 2024
0.5
Masked Unmasked
Masking
pass@1
ExcessCode
Yes
41.6
0.26 + 0.14
Loss
0.3
No
40.5
0.43 + 0.20
0.2
Figure 6: Effect of loss masking on the performance of DeepSeekCoder--.77- Base on EditPackFT.
0.0
50
100
150
200
25D
300
350
Step
Training loss curves for ChepSeekCodder6.77-BB3 trained on EditPackFT with and without loss masking.
Figure
24

---

Published as a conference paper at COLM 2024
C Example CANITEDIT Benchmark Problems
We showcase four examples from the CANITEDIT benchmark, which we believe repre- sentative of the types of problems present in the dataset.
External Libraries in CANITEDIT Our benchmark includes 21 problems that import external libraries, which are libraries outside of Python's 's standard environment. We report the list of external libraries used and their number of appearances in the dataset: NumPy (13), Pandas (6), SciPy (3), scikit-learn (3), PyTorch (3), Z3 (2), autograd (2), Flask (1), vLLM (1)
oop refactor Figure 8 details a task where the model refactors code using object-oriented programming (OOP) principles. Initially, the code a function for formatting messages based on type. The refactoring involves creating Tex TextMessage and ImageMessage as sub- classes of an abstract Message class and implementing a MessageFactory for message con- struction. This task provides an example of perfective edit, focusing on reorganizing the code into an OOP style without adding new features. The transformation is quite significant, and the largest relative transformation in our dataset: from single function to multi-class OOP program. The goal is to assess the model's 's proficiency in converting functional code into well-structured OOP designs based on comprehensive instructions and for the model to restructure small programs into much larger ones. Our test suites verify both functional correctness and the proper hierarchical class structure.
group group.theory .thoory Figure 9 features task to modify class from representing group C4 to group C8, including its operations like inverse and product. The problem highlights domain-specific problems in CANITEDIT, this one being set in the context of cyclic groups. Testing domain-specific edits is crucial, especially when comparing the capabilities of large proprietary models like GITT-4 with smaller open models. It requires the model to transform the C4 class (representing a 4-element cyclic group) into the C8 class (for an 8-element group), requiring extensive edits across various code sections. This complexity presents significant test for other code editing approaches, such as fill-n-the-mmiddee (Bavarian et al., 2022; Fried et al., 2023), which may struggle with multiple edit locations (Yee & Guha, 2023). Key edits involve altering the size and elements methods. The necessary understanding for these modifications stems from group theory, which is not explicitly explained in the problem. This setup tests the model's capability to execute domaif-specifie edits where contextual knowledge is implied rather than provided.
strategy Figure 10 presents an open-ended problem where the model devises game strategy to defeat the already implemented CornerStrategy in Tic Tac Toe. This task repre- sents an adaptive edit, focused on developing new feature without altering existing classes. The uniqueness in this problem lies in the lack of providing rules for the game, but rather requiring the model to infer them through understanding of the code. Additionally, it leaves the strategy design entirely to the model's discretion. Our tests ensure that the Game class remain intact and that the model's strategy consistently outperforms CornerStrategy in the game.
budoku_solves Figure 11 presents sudoku solver problem leveraging the Z3 satisfiability modulo (SMT) solver. The problem starts with an incomplete solver that lacks checks for 3x3 subgrids, both in its solving logic and board validity function. In sudoku, each 3x3 grid must contain distinct numbers from to 9. The task involves adding these checks to ensure the solver can correctly solve a sudoku board. This problem assesses the model's capability to implement edits across different code sections. Although it uses Z3, in-depth knowledge of the library or SMT isn't 't required; the necessary features needed to solve the problem can be inferred from the existing code, which already includes checks for row and column uniqueness.
25

---

Published as a conference paper at COLM 2024
def process_message (message, message_type):
if message_type "text":
return f"Processed text message: (message)"
4
elif message_type = "image": return f"Processed image message with description: {message}"
else:
return "Unknown message type"
(a) 'before' code segment of the oop refactor problem (Figure 8).
Instruction
Type
Abstract the code into an itself. To do that, create an abstract class (ABC)" which can be initialized with "content" string. The class should have abstract method "peoc)ss which should return string. Create two children classes "TextMessage" and "ImageMessage" which implement the processs method. Finally, create a "Mssageeactorry" that has static method "get_message (message_ methods can be defined with the "sstaticmethod" decorator. The "get_message" method should return "Message" corresponding to the "message_ (either "text" "image"), and it should throw ValueError if the message__ype
version
Descriptive
content) -> Message"; static
not valid.
Make the code abstract class "Message', and children classes "TextMessage" and "ImageMessage" The "Message" class should have method process(seif) that returns the message which was given to the constructor. Also, create "Mssaggeactoryy" that has static method "get_message (message_tue type, content) Message"; should raise an exception if the message type is not supported.
Specifically, create
Lazy
(b) Instructions for the rop-refactor problem (Figure 8).
26

---

Published as a conference paper at COLM 2024
from abc import ABC, abstractmethod
class Message (ABC):
Abstract class for messages н**
def iinit_ content):
self content content
10
@abstractmethed
11
def process
12
pass
13
14
class TextMessage (Message):
15
в**
16
Concrete class for TextMessage
17
18
def process(self):
19
return f"Processed text message: (self. content)
20
21
class ImageMessage (Message):
22
23
Concrete class for ImageMessage
24
н**
25
def process(self)-
26
return f"Processed image message with description: (self.
27
class MessageFactory:
29
30
Factory class for creating message objects
31
32
estaticmethod
33
def get_message (message_ type, content):
34
if message_type == "text":
35
return TextMessage (content)
36
elif message_ type
"image":
37
return ImageMessage saanntttegccontettt (content)
38
else:
39
raise ValueError ( ("Unknown message type")
40
(c) Canonical solution for the oop_refactor problem (Figure 8).
Figure 8: The oop_refactor problem from CANITEDIT. This is prime example of perfective type of edit, as asks the model to refactor code using OOP principles, without adding any additional features.
27

---

Published as a conference paper at COLM 2024
import torch import numpy as np import torch.
class C4 (nn. Module): """eerreeenss the 04 class of group theory, where each element represents discrete rotation.
def _.init__
10
super () D___inii__0
11
self begffteer_bffer(( iddenttt'', torch. Tensor ((O.)))
12
13
def sselff:
14
.*" Outputs the size of this group. ""*
15
return
16
17
def (eemmnts(selffi
18
"" "Returns all the elements this group" "*"
19
return torch. tensor (CO., np.pi / 2, pi.pi, 3 np. pi 2])
20
21
def product h, g):
22
""" Computte the product of two elements g and in the group "M" return torch. remainder (h g, * np.pi)
23
24
25
def inverse h): """ """Computes the inverse of the element h in the group c4""" return torch. remainder(hh,
26
27
np.pi)
28
29
def matrix_repreeeeee ratrixxrppreeentaiioon(.... h):
30
Returns the matrix representation of this element""*
31
cos_t (or)
32
sin_t torch.sin(h)
33
representation torch. tensor ([
34
[cos_t, -sin_t],
35
[sin_t, cos_t]
36
J. device=self identity device)
37
return representation
38
(a) 'before' code segment of the group. problem (Figure 9)
Instruction
Type
Edit the class, which represents rotations 0, 90, 180 and 270 degrees, to represent the class C8 which represents rotations of e, 45, 90, 135, 180, 225, 270 and 315 degrees.
Descriptive
Edit the C4 class and its methods to represent the C8 group instead
Lazy
(b) Instructions for the group theory problem (Figure 9).
28

---

Published as a conference paper at COLM 2024
import torch import numpy np import torch. as
class C8 (nn. Module): "eprresents the с8 class of group theory, where each element represents discrete rotation.
6
def ._init
10
super () -nni-init_
11
self register_buffer ('identity' torch. Tensor ([o. )))
12
13
def
size (selff:
14
.*" Outputs the size of this group. ""*
15
return
16
17
def (lements(self))
18
.." "eRturns all the elements of this group"""
19
delta = np.pi /
20
return torch. tensor ([o.. delta, delta x 2, delta 3, delta delta delta 6, delta 71)
21
22
23
def Product(self. h, g):
24
Compute the product of two elements g and in the group "8"*"
25
return torch. remainder g, pp.
26
27
def inverse h)
28
"*" Comppttes the inverse of the element h in the group ca" return torch. remainder (-h, 2 np.pi)
29
30
def mtrri_repprenttttionno..... h):
32
.*" Returns the matrix representation of this element""* cos_t t = torch.cos
33
34
sin_t t sirch.sin(h)
35
representation torch. tensor
[cos_t, -sin_t],
37
[sin_t, cos_t]
38
1, device=self stl.eessel..ddetttyy device)
39
return representation
(c) Canonical solution for group_theary problem (Figure 9).
Figure 9; The group theory problem from CANITEDIT. This exemplifies the subset of domain-specific problems our benchmark.
29

---

Published as a conference paper at COLM 2024
from abc import ABC
from abc import abstractmethod from typing import List, Tuple
class Strategy
@abstractmethed
def returnMove (self, board: List[List[bool]]) -> Tuple[int, int]:
"etuurns tuple(row, column) which indicates where to move in a 3x3 grid.
10
pass
11
12
class CornerStrategy (Strategy))
13
def returnMove board: List[List[bool]] >> Tuple[int, int]:
14
if board[0][e]] None: return (0, 0)
elif [o][2[
None: return (0, 2)
16
elif board == None: return (2, 0)
17
elif board[2][2] == None: return (2, 2)
18
else: raise Exception
19
20
class Game:
21
def
__init_ player1: Strategy, player2: Strategy):
22
self playerOne player1
23
self playerTwo player2
24
self board [[None for range (3)] for range (3)]
25
def (eoyer1ton(sslf]) 26
27
playerTurn True while (not self. pelf.ppaaerXWon (True) and not self playerXWon (False) and not self gamedver))
28
30
strat self.playerOne if playerTurn else self Pwayer move = strat. (self. board)
31
32
self. mo
playerTurn
33
playerTurn not playerTurn
34
if self gwreovameeeeeeccm (): return False
35
else: return self. pla.e..aaaexXooot. (True)
36
37
def gameOver (self):
for row in self.board.
39
for col in row:
40
if col == None: return False
41
return True
42
43
def playerXWon x: bool):
44
for i in (3): if self. rowNX x): return True
45
46
for i in range
if self. colNX x): return True
48
downDiag = self board = x and self board[1][1] == x and self board[2][2] ==
49
upDiag self. b]ff.boar[[][0]] == and self belf.board[1][1] and self board[0][2]
50
return downDiag or upDiag
51
52
def rself, int, bool): for col in self.board if col x: return False return True
53
54
55
56
57
def csllf
for row in int, bool); if row[n] x: return False return True
58
60
61
(a) 'before' code segment of the strategy problem (Figure 10).
30

---

Published as a conference paper at COLM 2024
Instruction
Type
The following code describes a tic-tac-too toe game which takes in two strategies and determines who wins if they play each other. The "Strategy" class defines abstract method, 'returnMove (aoror)(board"'.. which returns tuple representing where this strategy will move, given board state. The "orrerSttateey" class is a subclass of "Strategy" with concrete implementation of "returnMove (board) The "Game" class constructor takes two strategies. determines if the first strategy provided will beat the other if they both take turns alternating between moves. There are two methods, "playerXWon" and gameOver" which determine how a game is won and when it is over Create a class "GodStrrttgyy" which extends "Strategy" such that Game (GoodStrategy() CornerStrategy ()) playerrrWWNN))) returns "True". This can not be solved by modifying the Game", "Strategy". or "CornerStrategy" classes in any way.
has a method "player1 which
Descriptive
Create strategy "oodSStratgy"' that beats "CornerStrategy" Do not modify the "Game class.
Lazy
(b) Instructions the strategy problem (Figure 10).
31

---

Published as a conference paper at COLM 2024
from abc import ABC
from abc import abstractmethod from typing import List, Tuple
class Strategy (ABC): 4
@abstractmethod
def returnMove (self, board: List[List(bool]]) -> Tuple[int, int]:
"Returrns tuple column) which indicates where to move in a 3x3 grid. pass
10
11
12
class CornerStrategy (Strategy):
13
def returnMove (self, board: List[List[bool]] -> Tuple[int, int]:
14
if board[0][0] None: return (0, 0)
15
elif board[0][2]
None: return (0, 2)
16
elif board[2][0] None: return (2, 0)
17
elif board[2][2] =
None: return (2, 2)
18
else: raise Exception
20
class GoodStrategy (Strattgy::
21
def Sanit__(self -> None:
22
super in._-intt_.
23
self turn def returnMove (self, board: List[List[bool]]) Tuple[int, int]:
24
25
self. turn += if self. turn == 1: return (0, 1) elif self. turn == 2: return (1, elif self.ture
26
27
28
3: return (2,
raise Exception
30
31
class Game:
32
def
__init_ player1: Strategy, player2: Strategy):
self playerOne player1
34
self pllyerrro = player2
35
self. board = [[None for in range (3)] for in range (3) ]
36
def playerlWon(sell))
37
def gameOver (self):
39
40
def playerxwon (self, bool):
41
def rself n: int, x: bool):
43
44
def csll, n: int, x: bool):
45
46
(c) Canonical solution for the strategy problem (Figure 10).
Figure 10: The strategy problem from CANITEDIT. This problem is a prime example of adaptive type of edit, and characteristic the open-endedness of the instructions, both descriptive and lazy.
32

---

Published as a conference paper at COLM 2024
fram typing inport List, Optional from import ArithRef, Int, solver, Distinct, And, sat, Intval
str, solver: solver) ListCListEArithhef111
Creates board of 23 variables from string representation beard. For unknown cells, make the value be 8, and for known cells, make the value be number from 1-9. .an
10
board []
11
for line_counter, line eeunerate splttteneel)))
12
row for char_counter, character cnunerate(iine.srrip))) if pegracter.isdigite.
13
14
15
num = int (character))
16
17
cell Int(f"cell
18
nun
19
solver (ddeeeel >= 1, cell <= 9))
20
row aow.append
21
elif
18:
22
solver.add
(nuv)))
23
(cll))))
24
if len (row) I= 9:
25
raise ValueError(
26
f "IInvalid column count board, must be (len(row)))
27
board. (row)
28
29
if len(bo))d) I= ValueError(
31
I'Inalldd row count of board, nust be 9, got (beardd)d))
32
33
return board 34
35
def sellvv: Solver, z3_board: List(LiistAArtthee(D))
36
Assert rows unique
37
row in 23_board; aolver-add((e) (Bistiincttr)r)
38
39
40
Assert columns for col in zip (ip solver (osstttttctt)))))))
41
42
43
srrint_bbarddbaae List[List[ine]]): for row in board:
45
46
print(row)
47
48
def check (boadd: List[List[int])) >> bool:
in board:
50
if len(set != 9:
return
52
53
for col in zip (*board):
54
(snt((cc))) return False
55
56
57
return True
59
def (board text: str) >> Optional[[ist((Lisssinn]]]
60
solver Solver ()
61
z3_board 1ttee_9xx_3__brrd(bbeeeeeee
solver)
62
beard: List[List[int]] [C] for in (9). assert_uniq(salver, z3_board)
64
if solver, check solver. == sat:
selver.nodell0
66
for 1, row enunerate (23_bbar2223br))) row [model. evaluateeeeaa for cell row] board[i] row return board else: return None
67
type: ignore
68
69
70
71
72
(a) 'before' code segment the sudoku.solver problem (Figure 11).
33

---

Published as a conference paper at COLM 2024
Instruction
Type
This version the sudoku solver and checker does not reflect the original game of sudoku; the original game also checks for the uniqueness of 3x3 subgrids in addition to the "assert_uniq" function nine 3x3 subgrids, and update the theck_validd function to make sure that input grids have unique 3x3 subgrids.
and columns. Update the add new constraints for all
Descriptive
Make both the sudoku solver and verifier support the nine 3x3 subgrids that are in the original sudoku game
Lazy
(b) Instructions for the sudoku.solver problem (Figure 11).
34

---

Published as a conference paper at COLM 2024
from typing inport List, Optional from 23 import ArithRef, Int, Solver. Distinct, And, sat, IntVal
def make_9x9_22 btb. str, solver: Solver) >> List[ust(Arithiee]]]
def assert_ usslrt_ solver, z3_board: ListtList(Aritthef]]
unique
for row in 23_board: (dd(tisinnct(row))
11
12
Assert columns unique for col in zip :+ solver. add (Cistinct
13
14
15
16
Assert 3x3 squares unique range 3): for in range 9, 3):
18
19
square [23_board[x][y]
20
for range(i, 1+3) for in range(1, }+3)] solver.add (wt..ddissnnnnct(ssrees)))
21
22
23
def print_ (boadd: List[List[ine]])
for in board:
25
print(row)
26
27
def check_valid (boadd:: List[List(int]]) bool:
28
for row in board:
29
(see((rw))
False
30
col (ibaaaa)d);
32
if (sn(se(co))) 1= return False
33
for 1 range
35
for in range 9, 3):
square [board[x][y]
37
for in range(i, i+3) for in range(i, i+3)1 len(set(square) 9: return False
39
return True
40
41
sboard str) (gtiona][List(LLsssinn]]]:::
42
solver = Solver ()
z3_board
salver) [[] for in range (9)
44
board: ListfList(int]] assert_uniq ussert_ 23_board)
45
46
if solver.
sat:
47
madel = solver.madel0
48
enunerate
49
row [model evaluste (eell () V type: ignore
cell row] board[i] row return board
51
52
53
else: return None
(c) Canonical solution for the modoku.solver problem (Figure 11).
Figure 11: The sudoku uudoeu problem from CANITEDIT. This problem uses the Z3 theorem proving library, and is an example of corrective type of edit, as it requires the model to correct an existing solver to include checks for 3x3 subgrids.
35

---

Published as a conference paper at COLM 2024
D Example Model Completions
This section analyzes various completions from the models evaluated, displaying both correct and incorrect examples to highlight their strengths and weaknesses.
D.1 Excess Code Generation
Figure 12 provides instance of EDITCODER-1.3b generating excess code. This case underscores the importance of the ExcessCode metric (55.1), which penalizes models for generating unneeded code. Here, the model, while correctly altering the flatten method in Tensor, unnecessarily adds an unflatten method, resulting in an ExcessCode score of 66.6 66.67. To accurately measure ExcessCode, completions must pass the test suite for proper line coverage evaluation. Figure 13 shows another excess code example where the completion is failing the test suite, preventing the evaluation of the ExcessCode metric.
D.2 Model-Specific Completion Capabilities
Different models exhibit unique competencies in completing different tasks. Figure 14 highlights a solution by EDITCODER-333 for a complex problem involving the modification of maze solver class to use the A algorithm with Manhattan distance heuristic instead of the existing Uniform Cost Search (UCS) algorithm. A common error observed, particularly in GPT-4, the incorrect persistence of heuristic values in the A algorithm, where the heuristic value persists across multiple iterations of the algorithm inside the frontier queue. In contrast, Figure 15 presents problem solved exclusively by OpenAI models, requiring domain-specific knowledge on monte carlo approximation for kl-divergence, a skill less common in smaller open-source models. This illustrates the balance between using OpenAl models for domain-specific issues and open-source models for more general code editing tasks.
36

---

Published as a conference paper at COLM 2024
class Tensor:
def iinit__sself matrix):
self. = matrix
def m(self): return (omself.matriy
def nsseff):
return len (self.matrix(01))
10
11
def relu(self):
12
for i in range for j in (sel..sen.)
13
14
self melf.matrixii][i] (0, self (aF..matrixit](1)
15
def flatten(self): 16
17
sofar [] for i in range n))):
18
for j range ())
20
sofar. (self matrix(j](i])
21
return sofar
(a) 'efoore' code segment of Figure 12.
Edit Instruction
Change "flatten" to flatten lists left to right, top down.
(b) Instruction for Figure 12.
37

---

Published as a conference paper at COLM 2024
class Tensor:
def __init__ (self, matrix):
self matrix = matrix
4
def m(self):
return len
def (self))
return len matrix(81)
10
def relu(self):
12
for i in (ange(self. for i in range (n):
13
14
self matrixi((i (0, self matrix((i(())
15
16
def Flatten(self)::
17
sofar [] for i in range for j range (self.
18
20
append (self matrix
21
return sofar
22
23
def unflatten flat): self matrix for i in range
24
25
26
self, matrix appendd for j in range(self.m))
27
28
self. delf.matrix(i]. append (flat n() il)
(c) Model-generated completion for Figure 12.
Figure 12: An example of EDIrCODER-1.3 generating excess code. Tasked with modifying the flatten method in Tensor, the model correctly alters flatten but also adds an unwanted unflatted method. This instance scores 66.67 on the ExcessCode metric.
38

---

Published as a conference paper at COLM 2024
inport
from collections inport Counter
from typing List, Dict
--init_- def
self.
corpus: List[List(str]],
k1: float 1.5,
10
11
>> None:
colffcoopuu_size len(porp
13
self.doc_freqs
14
self.corpus corpus
15
self
16
self.idf = ()
17
self.doc_len
18
self.avgdl -1 inff._initialize(
20
self.k1 =
21
self.b
22
23
def .initialize (self) >> None:
document self.corpuu:
25
frequencies
26
del_.doclen.doe (en.een..eeeeeeeeennddccccee
27
for word in docunent:
28
frequencies(ward] self. doc_fresssss (tresss.pppeddeeeeeeeeeeee
geequencies.get 8)
29
30
for document in self. doc_freqs:
31
ward, freq docurent. itens():
32
self.df [word] self of get (word, 0)
33
for word, freq in self.df ittem():
34
self.idf[werd] nath.log() [self. sizp
35
free 0.5) (freq 6.5))
36
self.avgd1 sum(self. fooel.dddo_een self.corpus.sii.
37
calculate__n25(
39
self,
40
pruuentt_index. int. query: List[str]
41
42
float:
43
44
docunent = self. worppss(docueenttiiiiii
dac_len (daccument)
46
for tern in query:
47
if term in wew.fdec_ffeesseeeeeeeeeeeeeeeee freastdocument_indeee
48
tern_freq self.doc_ffe dbcc_freestoocueeet_ineeee
49
score += self inf.idf[eerm tern_freq
50
(self.k1
51
/ (tern_freq self.k1 (1 self.b self.b doc_len segdl
53
return score
54
55
def rank (self, query: List[str]) -> List[float]:
56
scores [ calculate_bbnss bn25 query)
57
for idx range(self corpus_size)] return scores
58
(a) 'before' code segment of Figure 13.
Edit Instruction
Optimize the bm25 algorithm by avoiding frequency calculations.
(b) Instruction for Figure 13
39

---

Published as a conference paper at COLM 2024
import math
from typing import List, Dict
4
class BM25:
def -_init__
self,
corpus: List(List(str]), k1: float = 1.5,
b: float = 0.75
10
None:
11
self selfccorus
corpus
12
cllp.corpss_size
len (corpus)
13
avgdl = sum (len (len(doc) for doc in corpus)
14
/ self. corpus_sizee
15
self.k1 k1
16
self.b
17
18
def calculate_bm25
19
self,
document_index: int,
query: List[str] float:
22
)
23
doc_ freqs: List(Dict[str, int]] = []
24
df: Dict[str, int] idf (
25
26
for document in corpuc
27
frequencies: Dict[str, int] = 0
28
for word in document:
29
frequencies[word] frequencies get 0) if word not in df: df[word]
30
31
32
df [f[wod]] += doc_freqs. apgpnnd (frequencies) for word, freq in df items
33
34
35
idf[word] math. (1 wellf.corpus_iiiie
36
freq 0.5) (freq 0.5))
37
score 0.0
38
document self. corpus corpus(documenttiitt [document_index)
39
doc_len len (document)
40
for term in query:
41
if term in doc_ (errress[ddccmmt_indedd]]] term_freq score idf[term] term_freq
42
fwe.ffrrrsssueeeee [
43
44
* (self.k1 + 1)
45
/ (term_free self.k1 * (1 self.b
46
+ self.b doc_len / self avgdl))
47
return score
48
49
def rank query: List[str]) -> List[float]:
50
scores [self calculate_bm22 (idx, query) for idx in range (self. corpus_size)] return scores
51
52
(c) Model-gennerted completion for Figure 13.
Figure 13: An example of excess added code to the solution, which is incorrect produced by the EDITCODER-332 The model adds a "doc_len" instance attribute which not necessary for the solution and is not implemented in the proposed solution shown above. In addition, it is not necessary to store the document lengths as it is quick lookup to get the length of list in Python.
40

---

Published as a conference paper at COLM 2024
from typing ispert List, Literal, Tuple from queue inport PriorityQueue
2
Literal["up", "down", "left", "right") I 8 up. 1 down, left, = right
NoveIndex Literaile. 1. 3] vall, start, end 2,
10
class Naze:
11
def -iinit.. List[List(Cell]]):
13
relf Len(raze)
14
self + -ien(raze(M])
seir.find_ssarre
self.fand,erdd
17
18
def std_sttrr(seef))) -> Tuple[int, int]: for in renge(self. range if self. [ol == 2: return row, col found")
21
22
ValueErrer("M)
def tind_and -> Tuplofint, Int]: for roN in range(seif..rus)) roas)) for range cols): selo: return rew. col raise ValueError (No end found")
26
27
29
30
get_nesgnbors{
33
self, rew: int, col: int List(Tuplelint, int]]:
34
35
row
maze[rew 1][eal] neighbors-apeeee((ree and self. naze(rox 1][col] 1=
37
col)]
38
1f rew self neighbars. ((reu col ne inighbars. ( col 1)]
col))
maze[row]]coll
41
42
if col self. cols and self, rar[rroooo la 1:
((ron,
return neighbors
45
46
def solve(self) -> Tuplelint, List[Tuplo[int, int]]l:
earuses
nunber nodes expanded exists. The cost of each nove is ".0 visited set()
path
49
50
frantier #Fiorityyuauet)) (1)
frantier.pute ((O,
33
aspanded =
54
while not Trontier, Trontiorrennty()) path frantier. get()
current visited:
37
continue
58
visited ad((cureent)
current self.end:
61
return exparded, nex_path
62
For neighhor self-get.
neighbor visited:
nea_cost cast
frontiar.pute ppt (nev.cost, neighbor, nev_path))
66
return expanded,
(a) 'effore' code segment of Figure 14.
Edit Instruction
Change the "solve" function to use A* with manhattan distance instead of using UCS.
(b) Instruction for Figure 14.
41

---

Published as a conference paper at COLM 2024
from typing inport List, Literal, Tuple queue import PriarityQueue
Hove Literrll"upp" "down", "left", "right"] #
down, 2 left. right MoveIndex Literal 1, 3]
empty, wall, 2 start, 3 end tell = Literallo, 1, 2, 3]
10
def manhattan_distanee
11 a: Tuple(int, intl. b: Tuple(int, int]) int:
12 return [bs b[0]) abs(a[l] b(1l)
13
14 class
15
def
..init_. (self, naze: List[List(cel]]):
self naze
17
self rows len (maze)
18
self.cols (maze
19
self.start self. find_start()
20
self.end = self. (ind_end()
21
22
def find staar((ssse))) -> Tuple[int, int]: row range (self.rows):
24
for col in range (self cols): self.maze[rom]]cool return col
25
26
27
raise ValueError ("No start found")
28
29
def (self) >> Tuple(int, int]: row (onef
31
for col in range(self (self.cols). melf.meze[roo][coll col
33
34
raise VolueError("No end found")
35
36
def get_neighbers(
37
self, row: int, >> List(TupleLint, intll:
38
39
neighbors and self.naze[raw, 1][col]
40
41
(rrg col))
42
self. ((row seeighorss.apeend((eee self melf.naze[rww]ccal
neze.ne 1][col] col))
43
neighbors..
45
neighbors... neighhorrs.apeedd((nee
col 1))
46
cal self.col. - and self melf.mazeCrwwIcal 1] ls anighbors..ppneeee appond col 1))
47
48
return neighbors
50
def sslve(self) -> Tuple[int, List[Tuple[int, int]]l:
the manhattan distance find path fron start to end, returning the nunber of nodes expanded and the path if one exists. visited set()
**"Uses
hauristic
52
54
55
# Each entry in the PQ is tuple of (f(n), s(n), coard, path)
57
frontier = Priorityqueue()
frontier. ((O, e, self.start, (1)) expanded while not conttyer.enpty() cost, current, if current in visited: continue
59
60
61
frontier. get()
62
63
visited (eurree))))
64
new_path path + [current]
if current self
66
return expanded. new_path
67
for neighbor in self. celf.get.eeighborssccrrrre
68
neighbor in visited:
69
cost +
dirrraanddisttaeeenngaaaa self.end)
71
= g frontier puf put g. neighbor, new_path))
72
73
expanded +=
74
return expanded, []
(c) Model-gemerated completion for Figure 14.
Figure 14: A correct code edit solution generated by EDITCODER-33b. The model correctly converts the UCS implementation to A and adds new manhattan distance function as standalone utility. Out of the models evaluated, only EDITCODER-33b was able to solve this problem.
42

---

Published as a conference paper at COLM 2024
import torch
def kl kl_div(
q: torch. distributions Distribution,
p: torch. wwt.:diiiiiitssssooooo Distribution
) ->
torch. Tensor:
return torch. distributions .l_ vergenee (q, mean ()
(a) 'eefore' code segment of Figure 15.
Edit Instruction
Change kl_div compute a monte carlo approximation divergence given num_samples as a parameter, which by default is set to 100000.
(b) Instruction for Figure 15.
import torch
def kl_div(
q: torch. distributions. Distrrbbuto..iii..n
p: torch,
Distributioin
num_samples: int = 100000
) ->
torch. Tensor:
samples q. sample (num__smples,
return (q. lo.log__roo (samples) p. p.log_prob (samples)) mean ()
(c) Model-generated completion for Figure 15.
Figure 15: A correct code edit solution generated by GPT-4. Interestingly, GPT-4 can solve this problem while all fine-tuned models are unable to solve it. This problem requires knowledge of sampling technique for approximating the KL divergence between two distributions.
43

---

Published as a conference paper at COLM 2024
E Using LLMs in Code Editing Tasks
In this section, we provide brief overview of the use of LLMs in code editing tasks. We showcase two scenarios: (1) humans interacting with chat models to edit code, and (2) models automatically generating edits for code. For the former, we analyze large dataset of LLM chatbot interactions, "Imsys/ kmsys/Imsys-chat-
which can be found on HuggingFacce hub, and for the latter, we analyze a sample reflection generated by GPT-4 using the Reflexion algorithm (Shinn et al., 2023).
E.1 Human-Instructed Code Editing
def preprocess petrriginaa_dcccmeeee
rows build_rows aagginal_dccumntttt packed r rox_boxss = list (rows. values ())
4
if len (paccee_rrow_boxes] return None, None 1:
paragraph.classea get_paragraphs rpacked_
cluster_threshold
ALUSTER_THRESHOLLE
10
distance_thresholld
DHSTANCE_THRESSOOLEE
11
paragraph_boxes = mak.eparrrrr....oorrrrrriiioeooe baragraph-classer
12
rest of code omitted for brevity
13
(a) Instruction: Can you refactor this python code?
(function() ()
var 10;
y 20;
var 30;
var a function(b, c) (
return b
function(d, e) { return e;
10
};
11
var c function(f, g) {
12
return f gi
);
14
console (a(x, y));
15
console (b x));
16
console (cg( y));
17
3) O)
(b) Instruction: cool, now please refactor the snippet to have exactly the same logic and be readable as possible.
Figure 16: Two example human editing requests taken from the "Imsys/lmmys-chatt dataset which contains 1-million in-the-wild conversations from 25 conversational LLMs
We analyze large dataset of human interactions with different conversational LLMs, users to interact with highly capable chatbot. The dataset, "Imsys/Imsys-chatlll contains 1-million real conversations from 25 conversational LLMs of varying sizes and capabilities. We analyze the dataset to understand how humans interact with LLMs to edit code. We find that 4188 of the 1-million conversations contain code-related request, and that 831 of those conversations contain code editing request. We found this number by
44

---

Published as a conference paper at COLM 2024
searching for markdown-formatted code blocks in the conversations, therefore the actual number of code-related requests is likely higher. We analyzed a subset of code editing requests to understand the types of requests humans make to LLMs. We find that almost all of the requests are of the "lazy" kind that we include in CANITEDIT. We provide two examples of human editing requests in Figure 16. The first example is request to refactor Python code snippet, and the second example is request to refactor a JavaScript code snippet. As shown, these requests are very informal and direct, and do not provide any information about the desired solution. Other instructions we found that we think exemplify the type of instructions humans give LLMs include:
Please change use scrappy inslead request.
change this code to python
Can you change above code to not use histogram but use two for loops to create the his- togram?
Very cool. Now change it that it compresses each file using Iz4 and saves it to file with the same name and extension, + ".1z4"
E.2
Model-Generated Instructions for Editing Code
butMahhlessweights List[int], k: int)
2
You have bags. You are given 8-indexed integer array weights where weights[i] is weight of the ith narble. You are also given the integer k.
4
Divide the narbles into the bags according to the following rules: bag enpty.
7
If the ith narble and jth narble are in bag, then all marbles with an index between the that sane bag.
ith jth indices should also If bag consists of all the marbles with an index from i to inclusively, then the cast the bag weights[i] weights[]].
11
The score after distributing the narbles is the sum of the costs of all the k bags Return the difference between the meximum and minimun scores among marble distributions.
12
13
14
code onitted for brevity
15
16
PASSING TESTS
17
18
none of the tests passing
19
20
FAILING TESTS
21
assert PutMarbles((I. (EI, 11, 2) actual output:
23
assert putMarbles 31, 2) == actual output: inf
The implementation failed 2 out of the test cases provided. The issue lies in the calculation of the difference between the maximum and minimum scores among marble distributions. The function returns the difference between the score for bags and the score for bag, which is not correct calculation for problem statement. To fix this issue, we need to find the maximum and minimum scores among all possible distributions and then return the difference between them. This can be achieved by iterating through all possible distributions and keeping track of the maximum and minimum scores, and then returning their difference.
(a) Instruction:
Figure 17: An example of model-gennrated instruction for code editing. The instruction is generated by GPT-4 using the Reflexion algorithm (Shinn et al., 2023), by making the model reflect on unit test failures. The problem is from the LeetCode Hard problem set.
This section provides an example of code editing guided by instructions generated by GPT-4 using the Reflexion algorithm. Reflexion is a versatile algorithm developed for enhancing model output through environmental feedback, as detailed in Shinn et al. (2023). While its application extends across various tasks, including reasoning and decision-making its utility in program synthesis is particularly notable. The process starts with generating unit tests for program given its natural language description, followed by the creation and evaluation of a candidate program against these tests. If the program fails, Reflexion induces the model to produce reflection, identifying potential errors and suggesting corrections.
45

---

Published as a conference paper at COLM 2024
This reflection serves an instruction for modifying the failing program, which are both provided to the model to edit the failing program into a new candidate, iterating until passes all tests predetermined stop condition is reached.
We provide an example of a model-generated instruction for code editing in Figure 17, where the model was tasked with addressing problem from the LeetCodeHardGym problem set (Shinn et al., 2023). The instruction, precise and detailed, pinpoints the specific issue in the function's 's logic and suggests clear approach for rectification. It emphasizes iterating through marble distributions to calculate the maximum and minimum scores, method not implemented in the original code. This example showcases how Reflexion can guide models to not only identify errors in logic but also propose viable solutions. This kind of guided instruction is useful for enhancing the accuracy and efficiency of models in complex code editing tasks; however, it is important to note that the instruction is not complete solution, and that these models may produce misleading or incorrect instructions. The instruction is quite verbose compared to the human examples shown in Figure 16, and it is unclear how humans would interact with such an instruction, as this amount of detail is not necessary for the task at hand.
46


Reviews:
[{"review_id": "2FEh3BjUGr", "reviewer": "Reviewer_NZ9v", "summary": "This paper proposes a benchmark to evaluate the instruction-following code editing capability of selected baselines. \n\n- The authors include several kinds of editing instructions in the design.\n- The authors provide an instruction-tuning training dataset bundle specifically for code editing enhancement.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review and suggestions on our work.\n\n> Quantified correlation with the code generation capability.\n\nWe agree that code generation and code editing capability are  correlated. It's possible that if a model exhibits strong code generation capabilities, it will also show strong performance on our benchmark. In Appendix A.4 we provide a table with results of some of our models on the HumanEvalPack benchmark, which includes a code generation split (Synthetize). We compute the Pearson correlation coefficient between the Synthetize scores and the average between the Lazy and Descriptive scores on CanItEdit. We find a coefficient of 0.661, suggesting a moderate positive correlation. We will add this result to the revised version of the paper.\n\n> Lacking of human evaluation. It would be better if the authors could provide the performance of human annotators in the benchmark as a guideline for the (half) open-ended problems.\n\nWe would be happy to include this in the camera-ready if needed for acceptance.\n\n> if all the \"before\" segments do not met the requirements, do they still fall into any of the three task categories?\n\nWe categorized problems as corrective, perfective, or adaptive based on the changes applied in the ground truth solution.\nA model may produce an edit that does not conform to the labeled changed type. However, such an edit likely won't pass the test cases, as we have built them to reflect the correct change kind. We will clarify in the revised version.\n\n> Can the authors provide clarification on the \"relatively small\" and \"quite low\" from \"The mean Levenshtein distance between the ‘before’ and ‘after’ code segments is 197.1 characters, indicating that the changes are relatively small. We also analyze the distribution of the commit message lengths, and find that the mean token count is 10.1, which is quite low.\"?\n\nWe believe these numbers are small with respect to the ones found in our benchmark. In our benchmark, the mean Levenshtein distance is 302.1, while the mean token count is 81.7 for descriptive instructions and 35.6 for lazy instructions. We will clarify in the revised version.\n\n> Is there any overlap between the two training sets?\n\nWe have found 4 overlapping items between EditPackFT and Commits2023FT by finding exact matches between the before, instruction, and after chunks. These overlapping items are filtered out by the MinHash+LSH deduplication process when the two datasets are combined.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Compare to the portion of editing tasks in code-related conversations (19%) , the research on such specific kind of task is absent for LLMs. We need to study more about the LLMs on the capability of manipulating existing codes, distinguished from direct code generation from natural languages. \n\n- The authors provide a compact hand-crafted code editing instruction dataset for benchmarking. The curated commit-based fine-tuning dataset would be a useful resource for future research.\n- The proposed benchmark adapts the existing  *corrective*, *perfective*, and *adaptive* editing task classification system, and further uses a *descriptive-lazy* instruction categorisation dimension, which is a concise fit for LLM research. Based on which, there are in-depth analyses of the results appended.", "reasons_to_reject": "- **Quantified correlation with the code generation capability.**  Although this paper focuses on the code editing task, the authors should also include explore the correlation between the general code generation and editing performances of the baselines in the main text, given that there is assumption that the former could be the basis of the latter.\n- **Lacking of human evaluation.** It would be better if the authors could provide the performance of human annotators in the benchmark as a guideline for the (half) open-ended problems.", "questions_to_authors": "\"These suites were designed to rigorously evaluate whether the ’after’ segment met the problem requirements while ensuring the ’before’ code did not.\" -- if all the \"before\" segments do not met the requirements, do they still fall into any of the three task categories?\n\nCan the authors provide clarification on the \"relatively small\" and \"quite low\" from \"The mean Levenshtein distance between the ‘before’ and ‘after’ code segments is 197.1 characters, indicating that the changes are relatively small. We also analyze the distribution of the commit message lengths, and find that the mean token count is 10.1, which is quite low.\"?\n\nIs there any overlap between the two training sets?", "extra_scores": {}}, {"review_id": "ANlVOaek6h", "reviewer": "Reviewer_b7CL", "summary": "The paper proposes a novel benchmark called CANITEDIT, which is designed to evaluate the instructional code editing skills of Code LLMs. It comprises 105 manual code editing tasks, each accompanied by two sets of natural language instructions and a comprehensive test suite for validation. Additionally, it presents  a specially tailored training dataset for code editing. Through extensive experiments, the work highlights a significant performance contrast between closed and open models among state-of-the-art Code LLMs. Moreover, the presented training dataset has been shown to be effective in improving code editting capabilities of models at various sizes.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review and suggestions on our work.\n\n> A potential concern arises from the limited scope of the benchmark, which encompasses only 105 manually crafted instructional code editing problems. This constraint could result in a restricted coverage of domains within the evaluation.\n\nWe acknowledge that CanItEdit only contains 105 tasks, but we point out the following:\n\n1. We provide two variants of task specifications per task, effectively creating 210 problems of varying difficulty.\n2. Several other notable code generation benchmarks are composed of between 100-200 tasks, such as HumanEval [1] and LeetcodeHardGym [2]\n3. Among the 210 problems, CanItEdit covers over 5 broad categories topics, with 3 different expected change types.\n\nWe hope that the high quality and diverse range of tasks within our dataset effectively mitigate limitations posed by its modest size, ensuring robust and comprehensive evaluations.\n\n[1] Chen et al., Evaluating Large Language Models Trained on Code\n\n[2] Shinn et al., Reflexion: Language Agents with Verbal Reinforcement Learning", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The proposed benchmark is novel. It expands the evaluation scope of code LLMs to include instructional code editing skills, an often overlooked aspect in their assessment. \n* The evaluation is comprehensive and provides insightful findings. This work underscores a significant performance contrast between closed and open code LLMs. Moreover, experimental results verify the efficacy of the curated training dataset in enhancing code editing capabilities across diverse model sizes.\n* This paper is well organized and presented.", "reasons_to_reject": "* A potential concern arises from the limited scope of the benchmark, which encompasses only 105 manually crafted instructional code editing problems. This constraint could result in a restricted coverage of domains within the evaluation.", "questions_to_authors": "See the comments above.", "extra_scores": {}}, {"review_id": "xDxmIRs4sT", "reviewer": "Reviewer_R93Z", "summary": "This paper proposes a new benchmark to evaluate the instruction following ability of Large Language Models (LLMs) in code editing tasks. The evaluation exposes a significant gap between the capabilities of state-of-the-art open and closed models. For example, even GPT-3.5-Turbo is better than the best open-sourced model at code editing tasks. This paper also collects a new dataset named CANITEDIT comprising 105 meticulously constructed Python code editing challenges.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review and suggestions on our work.\n\n> The evaluation metric is limited as there are only pass@k and ExcessCode in metrics. Authors might propose some refined assessment techniques for code editing tasks or add human evaluation.\n\nWe note that the outputs of certain models did go through human inspection to ensure that all questions were unambiguous. During this process, we were able to get a qualitative understanding of how our models were performing. We provide some of these completions in Appendix D. We would be happy to include a more comprehensive human evaluation in the camera-ready if needed for acceptance.\n\n> Similar contributions with Guo et al.[1]. Authors might consider to compare with this work if possible.\n\nPlease note that Guo et al. appeared on ArXiv in April 2024, after the COLM submission deadline. Guo et al. sources problems from popular competitive programming websites, such as LeetCode and CodeForces, thus focusing on data structures and algorithms, while our benchmark encompasses several other topic categories. Additionally, our dataset features unique sets of dual (lazy and descriptive) instructions for each problem, unlike the hardcoded prompts used in their edit categories. We will acknowledge Guo et al. as concurrent work in our revision.\n\n> The code languages types are limited since there is only Python language.\n\nWe acknowledge that there are inherent limitations to a benchmark of problems written only in Python. However, mechanically translating Python benchmarks to other languages is an active research topic [1, 2, 3]. We identify expanding our benchmark to other languages as an opportunity for future work.\n\n[1] Cassano et al., MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation\n\n[2] Orlanski et al., Measuring The Impact Of Programming Language Distribution\n\n[3] Athiwaratkun et al., Multi-lingual Evaluation of Code Generation Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a new benchmark containing 105 Python code editing challenges to evaluate the ability of LLMs in code editing tasks. The topic is interesting and the contributions are significant.\n2. Several popular open-sourced code LLMs such as CodeLlama and DeepSeekCoder have been evaluated and compared with strong closed-sourced LLMs such as GPT-4.\n3. The presentation is clear and the writing is easy to follow.", "reasons_to_reject": "1. The evaluation metric is limited as there are only pass@k and ExcessCode in metrics. Authors might propose some refined assessment techniques for code editing tasks or add human evaluation.\n2. Similar contributions with Guo et al.[1]. Authors might consider to compare with this work if possible.\n3. The code languages types are limited since there is only Python language.\n\n\n[1] Guo et al., CodeEditorBench: Evaluating Code Editing Capability of Large Language Models.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SljtM5WCFV", "reviewer": "Reviewer_Xyri", "summary": "This paper proposes CanItEdit, a hand-curated evaluation benchmark consisting of 105 Python code editing problems, spanning three different categories: corrective edits, adaptive edits, and perfective edits. Each problem comes with two different types of natural language instructions: descriptive (entailing specific details for more explicit guidance) and lazy (minimal instructions that more closely resemble typical user queries). They further construct a training set for this task by filtering the CommitPackFT dataset and also mining additional examples from GitHub commits, where the instructions correspond to commit messages. They demonstrate out-of-the-box, open-source models (CodeLlama, Mixtral, DeepSeekCoder, StarCoder, OctoCoder) fall behind closed models (GPT-3.5-Turbo, GPT-4) by significant margins on this benchmark. By fine-tuning DeepSeekCoder-Base on their new training set, they observe improvements which appear to reduce the gap to a certain extent.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review and suggestions on our work.\n\n> Missing references\n\nWe will cite all of these and acknowledge Guo et al. as concurrent work.\n\n> Please provide more details about the dataset creation procedure.\n\nThe annotators were recruited through academic networks, these included researchers and engineers in formal methods, programming languages, AI, astrophysics, and mathematics, all of whom use Python heavily in their work. They were encouraged to come up with problems covering a range of difficulty to differentiate models that are very strong at code editing tasks from ones that are not. The annotators were provided with the breakdown of questions from the outset, and each was told what topic and change type to focus on. For each problem, the annotator wrote both the lazy and descriptive text, which later required approval by the lead. There was no starter code for the annotators, however, the lead gave annotators example problems to look at before they began. They were taught how to write the questions unambiguously to give the model a fair chance of passing the unit tests. For example, an ambiguous instruction may cause the model to implement a function body correctly but with a name that was not consistent with the unit test. The project lead reviewed all instructions and several model completions to make sure this wasn't the case.\n\n> Does this evaluation benchmark include [documentation-related] edits?\n\nThis benchmark does not deal with updates in documentation such as comments or README's, since they can not straightforwardly be tested with unit tests.\n\n> \"GPT-4…\" Could you explain how you came to this conclusion?\n\nIt is true that GPT-4 is certainly trained on data involving code. We meant is that it's not trained primarily on code, which is the case for models such as StarCoder. We will clarify in the revised version.\n\n> It's not clear why Table 4 was not presented like Table 3\n\nIn our ablations we were interested in identifying the overall most effective training configuration, thus we displayed the overall pass@1 and ExcessCode scores. We will include the table with lazy and descriptive columns in the revised version.\n\n> Table 6 does not include all models in Table 3.\n\nWe provided only EditCoder and OpenAI models in Table 6 to compare the best open models to closed models. We agree that showing all models would be insightful and will provide the full table in the main body for the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors propose an evaluation set that captures diverse use cases (e.g., editing based on detailed specifications like the descriptive instruction versus editing based on brief user queries like lazy instruction) as well as many different categories of edits: corrective edits (35 examples), adaptive edits (35 examples), and perfective edits (35 examples). I believe this is a novel and very useful contribution to the community as it allows more fine-grained evaluation of LLMs with respect to code editing tasks.\n- While I would like to get some more clarity on the details behind this dataset creation procedure (see below), the benchmark is hand-curated, making it likely a very high-quality evaluation set, relative to the few existing code editing datasets. This makes it a very valuable resource for the research community.\n- The authors defined a new metric ExcessCode which I believe is very clever and useful. Benchmarks like SWEBench which rely on execution metrics only do not penalize models for making unnecessary changes to the code. I think this metric is also an interesting contribution to the community.\n- The experiments are very comprehensive, spanning many different models and dataset ablations. This has highlighted important results with respect to the effect of model size on pass@1 and ExcessCode, and the gap between open and closed source models on this task.", "reasons_to_reject": "- Very minimal details about the dataset creation procedure are provided. (Please see questions below). This makes it very difficult to understand how the dataset was constructed and consequently assess its quality. \n- The dataset is Python-only and does not allow evaluating across different programming languages.\n\nMissing references:\n- Automating Code Review Activities by Large-Scale Pre-training (https://arxiv.org/abs/2203.09095)\n- CoditT5: Pretraining for Source Code and Natural Language Editing (https://arxiv.org/abs/2208.05446)\n- Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM (https://arxiv.org/abs/2403.19114)\n- CodeEditorBench: Evaluating Code Editing Capability of Large Language Models (https://arxiv.org/abs/2404.03543). (Note: This can be considered contemporary work and I have just included it for reference.)", "questions_to_authors": "- Please provide more details about the dataset creation procedure. For instance, how were the annotators recruited? What were the exact instructions provided to them? It is not clear how the annotators come up with problems. Are they from scratch or is a starting point provided to them? If a starting point is provided, how is it derived? Do these annotators write both the descriptive and lazy instructions. Please explain in more detail the following claim: \"Upon completion of a problem, the lead generated sample completions to ensure that the failures and successes were reasonable and consistent with the problem's intent.\" \n- A common use case is updating documentation (e.g., comments, README); however, it is not clear how such documentation edits can be evaluated with test cases. Does this evaluation benchmark include such edits?\n- The following claim is made in the paper: \"GPT-4, despite not being specifically trained on code-related tasks, surpasses…\" Could you explain how you came to this conclusion? It does seem that GPT-4 was trained on a significant amount of code, including come commits data.\n- It's not clear why Table 4 was not presented like Table 3, with results for \"Descriptive\" and \"Lazy\" shown separately. Do the ablations perform differently for these different use cases?\n- Despite motivating the different categories of edits, the main paper does not dive into this at all. I see that some additional results are provided in Table 6 of the Appendix, but this does not include all models in Table 3. Could you share details on whether the same trends are there across these categories for all the models presented in Table 3?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work introduces a code-editing benchmark and conducts evaluation for many open-sourced and commercial code models.
Finetuning experiments are also presented in the paper, showing that they are helpful in improving performance across the board.
While the size (Reviewer b7CL) and language coverage (Reviewers R93Z and Xyri) of the introduced benchmark are limited, the paper presents a first-step effort toward evaluating LLM code editing. The paper is well-written, and the results are clearly presented.

Reviewers NZ9v and R93Z have suggested adding human evaluation. I agree with them that either comparing human and model behaviors or having human feedback would be an interesting direction that lies in the intersection among NLP, program synthesis, psychology, and HCI; however, since the task itself is well-defined and can be evaluated by comparing execution results, I don't think this paper needs to compare the model behaviors with humans for acceptance.

I agree with Reviewer Xyri that more details about the dataset curation would be helpful. Please incorporate your response into the next version of the paper.

[comments form the PCs] It's important to follow the AC's recommendation regarding dataset curation details.

[At least one review was discounted during the decision process due to quality]
Decision: Accept