Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions

Authors: federico  cassano
luisa  li
akul  sethi
noah  shinn
abby  brennan-jones
jacob  ginesin
edward  berman
george  chakhnashvili
anton  lozhkov
carolyn jane anderson
arjun  guha
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-19 18:36:13

Abstract: A significant amount of research is focused on developing and evaluating
large language models for a variety of code synthesis tasks. These include
synthesizing code from natural language, synthesizing tests from
code, and synthesizing explanations of code. In contrast, the behavior of 
instructional code editing with LLMs is understudied.
These are tasks in which the model is provided a block of code and an instruction to modify the code. 
The editing instruction may ask for a feature to be added or removed, describe a bug and ask
for a fix, or ask for a different kind of solution.
We introduce a carefully crafted benchmark of code editing tasks and use it
to evaluate several cutting edge LLMs. Our evaluation exposes a significant gap
between the capabilities of state-of-the-art open and closed models. For
example, even GPT-3.5-Turbo is better than the best open model at
code editing tasks. We also introduce a new, carefully curated, permissively licensed training dataset of code editing tasks
coupled with natural language instructions.
Using this training dataset, we show that we can fine-tune open Code LLMs to significantly
improve their code editing capabilities,
closing the gap between open and closed models.
All code, data, and models are available at https://github.com/nuprl/CanItEdit.

Main paper: 
 introduction: 
 Large language models of code (Code LLMs) are becoming an essential tool for software engineering practice and research. There has been significant research on synthesizing code from natural language instructions, but comparatively less attention has been given to code editing tasks. However, LLM users expect models to be capable of editing code. For example, the LMsys dataset of in-the-wild conversations with chatbots (Zheng et al., 2023) has 4,188 conversations containing code, and 831 (19%) of these involve code editing, where the user prompts the model to update code based on natural language instructions (Appendix E). In general, code editing with an LLM encompasses activities like feature addition or removal, bug fixing, and code refactoring (Zhang et al., 2023b; Moon et al., 2023; Shinn et al., 2023; Chen et al., 2023; Olausson et al., 2023; Jin et al., 2023) .The ability to edit code is also essential for a model to be useful for an AI-focused code editor such as Cursor (Cursor, 2023) , Copilot Chat (Copilot, 2023) , or ChatGPT Advanced Instruction: Edit the C4 class and its methods to represent the C8 group instead.-class C4(nn.Module): +class C8(nn.Module):-"""Represents the cyclic group C4, + """Represents the cyclic group C8, where each element represents a discrete rotation.""" def __init__(self): super().__init__() def size(self): """Outputs the size of this group.""" -return 4 + return 8def elements(self): """Returns all the elements of this group""" -return torch.tensor([0., np.pi/2, np.pi, 3*np.pi/2]) + d = np.pi / 4 + return torch.tensor ([0., d, d*2, d*3, d*4, d*5, d*6, d*7] )Figure 1 : An abbreviated example of a code editing task from the CANITEDIT dataset (Figure 9 presents the full example). The model is tasked with editing the C4 group to represent C8 instead. The model is expected to infer the after code segment from the instruction and the before code segment, as shown in the inferred code diff.Data Analysis (ADA) (OpenAI, 2023a) . Cursor and Copilot Chat facilitate edits with humanwritten instructions. In contrast, ADA uses both human-written instructions and modelgenerated reflections (Shinn et al., 2023; Fan et al., 2023; Phung et al., 2023) to extend and edit code. This approach represents a step towards AI-driven code assistance. In both scenarios, instructional code editing is employed, which we define as a function M(c, I) → c ′ , where c is the original code, I is the instruction, and c ′ is the modified code. Figure 1 illustrates this process, showing how the model edits a code segment from a given instruction.Model-generated reflections and human-written instructions both describe desired code changes. However, they differ in the level of detail: reflections, usually more detailed, are generated by a model with access to the code, offering richer context and potentially a strategic plan for code modifications. In contrast, human-written instructions are typically shorter and less detailed but may express the true user's intent more clearly. We refer to these as descriptive and lazy instructions, respectively. We thoroughly analyze examples of such instructions in Appendix E.In this work, we introduce CANITEDIT, a novel dataset comprising 105 hand-crafted instructional code editing problems, featuring both descriptive and lazy instructions and an extensive hidden test suite. Designed to assess a model's proficiency in handling diverse code editing scenarios, CANITEDIT serves as a benchmark for evaluating state-of-the-art Code LLMs in instructional code editing. Our evaluation focuses on measuring the accuracy of a given model's ability to write correct code modifications without introducing unnecessary code. We conduct comprehensive assessments of closed and open models, revealing significant performance disparities between the leading closed and open models ( §5). To help address this gap, we propose a training dataset and methodology for code editing. Our findings demonstrate that fine-tuning open Code LLMs on this dataset can significantly enhance code editing performance ( §4).To summarize, we make four main contributions: (1) We introduce CANITEDIT, a dataset of instructional code editing problems, designed to evaluate the code editing capabilities language without specific edit locations (Hu et al., 2023; Li et al., 2023b; Muennighoff et al., 2023) . However, this advancement lacks benchmark evaluations to effectively measure the models' code editing skills. Notably, StarCoder (Li et al., 2023b) , the first LLM trained on an extensive dataset of commits using the format <before><commit message><after>, we have shown enhanced code editing capabilities ( §5). Before this study, StarCoder's practical code editing performance had not been assessed. StarCoder2 has replaced commits with pull requests and issues, which typically include more natural language (Lozhkov et al., 2024) .CodeEditorBench (Guo et al., 2024b) evaluates code editing using competitive programming problems. Unlike CANITEDIT, it uses fixed prompts and lacks diverse topics and dual instruction sets. The recent introduction of InstructCoder (Hu et al., 2023) , a model explicitly trained and evaluated for instructional code editing, marks a significant step towards code editing with LLMs. However, its evaluation involved GPT-4-generated (OpenAI, 2023b) and human-provided labels, which raises issues regarding reproducibility and comparability in future research. Moreover, the model has not been publicly released, prohibiting us from evaluating it on our benchmark. 
the canitedit dataset: 
 Benchmark Overview CANITEDIT is a dataset comprising 105 meticulously constructed Python code editing challenges. Each problem includes the input code segment (before), the expected code segment (after), the two types of natural language instructions (descriptive and lazy), and a hidden test suite. These challenges span a broad spectrum of computer science domains, such as data structures, algorithms, mathematics, language processing, and game programming, requiring knowledge of popular external Python libraries like NumPy, Pandas, PyTorch, and others. Table 1 presents general dataset statistics for CANITEDIT.Following Swanson (1976) and follow-up work (Levin & Yehudai, 2017) , we classify code editing tasks into three distinct categories based on their primary goal: a corrective edit fixes errors, a perfective edit enhances existing features, and an adaptive edit meets new requirements. We have 35 problems per category for an even distribution across the different types of code changes. The dual instruction formats test the models' ability to execute tasks based on the provided context: descriptive instructions provide comprehensive details for explicit guidance, while lazy instructions offer minimal direction, challenging the model to infer the required actions. Both instructions should lead to an equivalent after segment. Descriptive instructions serve to replicate situations where users provide specific specifications or another model outlines a plan. In contrast, lazy instructions resemble typical user queries for LLMs in code generation. As each problem has two distinct instructions, the dataset effectively contains 210 problems. We showcase examples from CANITEDIT in Appendix C.To manually construct CANITEDIT, we assembled a team of eight experienced Python programmers, each with different domain expertise, and appointed one as the lead. The team, recruited through academic networks, included researchers and engineers from various fields who use Python extensively. Our objective was to fill each change category (corrective, perfective, and adaptive) with 35 problems, for a total of 105 problems. The change category is determined by the type of edit required to transform the 'before' code segment into an 'after' code segment that passes the hidden test suite.Before starting, we provided the team with verbal and written guidance, a standard template, and an example problem. They were instructed to begin by writing a brief description of the problem and the applied changes for review and refinement by the lead. Next, they were tasked to write the 'before' code segment and hidden test suite, followed by the 'after' code segment, along with both lazy and descriptive instructions. The lead initially reviewed all problems in development, and they were additionally reviewed by the entire team in weekly meetings. Upon completion of a problem, the lead generated sample completions to ensure that the failures and successes were reasonable and consistent with the problem's intent.The team dedicated significant effort to developing comprehensive test suites for each problem, incorporating a variety of testing techniques such as unit tests, property-based testing, mocking, fuzzing, and integration tests. These suites were designed to rigorously evaluate whether the 'after' segment met the problem requirements while ensuring the 'before' code did not. To confirm the completeness and correctness of the test suites, we created an automated verification pipeline that ensured 100% line coverage and that the suite passed all tests with the 'after' code while failing at least one with the 'before' code.The team also manually reviewed the tests to ensure correctness and completeness.Annotators were encouraged to create problems of varying difficulty and were assigned specific topics and change types. The lead provided example problems and guidance on writing unambiguous instructions to ensure fair model evaluation. All instructions and several model completions were reviewed by the project lead to maintain consistency and clarity across the dataset. 
fine-tuning: 
 We describe our approach to fine-tuning Code LLMs for code editing tasks, focusing on the DeepSeekCoder-Base family (Guo et al., 2024a) , a variant of CodeLlama (Roziere et al., 2023) trained on 2 trillion tokens of GitHub code and natural language, using StarCoder's filtering rules (Li et al., 2023b) . These models, top-performing in code generation and open-access under a permissive license, show robust performance on CANITEDIT without specific training for instructional tasks ( §5).For our ablation studies, we focus on the model with 6.7 billion parameters, which offers an ideal balance between size and performance. This allows us to extrapolate results to larger models with more parameters without the need for extensive training. Following the most performant training strategy identified, we also fine-tune the 1.3b and 33b models to evaluate the impact of model size on code editing performance. Our fine-tuned models are referred to as EDITCODER. These models have been full-parameter fine-tuned by calculating the loss on only the 'after' code segment. Appendix B provides further details and experiments on the training process.We experiment with two training datasets we gathered: EditPackFT and Commits2023FT, which we describe below. EditPackFT We created the EditPackFT dataset by further filtering the Python split of the CommitPackFT dataset (Muennighoff et al., 2023) . CommitPack is an extensive dataset comprising 4TB of permissively licensed commits from a 2016 GitHub snapshot across various programming languages. CommitPackFT is a subset of CommitPack, filtered to be amenable for instruction-tuning Code LLMs. The primary criterion for CommitPackFT's selection involved retaining commits whose messages begin with an imperative verb, mirroring the typical structure of natural language instructions. We apply a series of additional filtering steps, which make the dataset more suitable for code editing. We remove any item that pass any of the following predicates:1. The presence of an empty 'before' or 'after' code segment, disregarding whitespace. 2. No change detected in the 'before' and 'after' code segments. 3. The inclusion of the words TODO, FIXME, or BUG in the 'after' code segment, which signals an incomplete commit. 4. Incorrect parsing of the 'after' code using the Python ast module.Originally, the dataset contained 56,025 commits, and after applying the filtering steps, we are left with 22,602. As shown by Figure 2a and Table 2 , the mean number of lines in the 'before' and 'after' code segments is 29.2. The mean Levenshtein distance between the 'before' and 'after' code segments is 197.1 characters, indicating that the changes are small relative to CANITEDIT's 302.1 characters. We also analyze the distribution of the commit message lengths, and find that the mean token count is 10.1, which is quite low compared to CANITEDIT's 81.7 tokens for the descriptive and 35.6 tokens for the lazy instructions. We further analyzed the original CommitPackFT dataset to ensure that these findings weren't artifacts of our filtering strategy: the full dataset has similar statistics.To address the limitations of EditPackFT, we developed the Commits2023FT dataset. This dataset is filtered from 416,792 Python file changes gathered from commits in permissively licensed GitHub repositories. We name the unfiltered dataset Commits2023. Our objective is to create a dataset akin to CommitPackFT, but with more recent data and a more diverse example length distribution. We employed the same filters on this dataset as used for EditPackFT, and also applied the initial filters from CommitPackFT, which includes only commits with messages that start with an imperative verb followed by at least a noun. Additionally, we only retain one file from multifile commits to avoid exact duplicate commit messages in our dataset. After this filtering process, we obtain a dataset comprising 24,129 Python file changes. Figure 2b presents a broader distribution of the number of lines in the 'before' and 'after' code segments, with an average of 119.3. We also observe a much larger change distribution, with a mean Levenshtein distance of 406.6, signaling that this is not only a dataset with larger code segments, but also contains more varied changes. Figure 3 illustrates a sunburst plot of the most frequent initial verbs in the commit messages of Commits2023FT, along with their corresponding root nouns. This set of verbs is slightly more varied than those in Edit-PackFT, featuring 199 unique verbs in comparison to 184. Furthermore, the token count distribution of the commit messages is twice as high and much more varied than that of EditPackFT, with a mean of 23.1 and a standard deviation of 35.2.For ablation analysis, we generated two additional datasets: Com-mits2023Raw25k and Commits2023FT+EditPackFT. Commits2023Raw25k consists of a random selection of 25,000 commits from Commits2023. We use this dataset to assess the impact of the filtering process on the final dataset. Commits2023FT+EditPackFT represents the combined dataset of Commits2023FT and EditPackFT. We find that the combination of Commits2023FT and EditPackFT yields the best results by a significant margin ( §5.2), and thus we train our final models on this dataset. We believe that these results are due to the increased amount of data and the expanded length distributions. 
evaluation: 
 In this section, we evaluate the performance of various open and closed models on the CANITEDIT benchmark, as well our fine-tuned models.We run the open-access models using Hugging-Face Transformers (Wolf et al., 2020) and vLLM (Kwon et al., 2023) . We use the following hyperparameters for all inference experiments: 2048 maximum new tokens, temperature 0.2, and top-p sampling cutoff of 0.95. Following Cassano et al. (2023b) , we sample 20 completions for each problem. We run all tests in a Docker container to mitigate the risk of malicious code execution.We evaluate several state-of-the-art models of varying sizes, fine-tuning some of them to build EDITCODER. We group the models into two categories: open, models which we have access to their weights, and closed, models which we do not. We prompt each model with their recommended prompt template. The specific templates used appear in Appendix A.6. The full list of models and their sizes appears in Table 3 .We employed two metrics to assess model performance: pass@k assesses functional correctness, and ExcessCode assesses conciseness and precision of code edits.• pass@k is the likelihood that at least one successful edit was made from k attempts, as assessed by the test suite. In this section, we show results only for pass@1, and evaluation results for pass@10 and pass@100 with higher temperatures can be found in Appendix A.2. • ExcessCode evaluates the presence of unnecessary code changes, as indicated by the fraction of changed lines not covered by the test suite. We calculate this metric by averaging the mean line coverage for passing completions across all problems, omitting those with no successful completions. The Python code used to calculate this metric is found at Appendix A.1. We additionally report the standard error of the mean for this metric.We draw several conclusions from the full results in Table 3 .Our evaluation indicates a significant performance disparity between open and closed models. GPT-4, despite not being primarily trained on code-related tasks but general instruction-following, surpasses DeepSeekCoder-Instruct 33b -the leading open source model -by an average of 13% in pass@1 for both descriptive and lazy tasks. DeepSeekCoder-Instruct stands out as the only open model exceeding any closed model's performance, surpassing GPT-3.5-Turbo for descriptive prompts.DeepSeekCoder-Instruct utilizes an undisclosed instruction-tuning dataset, therefore direct comparisons with other open models may not be entirely fair. In contrast, Mixtral-Instruct (Jiang et al., 2024) , comparable to OpenAI models in its general instruction-following training focus, significantly lags in performance against both closed models and open models specialized in code generation tasks. Lastly, CodeLlama-Instruct-70b, ranked second in instruction-following capabilities, was developed using a dataset of examples generated by Llama 2 70b with a larger focus on code generation tasks. This may explain its superior performance compared to Mixtral-Instruct.Descriptive prompts yield better performance than lazy prompts. Descriptive prompts result in a 8.68 absolute increase on average in pass@1 compared to lazy prompts, which may be the result of more detailed information and additional pointers in descriptive problems. Lazy instructions generally lead to lower ExcessCode for larger models. This may be indicative of lazy instructions introducing less noise in the task, as there are less tokens to attend to in the prompt given.Larger models perform better than smaller models. Model size correlates positively with pass@1, and negatively with ExcessCode, indicating that larger models are more adept at precise edits to code. This pattern is most clearly seen in the evaluation results of DeepSeekCoder-Base, StarCoderBase, and StarCoder2, where a steady increase in performance is seen along with an increase in model size (Appendix A.5).Models pre-trained on commits are better at code editing. Among open models, StarCoder is pre-trained on GitHub commits, while StarCoder2 focuses on GitHub issues. StarCoder outperforms similar-sized DeepSeek and CodeLlama models in our benchmark, despite their superior code generation capabilities. OctoCoder, a StarCoder-based model fine-tuned on instructions (Muennighoff et al., 2023) , shows lower pass@1 performance, suggesting instruction fine-tuning on commit-based models may reduce code editing efficacy. Conversely, StarCoder2, exchanging commits with issues in its training, improves in editing tasks with larger models (Appendix A.5). This may be attributed to extended training and modern architecture rather than data source shift. 4 : Ablation results of training DeepSeekCoder-6.7b-Base on different datasets and evaluating on CANITEDIT. We show the total number of tokens and items in each dataset, as well as the pass@1 and ExcessCode (EC) metrics for both the descriptive and lazy prompts.The reported sizes of the datasets are after deduplication.In addition to evaluating existing open models, we also fine-tuned pre-trained DeepSeek models ( §4) to build EDITCODER, which we now evaluate.Optimal Dataset: Commits2023FT+EditPackFT. In finding the best training dataset for DeepSeekCoder-6.7b-Base, the base model for EDITCODER, various ablation datasets were tested. Results in Table 4 show Commits2023FT+EditPackFT is the top performer for both descriptive and lazy instructions. The dataset's larger size and diverse data types, including varied commits, edits, and instructions, likely contribute to its superior performance.surpasses all open models in pass@1 for both descriptive and lazy instructions types, showing an overall 10.7% increase in pass@1 and a notable decrease in ExcessCode compared to its base model, DeepSeekCoder-Base-33b. Additionally, we see a substantial increase in pass@1 for every iteration of EDITCODER over its corresponding base model, with the largest improvement being a 45.1% increase at 6.7b.Both EDITCODER-33b and EDITCODER-6.7b outperform GPT-3.5-Turbo in pass@1 for descriptive instructions, with EDITCODER-33b also matching GPT-3.5-Turbo in for lazy ones. In higher temperature scenarios (Appendix A.2), EDITCODER-33b beats GPT-3.5-Turbo in both instruction types for pass@10 and pass@100, and even surpasses GPT-4 in pass@100 for descriptive instructions. Analysis in Appendix A.3 shows EDITCODER excels in corrective changes but is less effective in perfective changes. Further analysis in Appendix A.4 shows that EDITCODER-33b outperforms all models, including GPT-4, in simple single-function bug fixes, and retains synthesis capabilities. This demonstrates the effectiveness of targeted fine-tuning on code editing datasets, addressing the distinct needs of instructional code editing compared to general code generation. 
conclusion: 
 We present CANITEDIT, a benchmark designed to assess the instructional code editing skills of Code LLMs. It includes 105 hand-written code editing problems, each accompanied by dual natural language instructions: a "lazy" instruction that a human may write, and a "descriptive" instruction that may be generated by an agent revising code in a loop Limitations We evaluated LLMs in reproducing the entire 'after' code segment, which may not be the most token-efficient method. A potentially more efficient strategy would involve generating a list of specific changes to be applied to the 'before' code segment. Furthermore, our study does not explore varying prompt formats. Instead, we have adopted a format consistent with that used by other models (Li et al., 2023b) . Another limitation is the size of our final training dataset, which is relatively modest. We have not investigated the potential benefits of utilizing larger datasets, which could notably enhance performance, particularly with larger models. Our work only targets Python. Similar results may be possible for other high-resource programming languages, but low-resource languages may require additional effort (Cassano et al., 2023a) . We identify these areas as opportunities for future work. 
 
 Appendix: In this section, we provide several additional details about our evaluation. We provide the following additional details:1. A Python implementation for computing the ExcessCode metric in A.1.2. Results of our evaluation at higher sampling parameters in A.2.3. A deeper analysis of our results per change type in A.3.4. An evaluation of EDITCODER on HumanEvalPack in A.4.A deeper comparison of the first and second versions of StarCoder in A.5.6. Each prompt format used in our evaluation in A.6.For our evaluation we use the following versions of the OpenAI models, which at the time of writing were the latest stable versions:• GPT-4: gpt-4-0613• GPT-3.5-Turbo: gpt-3.5-turbo-0125 A.1In this section, we evaluate the performance of various models under higher sampling parameters compared to those used in our main evaluation ( §5).In §5, we assessed several models on CANITEDIT using standard parameters: temperature of 0.2, top-p of 0.95, and 20 samples. These parameters, often used in code generation tasks (Chen et al., 2021; Cassano et al., 2023b; a; Muennighoff et al., 2023; Li et al., 2023b; Lozhkov et al., 2024) Table 5 : Evaluation results of models on CANITEDIT at higher sampling parameters. We report the pass@1, pass@10, and pass@100 metrics for both the descriptive and lazy prompts, as well as the ExcessCode metric. The size of the model is reported if available.tokens while allowing for sampling of lower probability tokens at conservative levels, addressing surface form competition (Holtzman et al., 2021) , making it typically more effective than greedy decoding (Chen et al., 2021) .Higher Sampling Parameters For this evaluation, we increased the sampling parameters to assess the models' robustness under more diverse generation conditions. Following Chen et al. ( 2021), we adopted a temperature of 0.8, top-p of 0.9, and 100 samples. These aggressive parameters allow the model to explore a wider range of possibilities, useful when multiple completion attempts are possible. Due to the higher computational costs, we limited our evaluation to a subset of models compared to the main evaluation.Metrics For this evaluation, we expand our metrics to include pass@10 and pass@100, alongside the standard pass@1 and ExcessCode. pass@10 and pass@100 offer deeper insights into model performance by evaluating the success rate across the top 10 and 100 completions, respectively. These metrics are crucial for understanding how models perform in scenarios that permit multiple attempts, such as when users are provided with a range of completions to select from or when an external verifier is used to determine the best completion.The results of our evaluation at higher sampling parameters are shown in Table 5 . We draw several conclusions from the results.pass@1 decreases for open models. Closed models maintain consistent pass@1 performance under higher sampling parameters. In contrast, open source models generally exhibit a decline, showing a 2-3% reduction in pass@1 performance compared to the main evaluation (Table 3 ).Multiple trials benefit open source models. Open source models significantly improve with multiple trials, showing larger gains in pass@10 and pass@100 compared to closed models, which also improve but to a lesser degree. Specifically, EDITCODER-33b outperforms all models, including GPT-4, in pass@100 for descriptive instructions and matches closely in pass@10. However, EDITCODER-33b lags behind GPT-4 in lazy instruction scenarios across all metrics and tends to generate more excess code for both prompt types. We expect that the performance of EDITCODER on lazy instructions will improve with more data and larger pre-trained models.Lazy instructions benefit more than descriptive instructions from multiple trials. The performance disparity between descriptive and lazy instructions persists, even under higher sampling parameters and multiple trials, as seen in pass@10 and pass@100. Despite this, the rate of improvement from multiple trials is greater for lazy instructions, with increases of 57.76% in pass@10 and 22.57% in pass@100, surpassing the gains for descriptive instructions, which are 48.18% and 17.23% respectively. This indicates a more pronounced benefit from multiple attempts in scenarios involving less structured prompts.Significant increase in ExcessCode. The average ExcessCode metrics for both descriptive and lazy instructions, at 0.507 and 0.449 respectively, have increased from the main evaluation's averages of 0.392 and 0.235.1 This is expected, as higher sampling parameters tend to yield a broader range of completions, consequently resulting in an increase in superfluous code. We report the pass@1 and ExcessCode metrics for each change type, as well as the size of the model if available. Results for lazy and descriptive prompts are aggregated across all problems. pass@1 is abbreviated to p@1. Closed models are the rows highlighted in gray.In this section we analyze the results of EDITCODER against the OpenAI models on CAN-ITEDIT per change type. We hope to gain insights into the strengths and weaknesses of these models for different types of code changes. Table 6 shows the results of our analysis, we aggregate the results for lazy and descriptive prompts across all problems, this is done for conciseness and to minimize the noise of our results, as each pass@1 and ExcessCode metric is calculated across 70 problems per change type, instead of 35. We utilize the same sampling parameters as in our main evaluation. Our key findings include:• GPT-4 outperforms other models in functional correctness across all types of changes.• Corrective changes typically incur minimal excess code, with some models achieving perfect scores in this area. Adaptive changes, intuitively, tend to introduce the most excess code.• Our EDITCODER-33b model surpasses GPT-3.5-Turbo in both corrective and adaptive changes, while EDITCODER-6.7b shows comparable performance to GPT-3.5-Turbo in these categories. However, for perfective changes, both EDITCODER-33b and EDITCODER-6.7b underperform compared to GPT-3.5-Turbo. This suggests a potential improvement in training data for perfective changes. As demonstrated in Figure 3 , verbs associated with perfective changes, such as refactor or improve, appear less frequently than those related to corrective or adaptive changes, such as fix or add, respectively. Artificially balancing the dataset with more examples of perfective changes could potentially enhance EDITCODER's performance. We leave this as an area for future work.• According to our dataset, the most challenging changes are perfective, followed by adaptive, with corrective being the simplest. Muennighoff et al. (2023) . The rest of the results are from our evaluation following the same methodology as in Muennighoff et al. (2023) .To draw similarities and differences between CANITEDIT and HumanEvalPack, in this section we evaluate models on the Python subset of HumanEvalFix and HumanEvalSynthesize.Results are available in Table 7 .Benchmark Overview HumanEvalPack is a benchmark comprised of 164 single-function problems aimed at evaluating both code generation and code editing. The problems are designed to not require domain-specific knowledge or familiarity with popular external libraries. HumanEvalFix contains only corrective code changes, while HumanEvalSynthesize purely focuses on code generation, tasking the model to generate a function from its signature and docstring.In this benchmark, EDITCODER-33b outperforms even GPT-4 in fixing bugs, while maintaining competitive synthesis capabilities against models trained on general instructional data, with EDITCODER-6.7b outperforming even larger models like CodeLlama-Instruct-34b. Additionally, we find a large disparity between the performance of models on HumanEvalFix and HumanEvalSynthesize for DeepSeekCoder-Instruct-1.3b, which performs significantly worse in fixing bugs than in synthesizing functions. These results demonstrate that HumanEvalPack and CANITEDIT complement each other: the former focuses on single-function algorithmic and puzzle-like problems, while the latter emphasizes code editing tasks requiring broader knowledge of software engineering concepts in a wide range of domains.To illustrate typical problems in HumanEvalPack, we selected examples from HumanEvalSynthesize and HumanEvalFix. Listing 2 presents a HumanEval-Synthesize problem where the model must complete a function based on its signature and docstring. Listing 3 demonstrates an incorrect function implementation from HumanEval-Fix along with it's ground truth unit test suite, where the model's task is to correct the implementation. Unlike in CANITEDIT, the model must infer the correct implementation solely from the faulty code and the test suite, without explicit instructions. One could argue that this falls under intrinsic code editing, rather than instructional code editing, since the model is not given any instructions about the intent of the function, making this benchmark more suitable for evaluating works such as Li et al. (2023a) and Gupta et al. (2023) .Write a Python function `has_close_elements ( numbers : List [ float ], threshold : float ) -> bool `to solve the following problem : Check if in given list of numbers , are any two numbers closer to each other than given threshold . >>> has_close_elements ([1.0 , 2.0 , 3.0] , 0.5) False >>> has_close_elements ([1.0 , 2.8 , 3.0 , 4.0 , 5.0 , 2.0] , 0.3) TrueListing 2: The prompt of a problem in HumanEvalSynthesize. The task for the model is to complete the function.def unique_digits (x): odd_digit_elements = [] for j , i in enumerate (x): if all ( int (c) % 2 == 1 for c in str (i)): odd_digit_elements . append (i) odd_digit_elements . append (j) return sorted ( odd_digit_elements ) def check ( unique_digits ):assert unique_digits ([15 , 33 , 1422 , 1] ) == [1 , 15 , 33] assert unique_digits ([152 , 323 , 1422 , 10] ) == [] assert unique_digits ([12345 , 2033 , 111 , 151] ) == [111 , 151] assert unique_digits ([135 , 103 , 31] ) == [31 , 135] check ( unique_digits ) Fix bugs in unique_digits .Listing 3: The prompt of a problem in HumanEvalFix. The implementation is incorrect, and the task for the model is to re-implement the function correctly.We hypothesize that the performance of models on CAN-ITEDIT is correlated with their performance on general code generation tasks. Intuitively, if a model exhibits strong capabilities in code generation tasks, it is likely to perform well on code editing tasks as well. We quantify the correlation between the performance of models on CANITEDIT and HumanEvalPack by calculating a Pearson correlation coefficient between the pass@1 metric of models on CANITEDIT and HumanEvalPack Synthesize, utilizing models on which we have results for both benchmarks. We find a correlation of 0.661, indicating a moderate positive correlation between the two benchmarks.StarCoder Models The first version of StarCoder models were pre-trained on several gigabytes of GitHub commits, as discussed in (Li et al., 2023b) Lozhkov et al. (2024) , rather than to the superiority of issue data over commit data for code editing tasks. However, we also believe that for utilizing StarCoder2 models directly for code editing tasks, the issue prompt format is a viable alternative to the commit format previously utilized by StarCoderBase models. We provide the prompt format we utilized for StarCoder2 models in Figure 5 .We evaluate all of our models on CANITEDIT using the same evaluation pipeline. However, for each model, we may utilize different prompts to generate the completions. These prompts are most aligned to how the model was trained, and are intended to maximize the model's performance on the task, while keeping the prompts as similar as possible across models. Figure 4 shows the prompts used for each model. For a fair comparison, we evaluate all models not trained on commits or explicit code editing tasks using a basic 1-shot prompt, showing the model how to add a sub function to a code segment with a add function, and changing the variable names from a and b to x and y.Furthermore, given the natural language characteristics of GitHub issue data, significant prompt-engineering was required to facilitate code editing tasks for StarCoder2 models.The specific prompt format used for StarCoder2 models is provided separately in Figure 5 .<user> You are PythonEditGPT. You will be provided the original code snippet and an instruction that specifies the changes you need to make. You will produce the changed code, based on the original code and the instruction given. Only produce the code, do not include any additional prose.## Code Before ```py def add(a, b): return a + b ``# # Instruction Add a "sub" function that subtracts two numbers. Also write docstrings for both functions and change a,b to x,y. <assistant> ## Code After ```py def add(x, y):"""Adds two numbers.""" return x + y def sub(x, y):"""Subtracts two numbers.""" return x -y ``< user> You are PythonEditGPT. You will be provided the original code snippet and an instruction that specifies the changes you need to make. You will produce the changed code, based on the original code and the instruction given. Only produce the code, do not include any additional prose. ## Code Before ```py {before} ``# # Instruction {instruction} (a) Conversation template utilized for all chat models without a 'system' prompt. This is the prompt utilized for OctoCoder. <system> You are PythonEditGPT. You will be provided the original code snippet and an instruction that specifies the changes you need to make. You will produce the changed code, based on the original code and the instruction given. Only produce the code, do not include any additional prose. <user> ## Code Before ```py def add(a, b): return a + b ``# # Instruction Add a "sub" function that subtracts two numbers. Also write docstrings for both functions and change a,b to x,y. <assistant> ## Code After ```py def add(x, y):"""Adds two numbers.""" return x + y def sub(x, y):"""Subtracts two numbers.""" return x -y ``< user> ## Code Before ```py {before} ``# # Instruction {instruction} (b) Conversation template utilized for all chat models with a 'system' prompt. The prompt is then adapted to the specific model chat format. This is the prompt utilized for: GPT-4, GPT-3.5-Turbo, CodeLlama-Instruct, and DeepSeekCoder-Instruct models. <commit_before> {before} <commit_msg> {instruction} <commit_after> (c) Prompt utilized for StarCoder and StarCoder-Base models of all sizes. StarCoder models are trained on commits in this format (Li et al., 2023b) . Please someone help me. Can you also provide the full code with the change? <issue_comment>username_1: Sure, no problem. I will be able to help. I am an expert in editing Python code.Here is the full code with the change: ```py def add(x, y):\"\"\"Adds two numbers.\"\"\" return x + y def sub(x, y): \"\"\"Subtracts two numbers.\"\"\" return x -y ``Ù pvotes: 200<issue_comment>username_0: Thank you so much! I have another program in Python that I'd like to change.Here is the code for the program: ```py {before} ``T he change I'd like to make is: {instruction} Please someone help me. Can you also provide the full code with the change? Upvotes: 100<issue_comment>username_1: Sure, no problem. I will be able to help. I am an expert in editing Python code.Here is the full code with the change: ```py {after} ``F igure 5: Prompt utilized for StarCoder2 models. StarCoder2 models are trained on GitHub issue data, which makes this prompt format amenable to code editing tasks (Lozhkov et al., 2024) .In this section we provide details on the training process for EDITCODER and ablation results on the loss masking technique used in training.For training all of our EDITCODER models, we utilize a fine-tuning pipeline based on the HuggingFace Transformers library (Wolf et al., 2020) . Additionally, we utilize DeepSpeed ZeRO 3 (Rajbhandari et al., 2020) to efficiently shard the model across multiple GPUs. Due to memory constraints, we offload the optimizer to the CPU for the 33b model. We also use FlashAttention 2 (Dao, 2023) to speed up training on large context window sizes. All of our models are trained on a single machine equipped with 8 NVIDIA H100 (80GB) HGX GPUs. The effective micro-batch size is set at 32 (4 gradient accumulation steps, with a single batch per GPU). We utilize the AdamW optimizer with a learning rate of 2 × 10 -5 , a linear decay scheduler, and 10 warmup steps. These parameters were chosen based on previous work on fine-tuning for code generation tasks (Cassano et al., 2023a) , it is likely that we could get superior results by running a hyperparameter search. To facilitate reproducibility, we set the random seed to 42 for all experiments.Prior to training, we shuffled the dataset randomly and deduplicated2 it following the method outlined by Li et al. (2023b) . This process combines MinHash (Broder, 2000) and Locality Sensitive Hashing (LSH) (Leskovec et al., 2014) . We format the training data as a prompt, with the 'before' code segment followed by the 'instruction' and the 'after' code segment, and mask the loss calculation to only consider the 'after' code segment. All models underwent training for 8 epochs, with a packed context window of 8192 tokens, including padding for the remaining tokens. We select the model from the epoch with the highest performance on a held-out validation set. The number of epochs chosen for each EDITCODER is the following:• EDITCODER-1.3b: 8• EDITCODER-6.7b: 4• EDITCODER-33b: 2As shown by the number of epochs, we found that larger models overfit to the data more quickly, suggesting that we could achieve better results with a larger dataset.In our training pipeline, we mask the loss calculation to only consider the 'after' code segment. The intuition behind this is that we don't need the model to learn how to reproduce the 'before' and 'instruction' segments, as these are always going to be provided as input to the model at inference time. The exact prompt format we use for training is shown in Figure 4d .We wish to verify that this loss masking is beneficial for our task. To assess our hypothesis, we train two DeepSeekCoder-6.7b-Base models on EditPackFT3 , one with loss masking and one without, calculating the loss on all tokens. We then evaluate both models on CANITEDIT, and report the pass@1 and ExcessCode metrics in Figure 6 . The reported result with loss masking is the same as the one reported in Table 4 . We find that the model trained with loss masking outperforms the model trained without it, and leads to a decrease in ExcessCode and its standard error. Furthermore, we plot the training loss curves for both models in Figure 7 . We observe that the model trained with the loss masking technique is more stable and converges faster than the model trained without it. We showcase four examples from the CANITEDIT benchmark, which we believe are representative of the types of problems present in the dataset.External Libraries in CANITEDIT Our benchmark includes 21 problems that import external libraries, which are libraries outside of Python's standard environment. We report the list of external libraries used and their number of appearances in the dataset: NumPy (13), Pandas (6), SciPy (3), scikit-learn (3), PyTorch (3), Z3 (2), autograd (2), Flask (1), vLLM (1) oop refactor Figure 8 details a task where the model refactors code using object-oriented programming (OOP) principles. Initially, the code is a function for formatting messages based on type. The refactoring involves creating TextMessage and ImageMessage as subclasses of an abstract Message class and implementing a MessageFactory for message construction. This task provides an example of a perfective edit, focusing on reorganizing the code into an OOP style without adding new features. The transformation is quite significant, and the largest relative transformation in our dataset: from a single function to a multi-class OOP program. The goal is to assess the model's proficiency in converting functional code into well-structured OOP designs based on comprehensive instructions and for the model to restructure small programs into much larger ones. Our test suites verify both functional correctness and the proper hierarchical class structure. group theory Figure 9 features a task to modify a class from representing group C4 to group C8, including its operations like inverse and product. The problem highlights domain-specific problems in CANITEDIT, this one being set in the context of cyclic groups. Testing domain-specific edits is crucial, especially when comparing the capabilities of large proprietary models like GPT-4 with smaller open models. It requires the model to transform the C4 class (representing a 4-element cyclic group) into the C8 class (for an 8-element group), requiring extensive edits across various code sections. This complexity presents a significant test for other code editing approaches, such as fill-in-the-middle (Bavarian et al., 2022; Fried et al., 2023) , which may struggle with multiple edit locations (Yee & Guha, 2023) . Key edits involve altering the size and elements methods. The necessary understanding for these modifications stems from group theory, which is not explicitly explained in the problem. This setup tests the model's capability to execute domain-specific edits where contextual knowledge is implied rather than provided.strategy Figure 10 presents an open-ended problem where the model devises a game strategy to defeat the already implemented CornerStrategy in Tic Tac Toe. This task represents an adaptive edit, focused on developing a new feature without altering existing classes. The uniqueness in this problem lies in the lack of providing rules for the game, but rather requiring the model to infer them through understanding of the code. Additionally, it leaves the strategy design entirely to the model's discretion. Our tests ensure that the Game class remain intact and that the model's strategy consistently outperforms CornerStrategy in the game.sudoku solver Figure 11 presents a sudoku solver problem leveraging the Z3 satisfiability modulo (SMT) solver. The problem starts with an incomplete solver that lacks checks for 3x3 subgrids, both in its solving logic and board validity function. In sudoku, each 3x3 grid must contain distinct numbers from 1 to 9. The task involves adding these checks to ensure the solver can correctly solve a sudoku board. This problem assesses the model's capability to implement edits across different code sections. Although it uses Z3, in-depth knowledge of the library or SMT isn't required; the necessary features needed to solve the problem can be inferred from the existing code, which already includes checks for row and column uniqueness.def process_message ( message , message_type ): if message_type == " text ": return f" Processed text message : { message }" elif message_type == " image ": return f" Processed image message with description : { message }" else :return " Unknown message type "(a) 'before' code segment of the oop refactor problem (Figure 8 ).Abstract the code into an object -oriented version of itself . To do that , create an abstract class `Message ( ABC ) `, which can be initialized with a `content string . The class should have an abstract method `process ( self ) `, which should return a string . Create two children classes `TextMessage `and `ImageMessage `, which implement the `process `method . Finally , create a `MessageFactory `that has a static method `get_message ( message_type , content ) -> Message `; static methods can be defined with the `@staticmethod decorator . The `get_message `method should return `Message `corresponding to the `message_type `( either `text `or `image `) , and it should throw a ValueError if the `message_type `is not valid .Make the code object -oriented . Specifically , create an abstract class `Message `, and children classes `TextMessage `and `ImageMessage `. The `Message `class should have a method `process ( self )`that returns the message which was given to the constructor . Also , create a `MessageFactory `that has a static method `get_message ( message_type , content ) -> Message `; should raise an exception if the message type is not supported .(b) Instructions for the oop refactor problem (Figure 8 ). from abc import ABC , abstractmethod class Message ( ABC ):""" Abstract class for messages """ def __init__ ( self , content ): self . content = content @abstractmethod def process ( self ): pass class TextMessage ( Message ):""" Concrete class for TextMessage """ def process ( self ): return f" Processed text message : { self . content }" class ImageMessage ( Message ):""" Concrete class for ImageMessage """ def process ( self ): return f" Processed image message with description : { self . content } " class MessageFactory :""" Factory class for creating message objects """ @staticmethod def get_message ( message_type , content ): if message_type == " text ": return TextMessage ( content ) elif message_type == " image ":return ImageMessage ( content ) else :raise ValueError (" Unknown message type ") (c) Canonical solution for the oop refactor problem (Figure 8 ).Figure 8 : The oop refactor problem from CANITEDIT. This is a prime example of a perfective type of edit, as asks the model to refactor code using OOP principles, without adding any additional features. """ Outputs the size of this group . """ return 4 def elements ( self ):""" Returns all the elements of this group """ return torch . tensor ([0. , np . pi / 2, np .pi , 3 * np . pi / 2]) def product ( self , h , g):""" Compute the product of two elements g and h in the group C4 """ return torch . remainder (h + g , 2 * np . pi ) def inverse ( self , h):""" Computes the inverse of the element h in the group C4 """ return torch . remainder (-h , 2 * np . pi ) def matrix_representation ( self , h):""" Returns the matrix representation of this element """ cos_t = torch . cos (h) sin_t = torch . sin (h) representation = torch . tensor ([ [ cos_t , -sin_t ] ,[ sin_t , cos_t ] ], device = self . identity . device ) return representation (a) 'before' code segment of the group theory problem (Figure 9 )Edit the C4 class , which represents rotations of 0, 90 , 180 and 270 degrees , to represent the class C8 , which represents rotations of 0, 45 , 90 , 135 , 180 , 225 , 270 and 315 degrees .Edit the C4 class and its methods to represent the C8 group instead(b) Instructions for the group theory problem (Figure 9 ). """ Outputs the size of this group . """ return 8 def elements ( self ):""" Returns all the elements of this group """ delta = np . pi / 4 return torch . tensor ([0. , delta , delta * 2, delta * 3, delta * 4, delta * 5, delta * 6, delta * 7]) def product ( self , h , g):""" Compute the product of two elements g and h in the group C8 """ return torch . remainder (h + g , 2 * np . pi ) def inverse ( self , h):""" Computes the inverse of the element h in the group C8 """ return torch . remainder (-h , 2 * np . pi ) def matrix_representation ( self , h):""" Returns the matrix representation of this element """ cos_t = torch . cos (h) sin_t = torch . sin (h) representation = torch . tensor ([ [ cos_t , -sin_t ] ,[ sin_t , cos_t ] ], device = self . identity . device ) return representation (c) Canonical solution for the group theory problem (Figure 9 ). The following code describes a tic -tac -toe game which takes in two strategies and determines who wins if they play each other . The `Strategy `class defines an abstract method , `returnMove ( board ) `, which returns a tuple representing where this strategy will move , given a board state . The `CornerStrategy `class is a subclass of `Strategy `with a concrete implementation of `returnMove ( board ) `. The `Game `class constructor takes in two strategies . It has a method `player1Won `which determines if the first strategy provided will beat the other if they both take turns alternating between moves .There are two methods , `playerXWon `and `gameOver `which determine how a game is won and when it is over . Create a class `GoodStrategy `which extends `Strategy `such that `Game ( GoodStrategy () , CornerStrategy () ). player1Won () returns `True `. This can not be solved by modifying the `Game `, `Strategy `, or `CornerStrategy `classes in any way .Create a strategy `GoodStrategy `, that beats `CornerStrategy `. Do not modify the `Game `class .(b) Instructions for the strategy problem (Figure 10 ). ... def gameOver ( self ):... def playerXWon ( self , x: bool ):... def rowNX ( self , n: int , x: bool ):... def colNX ( self , n: int , x: bool ):... This section analyzes various completions from the models we evaluated, displaying both correct and incorrect examples to highlight their strengths and weaknesses.Figure 12 provides an instance of EDITCODER-1.3b generating excess code. This case underscores the importance of the ExcessCode metric ( §5.1), which penalizes models for generating unneeded code. Here, the model, while correctly altering the flatten method in Tensor, unnecessarily adds an unflatten method, resulting in an ExcessCode score of 66.67.To accurately measure ExcessCode, completions must pass the test suite for proper line coverage evaluation. Figure 13 shows another excess code example where the completion is failing the test suite, preventing the evaluation of the ExcessCode metric.Different models exhibit unique competencies in completing different tasks. Figure 14 highlights a solution by EDITCODER-33b for a complex problem involving the modification of a maze solver class to use the A* algorithm with Manhattan distance heuristic instead of the existing Uniform Cost Search (UCS) algorithm. A common error observed, particularly in GPT-4, is the incorrect persistence of heuristic values in the A* algorithm, where the heuristic value persists across multiple iterations of the algorithm inside the frontier queue. In contrast, Figure 15 presents a problem solved exclusively by OpenAI models, requiring domain-specific knowledge on monte carlo approximation for kl-divergence, a skill less common in smaller open-source models. This illustrates the balance between using OpenAI models for domain-specific issues and open-source models for more general code editing tasks. 13 : An example of excess added code to the solution, which is incorrect -produced by the EDITCODER-33b. The model adds a "doc len" instance attribute which is not necessary for the solution and is not implemented in the proposed solution shown above. In addition, it is not necessary to store the document lengths as it is a quick lookup to get the length of a list in Python.searching for markdown-formatted code blocks in the conversations, therefore the actual number of code-related requests is likely higher. We analyzed a subset of code editing requests to understand the types of requests humans make to LLMs. We find that almost all of the requests are of the "lazy" kind that we include in CANITEDIT. We provide two examples of human editing requests in Figure 16 . The first example is a request to refactor a Python code snippet, and the second example is a request to refactor a JavaScript code snippet. As shown, these requests are very informal and direct, and do not provide any information about the desired solution. Other instructions we found that we think exemplify the type of instructions humans give to LLMs include:• Please change use scrappy instead request.• change this code to python• Can you change above code to not use histogram but use two for loops to create the histogram?• Very cool. Now change it so that it compresses each file using lz4 and saves it to a file with the same name and extension, + ".lz4" E.2 Model-Generated Instructions for Editing Code def putMarbles ( weights : List [ int ] , k : int ) -> int :""" You have k bags . You are given a 0 -indexed integer array weights where weights [ i ] is the weight of the ith marble . You are also given the integer k . Divide the marbles into the k bags according to the following rules : No bag is empty .If the ith marble and jth marble are in a bag , then all marbles with an index between the ith and jth indices should also be in that same bag . If a bag consists of all the marbles with an index from i to j inclusively , then the cost of the bag is weights [ i ] + weights [ j ] . The score after distributing the marbles is the sum of the costs of all the k bags .Return the difference between the maximum and minimum scores among marble distributions . """ # code omitted for brevity # PASSING TESTS # none of the tests are passing # FAILING TESTS assert putMarbles ([1 , 3 , 5 , 1] , 2) == 4 # actual output : 6 assert putMarbles ([1 , 3] , 2) == 0 # actual output : inf (a) Instruction: The implementation failed 2 out of the test cases provided. The issue lies in the calculation of the difference between the maximum and minimum scores among marble distributions. The function returns the difference between the score for k bags and the score for 1 bag, which is not the correct calculation for the problem statement. To fix this issue, we need to find the maximum and minimum scores among all possible distributions and then return the difference between them. This can be achieved by iterating through all possible distributions and keeping track of the maximum and minimum scores, and then returning their difference. Figure 17 : An example of a model-generated instruction for code editing. The instruction is generated by GPT-4 using the Reflexion algorithm (Shinn et al., 2023) , by making the model reflect on unit test failures. The problem is from the LeetCode Hard problem set. This section provides an example of code editing guided by instructions generated by GPT-4 using the Reflexion algorithm. Reflexion is a versatile algorithm developed for enhancing model output through environmental feedback, as detailed in Shinn et al. (2023) . While its application extends across various tasks, including reasoning and decision-making, its utility in program synthesis is particularly notable. The process starts with generating unit tests for a program given its natural language description, followed by the creation and evaluation of a candidate program against these tests. If the program fails, Reflexion induces the model to produce a reflection, identifying potential errors and suggesting corrections.These values are calculated by averaging over the results from the models in this table, and not the entire set of models in the main evaluation.Deduplication, achieved by concatenating the 'before' and 'after' code segments, helps mitigate overfitting to specific training examples(Lee et al., 2022).We chose EditPackFT for this experiment as it is the smallest dataset we use for training, allowing us to quickly compare the two methods. 


Decision: This work introduces a code-editing benchmark and conducts evaluation for many open-sourced and commercial code models.
Finetuning experiments are also presented in the paper, showing that they are helpful in improving performance across the board.
While the size (Reviewer b7CL) and language coverage (Reviewers R93Z and Xyri) of the introduced benchmark are limited, the paper presents a first-step effort toward evaluating LLM code editing. The paper is well-written, and the results are clearly presented.

Reviewers NZ9v and R93Z have suggested adding human evaluation. I agree with them that either comparing human and model behaviors or having human feedback would be an interesting direction that lies in the intersection among NLP, program synthesis, psychology, and HCI; however, since the task itself is well-defined and can be evaluated by comparing execution results, I don't think this paper needs to compare the model behaviors with humans for acceptance.

I agree with Reviewer Xyri that more details about the dataset curation would be helpful. Please incorporate your response into the next version of the paper.

[comments form the PCs] It's important to follow the AC's recommendation regarding dataset curation details.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: This paper proposes a benchmark to evaluate the instruction-following code editing capability of selected baselines. 

- The authors include several kinds of editing instructions in the design.
- The authors provide an instruction-tuning training dataset bundle specifically for code editing enhancement.
strength_weakness: strength_weakness: - **Quantified correlation with the code generation capability.**  Although this paper focuses on the code editing task, the authors should also include explore the correlation between the general code generation and editing performances of the baselines in the main text, given that there is assumption that the former could be the basis of the latter.
- **Lacking of human evaluation.** It would be better if the authors could provide the performance of human annotators in the benchmark as a guideline for the (half) open-ended problems.
questions: questions: "These suites were designed to rigorously evaluate whether the ’after’ segment met the problem requirements while ensuring the ’before’ code did not." -- if all the "before" segments do not met the requirements, do they still fall into any of the three task categories?

Can the authors provide clarification on the "relatively small" and "quite low" from "The mean Levenshtein distance between the ‘before’ and ‘after’ code segments is 197.1 characters, indicating that the changes are relatively small. We also analyze the distribution of the commit message lengths, and find that the mean token count is 10.1, which is quite low."?

Is there any overlap between the two training sets?
paper_summary: paper_summary: The paper proposes a novel benchmark called CANITEDIT, which is designed to evaluate the instructional code editing skills of Code LLMs. It comprises 105 manual code editing tasks, each accompanied by two sets of natural language instructions and a comprehensive test suite for validation. Additionally, it presents  a specially tailored training dataset for code editing. Through extensive experiments, the work highlights a significant performance contrast between closed and open models among state-of-the-art Code LLMs. Moreover, the presented training dataset has been shown to be effective in improving code editting capabilities of models at various sizes.
strength_weakness: strength_weakness: * A potential concern arises from the limited scope of the benchmark, which encompasses only 105 manually crafted instructional code editing problems. This constraint could result in a restricted coverage of domains within the evaluation.
questions: questions: See the comments above.
paper_summary: paper_summary: This paper proposes a new benchmark to evaluate the instruction following ability of Large Language Models (LLMs) in code editing tasks. The evaluation exposes a significant gap between the capabilities of state-of-the-art open and closed models. For example, even GPT-3.5-Turbo is better than the best open-sourced model at code editing tasks. This paper also collects a new dataset named CANITEDIT comprising 105 meticulously constructed Python code editing challenges.
strength_weakness: strength_weakness: 1. The evaluation metric is limited as there are only pass@k and ExcessCode in metrics. Authors might propose some refined assessment techniques for code editing tasks or add human evaluation.
2. Similar contributions with Guo et al.[1]. Authors might consider to compare with this work if possible.
3. The code languages types are limited since there is only Python language.


[1] Guo et al., CodeEditorBench: Evaluating Code Editing Capability of Large Language Models.
paper_summary: paper_summary: This paper proposes CanItEdit, a hand-curated evaluation benchmark consisting of 105 Python code editing problems, spanning three different categories: corrective edits, adaptive edits, and perfective edits. Each problem comes with two different types of natural language instructions: descriptive (entailing specific details for more explicit guidance) and lazy (minimal instructions that more closely resemble typical user queries). They further construct a training set for this task by filtering the CommitPackFT dataset and also mining additional examples from GitHub commits, where the instructions correspond to commit messages. They demonstrate out-of-the-box, open-source models (CodeLlama, Mixtral, DeepSeekCoder, StarCoder, OctoCoder) fall behind closed models (GPT-3.5-Turbo, GPT-4) by significant margins on this benchmark. By fine-tuning DeepSeekCoder-Base on their new training set, they observe improvements which appear to reduce the gap to a certain extent.
strength_weakness: strength_weakness: - Very minimal details about the dataset creation procedure are provided. (Please see questions below). This makes it very difficult to understand how the dataset was constructed and consequently assess its quality. 
- The dataset is Python-only and does not allow evaluating across different programming languages.

Missing references:
- Automating Code Review Activities by Large-Scale Pre-training (https://arxiv.org/abs/2203.09095)
- CoditT5: Pretraining for Source Code and Natural Language Editing (https://arxiv.org/abs/2208.05446)
- Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM (https://arxiv.org/abs/2403.19114)
- CodeEditorBench: Evaluating Code Editing Capability of Large Language Models (https://arxiv.org/abs/2404.03543). (Note: This can be considered contemporary work and I have just included it for reference.)
questions: questions: - Please provide more details about the dataset creation procedure. For instance, how were the annotators recruited? What were the exact instructions provided to them? It is not clear how the annotators come up with problems. Are they from scratch or is a starting point provided to them? If a starting point is provided, how is it derived? Do these annotators write both the descriptive and lazy instructions. Please explain in more detail the following claim: "Upon completion of a problem, the lead generated sample completions to ensure that the failures and successes were reasonable and consistent with the problem's intent." 
- A common use case is updating documentation (e.g., comments, README); however, it is not clear how such documentation edits can be evaluated with test cases. Does this evaluation benchmark include such edits?
- The following claim is made in the paper: "GPT-4, despite not being specifically trained on code-related tasks, surpasses…" Could you explain how you came to this conclusion? It does seem that GPT-4 was trained on a significant amount of code, including come commits data.
- It's not clear why Table 4 was not presented like Table 3, with results for "Descriptive" and "Lazy" shown separately. Do the ablations perform differently for these different use cases?
- Despite motivating the different categories of edits, the main paper does not dive into this at all. I see that some additional results are provided in Table 6 of the Appendix, but this does not include all models in Table 3. Could you share details on whether the same trends are there across these categories for all the models presented in Table 3?

Comments: title: Response to rebuttal
comment: Thanks for your detailed response. I will be keeping my score the same.