InstructCoder: Instruction Tuning Large Language Models for Code Editing (2024)

Abstract: Code editing encompasses a variety of pragmatic tasks that developers deal with daily. Despite its relevance and practical usefulness, automatic code editing remains an underexplored area in the evolution of deep learning models, partly due to data scarcity. In this work, we explore the use of Large Language Models (LLMs) to edit code based on user instructions. Evaluated on a novel human-written execution-based benchmark dubbed EditEval, we found current models often struggle to fulfill the instructions. In light of this, we contribute InstructCoder, the first instructiontuning dataset designed to adapt LLMs for general-purpose code editing, containing highdiversity code-editing tasks such as comment insertion, code optimization, and code refactoring. It consists of over 114,000 instructioninput-output triplets and covers multiple distinct code editing scenarios. The collection process starts with filtered commit data sourced from GitHub Python repositories as seeds. Subsequently, the dataset is systematically expanded through an iterative process, where both seed and generated tasks are used to prompt ChatGPT for more data. Our findings reveal that open-source LLMs fine-tuned on InstructCoder can significantly enhance the accuracy of code edits, exhibiting superior codeediting performance matching advanced proprietary LLMs. The dataset and the source code are available at https://github.com/qishenghu/ CodeInstruct.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 50–70 August 11-16, 2024 ©2024 Association for Computational Linguistics
The dataset and the source code are available at https://github.com/qishenghu/ CodeInstruct.

heading: 1 Introduction
text: Developers typically engage in a cyclic routine of writing and revising code. As a crucial element,
* Equal contribution. Ordering is determined by dice rolling.
† Equal advising. Ordering is determined by dice rolling.
code editing takes up a great portion of this process, encapsulating diverse sub-tasks such as code optimization, refactoring, and bug fixing, each posing distinct challenges. Automated code editing tools could substantially boost developer productivity by alleviating the burden of monotonous tasks. However, it remains an under-explored area, partly due to the lack of relevant data, hampering substantial progress by deep learning models.
Inspired by the recent advancements in LLMs (Brown et al., 2020; Chowdhery et al., 2022; Ouyang et al., 2022; OpenAI, 2022; Touvron et al., 2023a; OpenAI, 2023) and Code LLMs (Nijkamp et al., 2023a; Chen et al., 2021a; Li et al., 2023a), we explore the proficiency of LLMs in code editing tasks based on user instructions, for instance, “add a docstring to the function for clarity”, “remove redundant code”, or “refactor it into reusable functions”. These tasks are distinctly different from code completion, which involves generating code to complete given code snippets or comments. Code editing requires the model to not only understand the existing code but also execute modifications that are in line with the given instructions, while seamlessly integrating with the context. For example, removing redundant code or refactoring a function should not affect the return value.
To systematically evaluate LLMs for code editing, we created a novel benchmark named EditEval. It contains various types of code edits adapted from Github commits and existing datasets. Intriguingly, we found that open-source models yield unsatisfactory results, and even the most advanced proprietary LLMs struggle to solve these tasks.
In addressing this challenge, we present InstructCoder, a diverse dataset for instruction finetuning, particularly designed to improve the code editing
50
abilities of LLMs. Specifically, we first collect and manually scrutinize commit data from public repositories on GitHub as the seed code editing tasks. Then, we utilize the seed data to prompt ChatGPT (OpenAI, 2022) to generate new instructions and input-output pairs respectively. This process resembles the Self-Instruct (Wang et al., 2022a) and Alpaca (Taori et al., 2023) frameworks. By innovatively forcing scenarios to guide the generation process, our approach ensures that the tasks in InstructCoder are diverse and relevant to realworld programming situations, resulting in a robust dataset for instruction finetuning in the code editing domain. After proper deduplication and postprocessing, we retain over 114,000 samples in the dataset.
Our empirical studies reveal that LLMs display notable gains in code editing abilities after finetuning with InstructCoder. Code LLaMA achieves the best results through fine-tuning, attaining an accuracy of 57.22%, closely matching ChatGPT. Further studies also signify that while the pre-training of the models is fundamental, the code editing performance is highly influenced by the quality and volume of the instruction-tuning data.
In summary, the contributions of this work are (1) InstructCoder, the first instruction-tuning dataset featuring a wide range of diverse code editing tasks, and demonstrate the effectiveness of instruction-finetuning with InstructCoder; (2) EditEval, a novel human-written execution-based benchmark for the rigorous evaluation of generalpurpose code editing; (3) We find that opensource models instruction-tuned with InstructCoder can demonstrate strong code editing performance matching ChatGPT.

heading: 2 Related Work

heading: 2.1 Instruction Finetuning Datasets
text: Previous studies have concluded that instruction finetuning LLMs on a diverse collection of instructional tasks can further improve the ability of LLMs to generalize well on unseen tasks (Ouyang et al., 2022; Mishra et al., 2022; Wei et al., 2022; Chung et al., 2022; Wang et al., 2023c). To support these tasks, datasets consisting of a large number of code snippets with corresponding annotations are necessary. These instruction can be reformulated from existing datasets (Aribandi et al., 2022; Wei et al., 2022; Mishra et al., 2022; Longpre et al.,
2
Add new features Fix bugs Improve code readability Error handling General Code Refactor Add logging/monitoring Implement cache mechanisms Add comments/docstring Create unit tests Enhance performance Optimize complexity Implement multithreading Update library dependencies Ensure code security Internationalize code Adhere to coding standards Apply design patterns Reduce code duplication Integrate APIs/services Remove dead code Adapt code for accessibility Modularize code Migrate to new framework Complete incomplete code Address memory leaks Code Linting
0
2k
4k
6k
8k
10k
12k
14k
2k
4k
6k
8k
10k
12k
14k
Number
Category
N um
be r
(a) The top 20 most common root verbs with each top 4 noun objects in the instructions. Instructions with other infrequent root verbs take up 25%.
(b) Wordcloud of scenario domains. Each sector with a different color corresponds to a different scenario domain. Each domain is a cluster of similar scenarios.
Figure 3: Visualizations of InstructCoder data. Best viewed in zoom.
2023), or human-written with crowd-sourcing efforts (Ouyang et al., 2022; Wang et al., 2022b). Machine generation of instruction data has also been explored to reduce human labour (Wang et al., 2022a; Honovich et al., 2022; Taori et al., 2023; Xue et al., 2023). Despite the presence of elevated noise levels within the data, its effectiveness has been identified.

heading: 2.2 Code Synthesis
text: Code generation has been extensively studied (Zhang et al., 2023). Language models pretrained on large collections of code have demonstrated strong abilities in a variety of program-
ming tasks. Some general LLMs gain code generation abilities due to the mixture of code in the pre-training corpus (e.g. The Pile (Gao et al., 2020)), such as GPT-3 (Brown et al., 2020), ChatGPT, GPT-4 (OpenAI, 2023), LLaMA (Touvron et al., 2023a), BLOOM (Scao et al., 2022), GPTNeoX (Black et al., 2022), and Pythia (Biderman et al., 2023). LLMs specifically trained on code and optimized for code generation are also studied, e.g. Codex (Chen et al., 2021a), CodeGen (Nijkamp et al., 2023b), CodeGeeX (Zheng et al., 2023) and StarCoder (Li et al., 2023a). These models all adopt the decoder-only transformer architecture but differ in size and specific model design
3
(e.g. positional embedding, norm layer placement) as well as the selection and preprocessing of the pre-training corpus. The study of Code Synthesis has led to exciting applications (Li et al., 2024; Xiao et al., 2024).
On the other hand, relatively little literature addresses the objective of code editing. Previous works focus on a subset of code editing tasks, such as code infilling (Fried et al., 2023) and debugging (Just et al., 2014; Tarlow et al., 2020; Ding et al., 2020; Jimenez et al., 2023). The PIE (Madaan et al., 2023) dataset is a concurrent work most relevant to ours, which focuses on speeding up programs. Other works (Yin et al., 2018; Wei et al., 2023; Chakraborty et al., 2020) can not accept natural language as edit intentions, rendering them less user-friendly.
Nevertheless, datasets particularly tailored for general-purpose code editing are absent. To fill this gap, we introduce InstructCoder, a novel dataset aimed at further advancing the capabilities of code editing with LLMs.

heading: 3 EditEval: Evaluating Code Editing Models
text: As aforementioned, code editing is significantly different from code completion. Consequently, widely utilized datasets in the realm of code completion, such as MBPP (Austin et al., 2021) and HumanEval (Chen et al., 2021b), fall short in evaluating code editing capabilities. To rigorously evaluate the code editing capabilities, we curated a test set of 194 code editing tasks, derived from three key sources: GitHub commit data, MBPP, and HumanEval. We harness the input code from these sources and create plausible edit instructions. For GitHub sources, we manually create execution contexts so that the code is runnable. Each sample is accompanied by a canonical solution written by
humans to ensure the instruction is viable. The generated code edits are strictly assessed using automated test cases to evaluate the correctness of the edits. An edit is considered correct only if it passes all the test cases. This automated method provides a robust and objective evaluation framework, essential for benchmarking the model’s performance in diverse code editing situations. Appendix A showcases an example of the test set.
We benchmarked several instruction-tuned models on EditEval, and the results are listed in Table 1. Generally, the results reveal significant potential for improvement in code editing. Alpaca and CodeAlpaca exhibit accuracies below 20% with 7B and 13B sizes, and it only gets better at 33B. At this size, CodeAlpaca beats Alpaca, achieving 35.56% accuracy. Turning to the GPTs, the most advanced proprietary models up to this point, GPT-4 achieves the best performance at 68.56%. Even ChatGPT struggles at this task, scoring only 57.73%. Upon closer examination, we found the challenge of EditEval lies in the high demand for both instruction following and code understanding. The model has to have a grasp of the implicated context of the input code, and then accomplish the edit within its context.

heading: 4 InstructCoder: Instruction-tuning Empowers Code Editing
text: In this section, we introduce how we create InstructCoder to boost the code editing abilities of LLMs via instruction finetuning. We employed a method based on Self-Instruct (Wang et al., 2022a), which expands instruction finetuning data by bootstrapping off language model generation. The methodology of generating data with LLMs requires minimal human-labeled data as seed tasks while maintaining the quality and relevance of the tasks in the dataset. Through an iterative process of generating instructions and refining them with deduplication, we create a dataset of a wide range of code-editing tasks. Figure 1 illustrates the data collection pipeline of InstructCoder.

heading: 4.1 Seed Data Collection
text: GitHub is a code hosting platform whose version control service naturally records code edits with commits, which can be converted to instructions. The repositories on GitHub provide diverse data
4
with human-generated quality. However, the data is not suitable for direct utilization1. First, commit messages are mostly brief and resultant, missing detailed descriptions. Furthermore, they can be imprecise or even absent. Second, commits can be huge involving multiple files, which is beyond the scope of this work. In light of this, we direct our attention towards LLMs as a means to generate data, instead of the direct utilization of collected data.
Raw GitHub commit data were collated using BigQuery2. To ensure high quality and address licensing issues, we focused on Python repositories on GitHub with over 100 stars and permissive licenses. Our selection criteria was restricted to commits modifying only one code block within a single Python file. These commits were identified by git-diff3.
During the collection process, we came across many imprecise or emotionally charged commit messages. Codex (Chen et al., 2021a) was employed in such cases to clarify the changes made between versions and improve the commit messages, resulting in more precise and informative instructions. A total of 634 tasks were processed from the commit data through manual efforts and were used for the self-instruct process.
In addition to GitHub commit data, we also leverage high-quality generated samples as additional seed tasks. With manual inspection, a batch of 592 high-quality samples was compiled as additional seed tasks. This set of seed data covers a wide range of code-editing scenarios and enriches the basis on which InstructCoder is created, ensuring that the tasks are rooted in plausible real-world code-editing cases4.

heading: 4.2 Instruction Bootstrapping
text: Self-Instruct (Wang et al., 2022a) is as an effective automated framework for instruction data generation. It works by iterative bootstrapping off LLM’s generation, presenting a way to enrich the
1Initial attempts to utilize real-world GitHub commit data for model fine-tuning yielded suboptimal results. Please refer to Appendix B for a detailed discussion.
2https://cloud.google.com/bigquery 3https://git-scm.com/docs/git-diff 4Incorporating additional seeds also allows for modulating the distribution of generated data, facilitating customization for specific requirements.
instructional dataset while maintaining task quality and relevance from a small set of human-evaluated seed tasks. We leveraged a similar approach to generate diverse code editing instructional data. In each iteration, seven seed task instructions and one ChatGPT-generated task instruction are sampled and combined as a few-shot context to prompt ChatGPT for more instructions. To generate more diverse and practically applicable instructions, we also generated tasks across multiple sub-domains by specifying the editing intent in the prompt provided. Relevant prompts used can be found in Table 4 in Appendix C.

heading: 4.3 Scenario-conditional Generation
text: We originally found many generated samples share similar codebases despite different instructions and few-shot examples provided. Such similarity could largely diminish the dataset’s value. Empirical analysis suggests the issue could be attributed to LLM generating general codebases for input/output snippets when insufficient context is provided. As a countermeasure, we propose to introduce code editing scenarios for input/output code generation. We present some examples in Figure 9,10,11 in Appendix D, where we generally observe that instances generated with scenario demonstrate higher quality in terms of richer context and code structure compared to those without.
For each generated instruction, we first prompted ChatGPT to generate practical events as “real-world” scenarios where the editing instruction could be performed, and randomly select one for instance generation in the next step. Subsequently, the LLM was instructed to generate samples that correspond with the instruction and scenario, ensuring the codebases and variable names are appropriate. The prompt used can be found in Table 4 in Appendix C.
By incorporating scenario-conditional generation, the resulting samples exhibit increased variability regarding codebases and variable naming, thus augmenting the diversity of InstructCoder.

heading: 4.4 Postprocessing
text: Following Self-Instruct (Wang et al., 2022a), deduplication was applied on the generated instructions to remove instructions that have a ROUGE-L (Lin, 2004) overlap score larger than 0.7 with the ex-
5
isting instructions. For the code, we employed MinHash with Locality Sensitive Hashing (LSH) indexing to remove instances with a Jaccard similarity greater than 0.75. Ultimately, InstructCoder comprises over 114,000 distinct code editing tasks. For experimental purposes, we designated 95% of the tasks for training, while the remaining 5% formed our validation set.

heading: 5 Data Analysis
text: We analyze InstructCoder in terms of 1) diversity, 2) complexity, and 3) correctness. We provide distribution and complexity analyses of the task instances. Finally, we demonstrate through human investigation that our data is highly reliable.

heading: 5.1 Statistic Overview
text: InstructCoder comprises over 114k code editing instructions, each paired with an input/output instance. The token length distribution of input/output can be viewed in Figure 4 and Table 5 in Appendix E. Most of the data falls within a reasonable range in terms of length, while some extreme values reflect the breadth of our dataset.

heading: 5.2 Instruction Diversity
text: To explore the diversity of tasks in InstructCoder and their practical applicability, we present various instruction intents i.e. what the code edits intend to accomplish, and instruction verbs, i.e. how the code edit is accomplished.
Instruction Intents. We asked ChatGPT to classify the types of code edits in our dataset and manually identified 27 empirical genres. Figure 2 shows the distribution of the code edit intent categories
in InstructCoder, which include adding functionalities, optimizing code, improving readability, etc. These objectives underscore the extensive range of InstructCoder.
Instruction Verbs. The diversity of instruction verbs is also portrayed in Figure 3a. We demonstrate the top 20 root verbs and their top 4 direct nouns both ranked by frequency. While a great portion of the instructions can be roughly clustered as creation (e.g. “add”, “implement”, “creat”) and modification (e.g. “modify”, “replace”, “change”), InstructCoder presents a long-tail distribution with less common verbs other than the top-20 taking up 25.0% percentage. This demonstrates that the dataset contains a wide spectrum of instructions.

heading: 5.3 Scenario Diversity
text: InstructCoder is designed to cover a wide range of scenarios. As discussed in Section 4.3, each instruction was accompanied by different scenarios where the editing instruction could be performed to improve diversity. A word cloud is provided to show some of the scenario domains in our dataset, as illustrated in Figure 3b, with each sector referring to a different domain. The diversity of the dataset is emphasized by the presence of a wide range of domains such as image processing, web development, and cybersecurity.

heading: 5.4 Complexity
text: We reflect the complexity of a code edit task using the number of differing lines and their edit ratio in the input/output pair, which are defined as:
ndiff = |I ∪O \ I ∩O |, (1)
rdiff = ndiff
|I ∪O | , (2)
where I and O are sets of input/output code with single lines as elements.
We measure the differing lines of a code-editing task instance using the Python library difflib.5 We found that the average number of differing lines in InstructCoder is 11.9 and the average edit ratio is 0.52. These values suggest a fairly acceptable level of complexity, indicating that the dataset is neither
5https://docs.python.org/3/library/difflib. html
6
too easy nor too hard. InstructCoder strikes a balance in terms of complexity, making it well-suited for finetuning and evaluating LLMs in a wide range of code editing tasks. Figure 12 in Appendix E illustrates the distribution of the number of differing lines.

heading: 5.5 Correctness
text: We further randomly sampled 200 instances and invite annotators to evaluate the instances based on two criteria: the validity of the instructions and the correctness of the outputs. The validity assessment focused on determining if the instructions exhibit clear and appropriate editing intents. The correctness evaluation examines if the input-output pairs reflect the changes specified by the instructions.
The results in Table 2 indicate that most instructions in the InstructCoder dataset are valid. A few instances exhibited noise and occasional failure to follow the instructions, but high correctness was found overall. Out of the 200 evaluated instances, 180 were successfully solved, showcasing the overall quality and reliability of InstructCoder.

heading: 6 Experiments

heading: 6.1 Setup
text: Training. We experiment with two families of open-source language models with various sizes: LLaMA (LLaMA, LLaMA-2 and Code LLaMA) (Touvron et al., 2023a,b; Roziere et al., 2023) and BLOOM (Scao et al., 2022).
LLaMA is a series of LLMs with parameters ranging from 7 to 65 billion. They have been pretrained on a vast corpus, of which approximately 4.5% comprises code. The LLaMA-2 series extends the family with more intensive pre-training. Additionally, Code LLaMAs are built on LLaMA-2 and specifically trained on 500B tokens of code to enhance its code understanding and generation capabilities. BLOOM is a multilingual LLM capable of generating human-like outputs in 46 languages
and 13 programming languages. A full finetuning updating all the parameters in an LLM can be computationally expensive. Instead, we adopt LoRA (Hu et al., 2022), a parameter-efficient finetuning method that optimizes an approximated low-rank delta matrix of the fully-connected layers. In this way we could fine-tune a 33B model in a single A100-80GB GPU card. In our experiments, LoRA is applied to the query, key, value, and output transform weights of the Transformer architecture (Vaswani et al., 2017). All hyperparameters can be found in Table 6 in Appendix F.
Baselines. We select ChatGPT (OpenAI, 2022), GPT-4 (OpenAI, 2023) and GPT-4 Turbo as strong baselines. The aforementioned open-source models along with an instruction-tuned LLaMA model called Alpaca (Taori et al., 2023) are included, and their zero-shot performance is reported.
Concurrent to our work, CodeAlpaca6 is a popular dataset generated with the pipeline of Alpaca, differing in that its seed data is replaced by handwritten easy instructions with short programs. We fine-tune LLaMA models with CodeAlpaca and Alpaca and compare the results.

heading: 7 Results

heading: 7.1 Finetuning Efficacy with InstructCoder
text: In this section, we demonstrate the value of our InstructCoder dataset. Table 3 presents a detailed comparison of EditEval performance across models fine-tuned with InstructCoder and baseline models. While very low accuracies are observed in
6https://github.com/sahil280114/codealpaca
7
open-source plain models, finetuning with InstructCoder significantly boost the accuracy, highlighting the effectiveness of efficient instruction finetuning with machine-generated code edit pairs.
Code LLaMA 13B matches ChatGPT’s performance and surpasses other open-source models with a 57.22% accuracy rate. The more substantial LLaMA-33B model shows a notable 35.56% improvement, yet it falls behind Code LLaMA-7B, which benefits from extensive pre-training on code. For qualitative results, see Appendix G.
As expected, the pre-training foundation of LLM significantly influences code-editing efficacy. LLaMA demonstrated higher accuracies than BLOOM models of similar sizes. Among LLaMAs, those pre-trained on more tokens (LLaMA-2 series) outperformed earlier versions. Furthermore, Code LLaMAs exceed LLaMA-2 models as a result of their extensive pre-training specifically on coding data. Despite the varying capabilities of the foundational models, our dataset consistently enhances performance.

heading: 7.2 Dataset Scaling
text: InstructCoder has a scale considerably smaller than what LLMs are typically pre-trained on. To ascertain the sufficiency of this scale, we conducted an experiment wherein we fine-tuned the LLaMA models using varying proportions (1%, 10%, and 100%) of the dataset. The smaller subsets are guaranteed to be encompassed within the larger subsets. The results are shown in Figure 5. The identified trend demonstrates a positive correlation between the model’s accuracy and the scale of the training set.
Fine-tuned with merely 1% of the data, the mod-
els experience a limited number of parameter updates but quickly adapt to the tasks, surpassing their respective zero-shot accuracy scores by significant margins. This underscores the significance of instruction tuning. As the volume of training data increases, we observe consistent improvements in model accuracy, approximately growing linearly with respect to the logarithmic scale of the number of samples. Crucially, our experiment empirically suggests that larger models are more effective with a constrained training compute budget.

heading: 7.3 Edit Ratio
text: Figure 6 depicts the accuracy of fine-tuned LLaMA models as evaluated by GPT-4 across five edit ratio levels, using 2000 random samples from the validation set. This evaluation, justified in Appendix H, involves prompting GPT-4 for a quick and general assessment of code edits, offering an alternative perspective to code edit evaluation. In this assessment, larger models consistently outperform their smaller counterparts. Notably, accuracy decreases with lower edit ratios, potentially due to the models adopting the shortcut of copying inputs to minimize loss in scenarios requiring fewer edits. This trend, however, is less pronounced in larger models, which show a greater ability to discern subtle differences in cases of low edit ratios.

heading: 8 Conclusion
text: We introduce InstructCoder, the first instructiontuning dataset for general-purpose code-editing tasks. It comprises generations of LLMs, where real GitHub commits serve as seed tasks to guide the generation process. A scenario-conditional approach is introduced to ensure both diversity
8
and high quality of the data. Our experiments on the novel EditEval benchmark show that opensource models can gain huge improvements and even yield performance matching proprietary models through computationally lightweight parameterefficient fine-tuning with InstructCoder. We also reveal that the LLM base model and the scale of fine-tuning data are both profound factors of codeediting ability. We hope the dataset can benefit and inspire more research in this area towards building more powerful coding models.
Limitations
Our approach did not encompass code changes involving multi-file contexts, which might be useful in development. We hope to explore these aspects further and incorporate additional programming languages in our future research.

heading: A An Example Test of EditEval
text: An example of EditEval the test set is showcased below. To accomplish the task, the model must not only adhere to the user’s instructions but also comprehend the input code in the context provided.
12

heading: B Comparing Machine-Generated Data and Real-World Data
text: Given the substantial repository of code and commit data available on GitHub, a natural idea is to utilize these real-world data to fine-tune a model to perform code editing. However, as discussed in Section 4.1, these data from GitHub can be extremely noisy, especially in the commit messages, rendering them a sub-optimal choice for instruction-tuning. On the other hand, machine-generated data is increasingly recognized for its utility, as evidenced by various studies that achieves enhanced results with this type of data (Gunasekar et al., 2023; Li et al., 2023b; Wang et al., 2023b). This approach provides better controllability over the distribution of the generated contents and facilitates the collection of diverse data, including those under-represented or difficult to mine and clean from real-world data.
The experiment results in Figure 8 corroborate the usage of machine-generated data. We further collected GitHub commits matching the size of InstructCoder, and used the same hyperparameters for instruction fine-tuning. As can be seen in the results, InstructCoder significantly outperformed raw GitHub commits, and the lead is more profound with a larger base model, demonstrating the effectiveness of InstructCoder. On the other hand, fine-tuning with GitHub commits yields poor results, and is the worst among all three data sources on LLaMA-1 13B. The observation suggests that using machine-generated data for instruction fine-tuning is superior in terms of training code editing models.
13

heading: C Prompts
text: The prompts used in our data collection and experiments are listed in Table 4.
14

heading: D Qualitative Examples of Scenario-Conditional Generation
text: Three comparisons are presented, each showing instances that were generated with or without the inclusion of a scenario.
15
16

heading: E Additional statistics of InstructCoder
text: 17

heading: F Hyperparameters
text: The hyperparameters used in all finetuning experiments are listed in Table 6. For all inferences, we utilize greedy decoding. For OpenAI’s GPTs, we achieve this by setting its temperature to 0.

heading: G Qualitative Examples Generated by Finetuned LLaMA-33B
text: We demonstrate some qualitative example responses generated by finetuned LLaMA-33B.
18

heading: H Alignment of GPT-4 Evaluation and Human Evaluation
text: Due to the extremely demanding nature of creating automated tests, we seek to investigate the viability of using GPT-4 as an automatic evaluator to lessen the extensive human effort involved. Using LLMs as generation evaluators has been demonstrated effective in NLG tasks (Liu et al., 2023; Wang et al., 2023a; Fu et al., 2023), and especially in code generation (Zhuo, 2023). To further validate this idea, we collected an additional 134 commits data for testing purposes and processed them in the same manner as the seed tasks. Both GPT-4 evaluation and human evaluation are conducted on this dataset to assess their alignment.
Human evaluation. Each sample is annotated by three examiners, and the average accuracy is recorded. We developed an annotation tool to ensure the impartiality of evaluation (see Figure 14 for the user interface). Generations of different models are shuffled and the anonymity of the models is guaranteed. The edit is annotated as correct if it correctly reflects the instruction demands and wrong if it fails to follow the instruction.
GPT-4 evaluation. We ask GPT-4 to evaluate if the code edit is an acceptable response to the input and collect the correct rate. The prompts for GPT-4 evaluation can be found in C.
Results. We carry out the experiments on the code edits generated by ChatGPT and LLaMA of three sizes fine-tuned with InstructCoder. While we found that the human annotators are always slightly stricter than the GPT-4 evaluator, the overall Cohen’s Kappa value of the GPT-4 evaluations and human evaluations reaches 0.665, which is substantial according to Cohen (1960). This renders GPT-4 evaluation as a convenient and effective method for evaluating the correctness of code edit tasks.
19
20

heading: I Data Filtering Process
text: The detailed process of filtering the dataset is listed below:
• We selected GitHub repos with over 100 stars to ensure the overall quality. We only utilized repos with permissive licenses (MIT, Apache-2.0, GPL-3.0, GPL-2.0, BSD-2.0, BSD-3.0, LGPL-2.1, LGPL-3.0, AGPL-3.0).
• We kept commits in which only one single .py file was changed. Using git-diff, we identified and preserved commits where only one code block was changed.
• We discarded commits with single-word or empty commit messages.
• We removed commits with over 100 edited rows.
Manual:
• We discarded rare commits containing inappropriate language.
• We discarded commits where the change in the source code does not match the commit message.
• We filtered out project-specific adjustments that lack sufficient context.
• We utilized Codex (Chen et al., 2021a) to rewrite ambiguous commit messages, enhancing the clarity of the intended code edits.
21

heading: Future Work
text: We hope to explore these aspects further and incorporate additional programming languages in our future research