WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuning (2024)

Abstract: Recent work demonstrates that, after instruction tuning, Code Large Language Models (Code LLMs) can obtain impressive capabilities to address a wide range of code-related tasks. However, current instruction tuning methods for Code LLMs mainly focus on the traditional code generation task, resulting in poor performance in complex multi-task scenarios. In this paper, we concentrate on multiple code-related tasks and present WaveCoder, a series of Code LLMs trained with Widespread And Versatile Enhanced instruction data. To enable the models to tackle complex coderelated tasks, we propose a method to stably generate diverse, high-quality instruction data from open source code dataset in multitask scenarios and obtain CodeSeaXDataset, a dataset comprising 19,915 instruction instances across 4 code-related tasks, which is aimed at improving the generalization ability of Code LLM. Our experiments demonstrate that WaveCoder models significantly outperform other open-source models in terms of the generalization ability across different code-related tasks. Moreover, WaveCoder-Ultra-6.7B presents the state-of-the-art generalization abilities on a wide range of code-related tasks.

heading: 1 Introduction
text: Recently, Large Language Models (LLMs) such as ChatGPT, GPT-4 (OpenAI, 2023), and Gemini 1 have attained unprecedented performance levels in a broad array of NLP tasks. These models utilize a self-supervised pre-training process, and subsequent supervised fine-tuning to demonstrate exceptional zero/few-shot capabilities, effectively following human instructions across various tasks.
For code-related tasks, several previous works, including Codex (Chen et al., 2021), StarCoder (Li
∗Work was done during an internship at Microsoft. †Corresponding author.
1https://deepmind.google/technologies/ gemini
et al., 2023a), CodeLLaMa (Roziere et al., 2023) and DeepseekCoder (Guo et al., 2024), have successfully demonstrated that pre-training on code corpus can significantly improve the model’s capability to tackle code-related problems. After the process of pre-training, instruction tuning (Wei et al., 2022; Aribandi et al., 2022; Chung et al., 2022) has shown its effectiveness in the aspect of improving the quality of LLM responses. To specifically enhance the performance of Code LLMs on code-related tasks through instruction tuning, many existing methods for instruction data generation have been designed. For example, Code Alpaca (Chaudhary, 2023) utilizes the method of self-instruct (Wang et al., 2023a) within the coding domain, leveraging the few-shot capabilities of teacher LLM to generate instruction data. Similarly, WizardCoder (Luo et al., 2024) applies the evol-instruct (Xu et al., 2024) approach based on Code Alpaca, demonstrating a novel and effective method for the generation of instruction data. These applications underscore the potential of utilizing teacher LLMs to produce instructional content effectively, thereby offering an avenue for the creation of instruction data in the code domain. However, the quality of the data they generate heavily relies on the performance of the teacher LLM and the limited initial seeds, which often produces a large amount of duplicate instruction instances and reduce the effectiveness of instruction tuning (Xu et al., 2022; Yan et al., 2023; Lee et al., 2022). To break away from dependence on teacher LLMs, Octopack (Muennighoff et al., 2024) constructs a code instruction dataset leveraging the natural structure of Git commits. Nonetheless, ensuring the quality of data in git messages presents a considerable challenge, and the comprehensive screening of data through artificial filtering rules is often a complex task. Additionally, these endeavors are predominantly centered on traditional code generation tasks and lack the capability to produce detailed, task-
5140
specific instructions in multi-task scenarios. In this paper, we primarily focus on multiple code-related tasks, aiming to generate high-quality and diverse instructional data tailored to specific task requirements. Addressing the aforementioned challenges, we refine the instruction data by classifying the instruction instances to four universal code-related tasks in CodeXGLUE (Lu et al., 2021): 1) Code Summarization, 2) Code Generation, 3) Code Translation, 4) Code Repair and propose a widespread and versatile enhanced instruction generation method that could make full use of open source code data and stably generate high quality and diverse instruction data in multitask scenarios. By this generation strategy, we obtain a dataset of 19,915 instruction instances across four code-related tasks, termed CodeSeaXDataset.
To validate our approach, we train StarCoder (Li et al., 2023a), CodeLLaMa (Roziere et al., 2023), and DeepseekCoder (Guo et al., 2024) with our initial CodeSeaXDataset dataset and get WaveCoder. Following a thorough assessment on HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), HumanEvalPack (Muennighoff et al., 2024) benchmarks, experimental results show that our WaveCoder exhibits outstanding generalization ability based on widespread and versatile enhanced instruction tuning. Moreover, to further explore the improvements brought by data quality, we use GPT-4 (OpenAI, 2023) to regenerate response
for the instruction in CodeSeaXDataset. Finetuned with the enhanced 20K CodeSeaXDataset dataset, we obtain WaveCoder-Pro-6.7B which achieve 72.0% pass@1 on HumanEval (Chen et al., 2021) and surpass open source Code LLMs but still behind SoTA Code LLM. Combining enhanced CodeSeaXDataset with WaveCoder-evolcodealpaca, the decontaminated Magicoder-evolcodealpaca 2 dataset, we present WaveCoderUltra-6.7B, with SoTA generalization capabilities on multiple code-related tasks.

heading: 2 CodeSeaXDataset: Four-task
text: Code-related Instruction Data

heading: 2.1 Tasks Details
text: Given the code-related tasks from CodeXGLUE (Lu et al., 2021), we select four of the most universally representative and common tasks from the three generative tasks (code-to-text, text-to-code, and code-to-code) for further exploration including Code Summarization, Code Generation, Code Translation, and Code Repair. Detailed descriptions of these tasks can be found below. Code Summarization (code-to-text). This task aims to create a brief summary of a given code. The raw code is used as input and the teacher model’s response is reformulated into an instruction format.
2https://huggingface.co/datasets/ ise-uiuc/Magicoder-evol-codealpaca-110K
Code Generation (text-to-code, code-to-code). In this task, the model is expected to generate code based on a user’s demand description. Therefore, the teacher model is expected to generate instructions and solution code given the raw code as a instruction-solution pair. The generated solution code is then considered as the output. Code Translation (code-to-code). This task involves converting one programming language into another. The task-specific prompt and raw code are given to the teacher model, then the model generates instructions and the translated code. Code Repair (code-to-code). The aim of this task is to provide correct code based on potential issues in the given code. The teacher model is expected to generate solutions for the incorrect code, typically with the correct code and some descriptions, which are then taken as the output.

heading: 2.2 Widespread and Versatile Enhanced Instruction Generation
text: In past research work (Zhou et al., 2023; Gupta et al., 2023), many researchers have discovered that data quality and diversity often play a more important role in instruction tuning process than data amount. The improvement of data quality and diversity are directly related to the performance of the fine-tuned LLM. Therefore, to ensure the data quality and diversity of instruction instance, we propose a widespread and versatile enhanced instruction generation method including two the following parts: 1) a method that can retain the diversity of instruction data by retainig the diversity of raw code to the utmost extent. 2) a LLM-based
Generator-Discriminator framework to stably generate high-quality instruction data.

heading: 2.2.1 Raw Code Collection
text: To ensure the quality and diversity of raw code, we manually define some filtering rules and utilize a cluster method KCenterGreedy (Sener and Savarese, 2018; Chen et al., 2023) to get the raw code collection from the open source code dataset. In this work, we select CodeSearchNet 3, which contains 2 million of <comment, code> pairs from open-source libraries hosted on GitHub, as our foundation dataset and process it with the following steps: Manually defined filtering rules. In order to select high-quality code for instruction-tuning, we make the following rules to filter the foundation dataset: i) In this work, we filtered the code to make sure that the length of required code is neither too long nor too short. ii) Followed Code Alpaca (Chaudhary, 2023), we have eliminated the raw code containing words from the blacklist, which could potentially reduce the performance of the resulting model. Coreset selection method. To ensure the data diversity when select raw code samples, we employed KCenterGreedy (Sener and Savarese, 2018) algorithm, which has been proven efficient in obtaining a set of core samples of one distribution, to select representative samples from the open source code dataset based on the code embeddings encoded by the same embedding model (roberta-large-v1 (Liu et al., 2019)).
By incorporating such a method into the open source code dataset, the diversity of the generated data no longer relies solely on capability of the teacher LLM itself or initial seed. Moreover, due to the application of the KCenterGreedy algorithm, the diversity of languages is also significantly retained, as shown in Table 2.
3https://huggingface.co/datasets/code_ search_net

heading: 2.2.2 LLM-based Generator-Discriminator Framework
text: After the process of raw code collection, the data diversity from raw code has been retained, where the next step is to generate instruction data for supervised fine-tuning from the raw code. To further ensure the quality of generated instruction data, shown in Figure 2, we propose a LLM-based Generator-Discriminator framework where the generator can leverage an extensive amount of unsupervised open source code to generate supervised instruction data and the discriminator can generate analysis for each component in instruction data. Generation Phase. In the generation phase, we utilize GPT-4 to generate definitions for each coderelated task. As shown in Figure 2, following the model-generated task definition, we manually develop the generation requirements for the each code-related task. Integrating both the task definition and all associated requirements into the generation prompt, we take the raw code as an input and select different examples from the example database to generate instruction data by GPT-3.5. Discrimination Phase. During the exploration of the instruction generation process, we noticed that the data quality of the instruction instances cannot be guaranteed through the generation phase alone. In order to enhance the controllability of data generation and further ensure the data quality, we employ GPT-4 as a LLM-based discriminator to continue analyzing and filtering the instruction data. Subsequently, inspired by Zero-shot-CoT (Kojima et al., 2022), we establish a series of rules, exemplified in Figure 4 and disassemble them to some subtopics to ensure the discriminating accuracy where the LLM-based discriminator can analyze the gener-
ation step by step. By adopting this method, the discrimination rules can be modified partially to address certain issues. After the discrimination process, as shown in Figure 1, each instruction instance is classified as either a good or bad case and the classification information is subsequently random selected in the following generation as examples. For the reusage of these classified instruction instance, different from self-instruct (Wang et al., 2023a) which solely utilize the initial seed task as good example, we exploit both the good generation and bad generation as few-shot example so that the generator can learn from the mistake in different bad example. Therefore, this framework provides a comprehensive approach to generating and evaluating instruction data, ensuring a highquality training dataset.

heading: 3 Experiments

heading: 3.1 Setup
text: Unlike the previous work (Luo et al., 2024; Shen et al., 2023; Gunasekar et al., 2023) that mainly focus on code generation task, we generate about 20K dataset covers 4 common code-related tasks to enhance the geralization abilties of Code LLM. To obtain WaveCoder models, We choose StarCoder15B, CodeLLaMa (7B and 13B), DeepseekCoder6.7B as the base model and fine-tune all the base model for 3 epochs using NVIDIA A100-80GB GPU. For StarCoder-15B, CodeLLaMa-7B and CodeLLaMa-13B, we set the global batch size to 256 using Tensor Parallel and set the initial learning rate at 2e-5. For DeepseekCoder-6.7B, we set the global batch size to 512 using the Fully Sharded Data Parallel (FSDP) module from Pytorch and set the initial learning rate at 5e-5.
Table 3: Results of pass@1 on HumanEval and MBPP benchmark. We use self-reported scores whenever available. The abbreviations "CL", "SC", "DS" refer to the base models CodeLLaMa and StarCoder and DeepseekCoder, respectively. "WaveCoder-Pro-6.7B" and "WaveCoder-Ultra-6.7B" is detailed in the last paragraph of Section 1. Due to the difference in decoding strategies from previous evaluation work, we marked the results of greedy
decoding in blue and n = 200 samples in red . -: Not reported in their paper.
Model Params Base Model InsT Data HumanEval MBPP (500)
Proprietary Models
GPT-4 - - - 85.4 / 67.0 - ChatGPT - - - 73.2 / 48.1 52.2
Open-Source Models
StarCoder 15B - ✘ 33.6 43.3 OctoCoder 15B StarCoder 13K 46.2 43.5 WizardCoder 15B StarCoder 78K 57.3 51.8 WaveCoder-SC-15B 15B StarCoder 20K 50.5 (+16.9) 51.0 (+7.4)
CodeLLaMa 7B - ✘ 33.5 41.4 CodeLLaMa-instruct 7B CodeLLaMa 14K 34.8 44.4 WaveCoder-CL-7B 7B CodeLLaMa 20K 48.1 (+14.6) 47.2 (+5.8)
CodeLLaMa 13B - ✘ 36.0 47.0 CodeLLaMa-instruct 13B CodeLLaMa 14K 42.5 49.4 WaveCoder-CL-13B 13B CodeLLaMa 20K 55.4 (+19.4) 49.6 (+2.6)
DeepseekCoder 6.7B - ✘ 49.4 60.6 Magicoder-DS 6.7B DeepseekCoder 75K 66.5 60.4 WaveCoder-DS-6.7B 6.7B DeepseekCoder 20K 64.0 (+14.6) 62.8 (+2.2) WaveCoder-Pro-6.7B 6.7B DeepseekCoder 20K 72.0 (+22.6) 63.6 (+3.0)
SoTA Open-Source Models
DeepseekCoder-instruct∗ 6.7B DeepseekCoder - 73.8 62.8 Magicoder-S-DS 6.7B DeepseekCoder 185K 76.8 64.6 WaveCoder-Ultra-6.7B 6.7B DeepseekCoder 130K 78.6 (+29.2) 64.4 (+3.8)
Benchmarks and Baselines. To ensure a thorough assessment of the model’s generalization ability, we score our model on three code benchmarks across different code related tasks: HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021) and HumanEvalPack (Muennighoff et al., 2024), as illustrated in Appendix D. Proprietary Models. We present the self-reported results from an array of SoTA LLMs, including ChatGPT (gpt-3.5-turbo), GPT-4. If not reported, we use the results from Octopack (Muennighoff et al., 2024) or evaluate by ourselves. Open Source Models. To ensure an equitable comparison, we opted to select models that have been trained with the similar amount of instruction instances for our comparative analysis. SoTA Open Source Models. We compared WaveCoder-6.7B with the SoTA open source Code LLM, includes Magicoder-S-DS (Wei et al., 2023) and DeepseekCoder-instruct-6.7B (Wei et al., 2023) on a wide range of code-related tasks. All the result of SoTA open source models is presented from EvalPlus. (Liu et al., 2023) If not reported, we evaluate it by ourselves.

heading: 3.2 Result
text: Evaluation on Code Generation Task. HumanEval and MBPP are two representative benchmarks for code generation task, as illustrated in Appendix D. Table 3 shows the pass@1 score of different LLMs on both benchmarks. From the results, We have the following observations: 1) WaveCoder-Pro-6.7B outperforms other open source models with only 6.7B parameters and 20K instruction data. Trained with GPT-4 enhanced CodeSeaXDataset dataset, WaveCoder-Pro-6.7B achieve 72.0% pass@1 and on HumanEval and 63.6% on MBPP, surpassing all open source models but still behind proprietary models and the SoTA open source models. 2) Refined and diverse instruction data can significantly improve the efficiency of instruction tuning. As delineated in Table 3, WaveCoder demonstrates commendable performance, utilizing a dataset comprising merely about 20K Instruction Tuning Data (InsT Data), which positions it on an equal foot-
ing with its contemporaries. Despite a discernible shortfall in the code generation benchmarks relative to WizardCoder (50.5 vs 57.3) and Magicoder (64.0 vs 66.5), it is imperative to consider the substantial disparity in the volume of training data. Moreover, it is observed that WaveCoder-pro-6.7B significantly outperforms Magicoder-DS-6.7B (72.0 vs 66.5), demonstrating the effectiveness of data quality and diversity in instruction tuning.
Evaluation on Other Code-related Task. We score WaveCoder with state-of-the-art Code LLMs on HumanEvalPack (Muennighoff et al., 2024) in Table 4 and Table 5, highlighting the the following salient observations:
1) WaveCoder models outperform all open source models on other code-related task. Building upon Starcoder, our proposed WaveCoder-SC has exhibited exceptional performance, transcending the capabilities of both WizardCoder and OctoCoder as evidenced by the HumanEvalFix (33.0 vs 25.7 vs 27.0) and HumanEvalExplain (30.8 vs 27.5 vs 24.5) benchmarks, which is also shown in other base models. Notably, WaveCoder-DS-6.7B achieves 49.4% average pass@1 score on HumanEvalFix and 41.3% on HumanEvalExplain, surpassing all open source models and demonstrating strong generalization capabilities in multi-task scenarios.
2) The enhancement in data refinement and di-
versification can markedly bolster the efficacy of instruction tuning in multi-task scenarios. Such data refinement, coupled with the categorization of instructions into four code-related tasks, has propelled our models to reach an unforeseen generalization capabilities in various code-related tasks. Remarkably, our WaveCoder-DS-6.7B model outperforms GPT-4 (49.4 vs 47.8) on HumanEvalFix, thereby underscoring the potential of smaller models to achieve near-parity with parameter-heavy models when optimized efficiently. WaveCoder-Ultra-6.7B. Inspired by MagicoderS-DS-6.7B (Wei et al., 2023), we combine CodeSeaXDataset with WaveCoder-evol-codealpaca to a 130K dataset. Fine-tuned with this combination of two datasets, we obtain WaveCoder-Ultra-6.7B. As illustrated in Table 3, 4, 5, WaveCoder-Ultra6.7B has the state-of-the-art generalization abilities on a wide range of code-related tasks, which highlights the significance of our CodeSeaXDataset dataset again and demonstrates the potential of larger datasets.

heading: 4 Ablation and Analysis

heading: 4.1 Ablation of Code-related Tasks
text: To explore the relationship between different tasks, we conduct an ablation study about the task type of instruction data. Using DeepseekCoder-Base-
6.7B as our base model and initial 20K CodeSeaXDataset data as our base dataset, we have the following observations from Table 6: 1) Refined instruction data can significantly improve the generalization ability of pre-trained models without a tradeoff. As shown in Table 6, incorporating all 4 code-related tasks into training data, WaveCoder-DS-6.7B achieves the best performance on benchmark of all tasks. For example, the participation of the Code Repair task yields a considerable average improvement of 33.7% absolute for HumanEvalFix without any significant decline in other tasks, and even improved by 3.1% absolute for HumanEval benchmark. 2) Different tasks can promote each other so that the model can show a generalization ability. From Table 6, we can observe that any combination of three tasks resulted in a lower score than all tasks.
For example, the addition of the code summarization task offers a modest yet significant average improvement on all benchmarks. Moreover, the absence of any task will cause the score of HumanEval to drop, which also reflects the mutual promotion between different tasks.

heading: 4.2 Discussion about Data Leakage
text: In this section, we explore the potential leakage through three instruction datasets about code (i.e. Code Alpaca, CodeSeaXDataset, Magicoder-evolcodealpaca). To ensure an accurate analysis, we employ SoTA embedding model GTE-Large (Li et al., 2023b) to encode the canonical code in test benchmarks and all code in training set. Subsequently, we find the nearest neighbour in train set for each questions in test benchmark. As illustrated in Figure 3, CodeSeaXDataset has the lower
average cosine similarity than other datasets. Figure 6 in Appendix presents two examples about the data leakage in these training set. Moreover, we analyze all benchmarks and notice a serious data leakage issue between HumanEval and Magicoder-evol-codealpaca dataset. Therefore, we decontaminate Magicoder-evol-codealpaca for each evaluation problem in HumanEval and obtain WaveCoder-evol-codealpaca. As illustrated in Figure 3, WaveCoder-evol-codealpaca has lower similarity than Magicoder-evol-codealpaca.

heading: 5 Related Work
text: Instruction Tuning. Recent studies, such as FLAN (Wei et al., 2022), ExT5 (Aribandi et al., 2022), and FLANT5 (Chung et al., 2022), have underscored the efficacy of integrating diverse tasks within training process to bolster the adaptability of pre-trained models for downstream tasks. Specifically, Flan-PaLM 540B’s (Chung et al., 2022) instruction-tuning over 1.8K tasks has demonstrated that a widespread and versatile enhanced
instruction dataset markedly enhances language model performance. InstructGPT (Ouyang et al., 2022), with its incorporation of premium instruction data crafted by human annotators, has shown significant promise in aligning model outputs with user intents, prompting further investigation into instruction-tuning mechanisms. Additionally, Stanford Alpaca (Taori et al., 2023) has innovatively employed GPT-generated instruction data via selfinstruct (Wang et al., 2023a) for instruction tuning process. WizardLM (Xu et al., 2024) has built upon these advancements by applying the evol-instruct methodology, collectively illuminating the transformative impact of instruction tuning on the overall capabilities of LLM. Code Large Language Models. Recent advancements in code generation have been propelled by Code LLMs such as CodeGen (Nijkamp et al., 2022), CodeT5 (Wang et al., 2021), StarCoder (Li et al., 2023a), CodeLLaMa (Roziere et al., 2023) and Deepseek-Coder (Guo et al., 2024), which benefit from extensive pre-training on expansive code corpora. Efforts to further enhance efficiency and problem-solving capabilities have led to the development of instruction-tuned models like InstructCodeT5+ (Wang et al., 2023b), WizardCoder (Luo et al., 2024), Pangu-coder2 (Shen et al., 2023), However, all the instruction data they used is from Code Alpaca, which is not refined enough in the context of multi-task environment, which drives us to propose new methods for instruction data generation. Concurrently, with the release of our contemporaneous work Magicoder (Wei et al., 2023), we offer a concise analysis in Section 3.

heading: 6 Conclusion
text: This paper presents WaveCoder, a Code LLM finetuned with widespread and versatile enhanced instruction data. By enabling language models to effectively tackle complex code-related tasks, our approach demonstrates the potential of integrating multiple code-related tasks into instruction tuning for Code LLM and generating high-quality and diverse instruction data for specific task requirements in multi-task scenarios. WaveCoder achieves stateof-the-art generalization performance on different code-related tasks surpassing existing open source Code LLMs. Furthermore, our analysis of the relationship of different tasks provides valuable insights for future research, paving the way for more extensive code-related tasks and larger dataset.
Limitations
We present WaveCoder and propose a data generation method which can stably generate high-quality and diversity instruction data from open source dataset in multi-task scenario. One limitation of our work is that the training dataset we used only includes 19,915 instructions, which produces limited enhancements to the model. As illustrated Section 3, we expand the training dataset to a larger amount and the resulted model still have significant improvement. Therefore, future work should focus on more code-related task types and larger dataset.
Ethics Statement
We constructed our CodeSeaXDataset dataset from open source code. For each code snippet we used, we are committed to adhering to the terms of its license, which includes proper attribution ensuring that any modifications or derivative works are also shared under the compatible terms. Moreover, we notice a serious data leakage issue in the Magicoder-evol-codealpaca dataset. To ensure a fair comparison, we remove three nearest neighbours of each question in test benchmark from train set. However, if all similar samples are accidentally removed, the integrity of the data will be damaged, which is harmful to model training. Therefore, this phenomenon should be attributed to the fact that the problems in the current test benchmarks are some of the most basic algorithm logic. To this end, we call for more comprehensive and complex test benchmarks for Code LLMs which will not easily cause data leakage problem.

heading: A Prompt
text: Followed Alpaca (Taori et al., 2023), we set the fine-tuning prompt as follows:

heading: B Examples of the LLM-based Generator-Discriminator framework

heading: C Comparison with CodeAlpaca
text: CodeAlpaca dataset contains 20K multi-task instruction-following data generated by the techniques in the self-instruct (Taori et al., 2023). To ensure a fair and multidimensional comparison, we randomly sampled 1K and 5K from both datasets (CodeAlpaca and CodeSeaXDataset), set the same set of training hyper-parameters set (epoch = 3, learning rate = 1e-4, LoRA rank = 8) and used the same training prompts. To prevent overfitting, we use Low-Rank Adaption (LoRA) (Hu et al., 2022) for fine-tuning if the size of instruction-follow training dataset is less than 5K and perform full fine-tuning on whole 20K dataset.
1) After being fine-tuned with 1K, 5K and 20K of instructional data respectively, the performance of base model improves significantly on HumanEval shown in Figure 5. Taking Starcoder as the base
model, CodeSeaXDataset surpasses the CodeAlpaca (44.9% vs 41.7%, 45.7% vs 48.1% and 47.0% vs 50.5%) shown in Figure 5 (a), which emphasizes the effectiveness of our method on refining instruction data. As shown in Figure 5 (b), The results of different base models on CodeSeaXDataset surpasses the results on CodeAlpaca, which emphasizes the effectiveness of CodeSeaXDataset dataset in enhancing the instruction-following ability of the base model.
2) According to Table 4 and Table 5, All WaveCoder models significantly outperform the model fine-tuned with CodeAlpaca. Remarkably, The pass@1 score of WaveCoder-CL-13B outperforms CodeLLaMa-CodeAlpaca-13B achieving 10.6% absolute improvements on HumanEvalExplain. This emphasizes the effectiveness of defining and classifying code-related tasks on enhancing the generalization ability of Code LLMs.

heading: D Evaluation Benchmark
text: HumanEval 4, which consists of 164 manually-written Python programming problems and an average of 9.6 test cases allocated to each problem is now the most extensively adopted benchmark for Code LLMs. MBPP 5 consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. In this paper, we choose the 500 problems test dataset to evaluate both few-shot inference of fine-tuned models. For whose MBPP (500) result is not reported or not used, we reproduced for them using bigcode-evaluation-harness 6. HumanEvalPack 7 is an extension of OpenAI’s HumanEval to cover 6 total languages across 3 tasks. In this paper, we select the HumanEvalFix to evaluate the code to code ability especially on code repair task and HumanEvalExplain benchmarks to evaluate the code to text ability especially on code summarization task.
4https://huggingface.co/datasets/openai_humaneval 5https://huggingface.co/datasets/mbpp 6https://github.com/bigcode-project/bigcode-evaluation-harness 7https://huggingface.co/datasets/bigcode/humanevalpack

heading: Future Work
text: Furthermore, our analysis of the relationship of different tasks provides valuable insights for future research, paving the way for more extensive code-related tasks and larger dataset. Limitations
We present WaveCoder and propose a data generation method which can stably generate high-quality and diversity instruction data from open source dataset in multi-task scenario. One limitation of our work is that the training dataset we used only includes 19,915 instructions, which produces limited enhancements to the model. As illustrated Section 3, we expand the training dataset to a larger amount and the resulted model still have significant improvement. Therefore, future work should focus on more code-related task types and larger dataset