CODEM: Less Data Yields More Versatility via Ability Matrix (2024)

Abstract: In the era of code large language models (code LLMs), data engineering plays a pivotal role during the instruction fine-tuning phase. To train a versatile model, previous efforts devote tremendous efforts to crafting instruction data that covers all the downstream scenarios. Nonetheless, this will incur significant expenses in data construction and model training. Therefore, this paper introduces CODEM, a novel data construction strategy, which can efficiently train a versatile model using less data via our newly proposed ability matrix. CODEM uses ability matrix to decouple code LLMs’ abilities into two dimensions, constructing a lightweight training corpus that only covers a subset of target scenarios. Extensive experiments on HumanEvalPack and MultiPL-E reveal that code LLMs can combine the singledimensional abilities to master composed abilities, validating the effectiveness of CODEM.

heading: 1 Introduction
text: Code large language models (code LLMs) have been booming recently (Zan et al., 2023; Zhang et al., 2023b). An abundance of code LLMs are released in succession, e.g., Codex (Chen et al., 2021), AlphaCode (Li et al., 2022), StarCoder (Li et al., 2023), and CodeLlama (Rozière et al., 2023). Recent trends have witnessed the versatility of code LLMs, aiming to train a multilingual multitasking model. To meet this need, some efforts (Di et al., 2023; Zheng et al., 2023a) typically created the corresponding instruction training data for every downstream language and task, to fine-tune the model. However, this will entail significant costs in constructing data and training models, considering the countless downstream scenarios (Zheng et al., 2023b; Cassano et al., 2022). For instance, if we enable the model to support 3 tasks and 6 languages, we need to laboriously craft training
∗Equal contribution. B. Guan, Q. Wang, and L. Cui are corresponding authors.
data for a total of 18 diverse scenarios. Moreover, our expectations for the versatile abilities of code LLMs are continually growing (Muennighoff et al., 2023; Cassano et al., 2022; Zheng et al., 2023a).
To more efficiently fine-tune code LLMs, we propose a novel data construction strategy, namely CODEM, which can empower CODE LLMs with powerful Multilingual Multitasking abilities using less training data via our newly proposed ability Matrix. This matrix assists CODEM in decoupling the versatile abilities of code LLMs into two ability dimensions (understanding languages and completing tasks). Then, CODEM only requires constructing the training datasets that cover each single-dimensional ability (e.g., Python, CG, or CE), rather than that of all composed abilities (e.g., Python+CG, Java+CE, or Go+CR). We suppose that those composed abilities not covered by the constructed dataset can be implicitly learnable based on two conjectures: (1) code LLMs can generalize based on explicitly learnable abilities; (2) code LLMs can imitate based on the relationships between multiple explicitly learnable abilities.
As shown in Figure 1, compared to its baseline SFT-Mixed, CODEM only needs to cover a subset of 18 composed abilities by picking out one row and one column. CODEM aims to master all com-
714
posed abilities (e.g., Java+CE) by combining the explicitly learned single-dimensional abilities (e.g., Java and CE), and showcases uncompromising performance in these implicitly learned abilities. To evaluate CODEM, we first craft the corresponding instruction training data for each of these 18 scenarios. Then, we conduct experiments to validate CODEM’s effectiveness on HumanEvalPack. Extensive results demonstrate that CODEM can rival SFT-Mixed, while utilizing less than half the data.
To more comprehensively verify CODEM, we explore the proposed two conjectures for CODEM. We first verify the generalization of code LLMs across 15 languages, 7 tasks, and even 3 domains on MultiPL-E. Our findings reveal that the model trained on one scenario training data can produce a universal generalization. Secondly, to verify whether the code model can imitate, we deliberately remove key data from CODEM’s training corpus to disrupt the preconditions for imitation, and observe performance changes. The results prove the existence of imitation behavior in code LLMs.
In a nutshell, our contributions can be listed as follows: (1) We introduce CODEM, a simple yet effective data construction strategy, which can yield more versatility of code LLMs using less training data via our newly proposed ability matrix. This matrix is capable of decoupling multiple scenarios into two ability dimensions, thereby aiding CODEM in achieving efficient training. (2) We carry out extensive experiments on HumanEvalPack and MultiPL-E to validate CODEM’s effectiveness, as well as offer some valuable analyses. (3) Our work has been open-sourced at https: //github.com/NL2Code/CodeM.

heading: 2 CODEM

heading: 2.1 Task Definition
text: The goal of CODEM is to obtain a versatile code LLM via efficient instruction fine-tuning. By combining multiple programming languages and coding tasks, we can derive a scenario set, formatted as S = L× T , where L denotes a set of target programming languages and T denotes a set of coding tasks. Each element s = (l, t) ∈ S corresponds to a concrete scenario, requiring code LLMs to complete a task t with the language l. Thus, we in total obtain |L|·|T | different scenarios. Given a scenario set S, a versatile model is expected to support all scenarios and achieve a balanced performance across different scenarios.
Although the scenario set can have countless variations in practice, we mainly focus on one specific version to validate CODEM. Following HumanEvalPack, we regard 3 coding tasks across 6 languages as our target scenario set S in this paper. The 6 languages include Python, JavaScript, Java, Go, C++, and Rust and the 3 tasks include code generation (CG), code explanation (CE) and code repair (CR), leading to a total of 18 scenarios.

heading: 2.2 Approach
text: CODEM is essentially a data construction strategy for efficient fine-tuning, aiming to empower code LLMs with powerful multilingual and multitasking abilities using less training data. To train a versatile model, a naive idea is to collect training data for each scenario. However, this approach will suffer from overabundant downstream scenarios, resulting in considerable computational resource consumption. CODEM introduces a new concept of ability matrix, aiming to select a subset of scenarios from the fullset S to train the model, while still maintaining an uncompromising performance.
Ability Matrix As depicted in Figure 2, the ability matrix refers to an |L|×|T | matrix which covers all the abilities requested by the target scenario set. In the ability matrix, the two axes denote two ability dimensions. In detail, the vertical axe corresponds to the ability dimension of understanding the target programming languages, while the horizontal one corresponds to that of completing the target tasks. Based on the two dimensions, we define the |L|·|T | intersections between them as composed abilities.
Upon the above defined ability matrix, CODEM picks out one row and one column to select a scenario subset (|L| + |T | − 1 scenarios) from our
target scenario set (|L| · |T | scenarios). For each selected scenario, we collect the corresponding training dataset and merge them to obtain a training corpus. This training corpus covers all the singledimensional abilities without requiring coverage of all composed abilities. Given such a training corpus, the model is expected to combine the explicitly learned single-dimensional abilities, thereby generalizing to all composed abilities. We argue that the ability combination can be realized based on two conjectures (also illustrated in Figure 2): (1) code LLMs can generalize: the model can generalize well in every ability dimension, e.g., Python to Java and CG to CE; (2) code LLMs can imitate: the model can imitate based on the relationships between multiple abilities that have been explicitly learned. e.g., by imitating the relationship of CG and CE in the Python version, the model can generalize from CG in the Java version (explicitly learnable) to the CE in the Java version (implicitly learnable). If the two conjectures hold, all the composed abilities in the ability matrix can be reachable even though only a subset of them are explicitly learnable. The validity of the claimed two conjectures will be further discussed in Section 4.3.

heading: 3 Training Corpus Construction
text: Regarding the 3 tasks (CG, CE, and CR) across 6 languages (Python, JavaScript, Java, Go, C++, and Rust) mentioned in Section 2.1, totaling 18 scenarios, we craft corresponding instruction dataset for each of these by harnessing the capability of OpenAI’s GPT-3.5-turbo. In detail, we start from CodeAlpaca1, an instruction dataset of code generation, from which we extract those Python-related instances as our seed data. Based on these seed data, we meticulously design prompts for each of the 18 scenarios to request OpenAI’s GPT-3.5-turbo, deriving the corresponding instruction datasets. More details about prompt engineering can be seen in Appendix A. In our experiments, each training dataset contains 9.6K data pairs, where a concrete data pair is displayed in Appendix Figure 6.

heading: 4 Experiments

heading: 4.1 Experimental Setup

heading: 4.1.1 Benchmarks
text: We use HumanEvalPack (Zheng et al., 2023a) as our primary benchmark to evaluate the versatility of
1https://huggingface.co/datasets/ sahil2801/CodeAlpaca-20k
code LLMs brought about by CODEM. It provides a series of evaluations for 3 tasks (code generation, code explanation, and code repair) across 6 programming languages (Python, JavaScript, Java, Go, C++, and Rust). Besides that, we also conduct experiments based on MultiPL-E (Cassano et al., 2022), which focuses on the code generation evaluation across 11 programming languages, including Python, C, C++, JavaScript, TypeScript, PHP, Go, Rust, Bash, Java, and Racket. The two benchmarks are created by adapting HumanEval (Chen et al., 2021), which is a hand-written benchmark for Python code generation comprising 164 programming problems with comprehensive test cases.

heading: 4.1.2 Evaluation Metrics
text: Following HumanEvalPack (Muennighoff et al., 2023) and MultiPL-E (Cassano et al., 2022), we adopt pass@1 (Chen et al., 2021) as our metric to evaluate all models. Each model generates one answer using the greedy decoding strategy for each programming problem, and the answer would be executed on the given test cases. Only when all the test cases are passed, the programming problem can be considered solved with the generated code. In this setting, pass@1 can be formulated as |Pc||P | , where |P | denotes the total number of programming problems and |Pc| represents the number of solved problems. In essence, the pass@1 metric we use can be considered as the accuracy.

heading: 4.1.3 Baselines
text: To prove the effectiveness of CODEM, we design some baselines. The downstream benchmark for CODEM is HumanEvalPack, which covers 18 scenario tasks; CODEM chose 8 of these scenarios to train code LLMs. Hence, the primary baseline of CODEM is using data from 18 scenarios, called SFT-Mixed, as shown in Figure 1 (b). Of note, for a fair comparison, SFT-Mixed contains a total of 9.6K data pairs by default, aligning with the size of each training dataset mentioned in Section 3. To thoroughly showcase CODEM’s superiority in terms of data volume, we also craft multiple versions of SFT-Mixed, each with different amounts of data pairs, spanning 4.3K, 19.2K, 28.8K, 48K, 76.8K, and 100K. Each of 18 scenarios in SFTMixed has the same data volume (e.g., 533 in 9.6K version of SFT-Mixed). In addition to the mixedscenario training data, we also compared the models trained on single-scenario data from each of the 18 scenarios. For example, SFT-CG (Py version) in
Table 1 is trained on the 9.6K instruction data pairs of Python code generation mentioned in Section 3. Furthermore, we also compare a variety of off-theshelf models, including CodeGeeX2 6B (Zheng et al., 2023a), CodeLlama 7B (Rozière et al., 2023), DeepSeekCoder 7B (Guo et al., 2024), WizardCoder 15B (Guo et al., 2024), CodeFuse 15B (Di et al., 2023), OctoCoder 15B (Muennighoff et al., 2023), StarCoder 1B, 3B, and 7B (Li et al., 2023). CODEM is based on StarCoder 7B for instruction tuning by default. For more implementation details, please refer to Appendix C.

heading: 4.2 Main Results
text: CODEM uses the data corresponding to any row and column in Figure 1, covering a total of 8 scenarios, to train code LLMs. Figure 1 comprises a total of 6 rows and 3 columns. We thus can derive 18 different versions of CODEM, where each of the rows and columns can be combined with each other. For instance, the model, trained on the Python row and the CG column in Figure 1, is named CODEM-CG#Py. Table 1 and Appendix Table 5 display the results of CODEM with varying amounts of training data and all baselines on HumanEvalPack. Among them, CODEM (4.3K) and SFT-Mixed (9.6K) each contain 533 data pairs per scenario, where the former spans 8 scenarios and the latter 18. By examining these results, we find that all CODEM (4.3K) variants perform on par with or even sometimes outperform SFT-Mixed (9.6K) across every downstream task. For instance, CODEM-CE#Java (4.3K) achieves an absolute pass@1 improvement of 2.4% over SFTMixed (9.6K) in the code explanation task of the Python version. Meanwhile, CODEM of 9.6K version exceeds SFT-Mixed with equal training scale and even rivals the 29K SFT-Mixed in some downstream tasks. As an illustration, CODEM-CR#C++ (9.6K) eclipses SFT-Mixed (29K) in the C++ code generation, and the Python&C++ code repair task. All these findings highlight CODEM’s effectiveness and superiority in constructing training data, even with a lower volume of data. Moreover, we observe that models trained on mixed scenario data, such as CODEM and SFT-Mixed, consistently surpass those trained on single scenario data like SFTCG/CE/CR in all downstream evaluation tasks. The results underscore the importance of data diversity, suggesting that mixed scenario data can complement each other and further elevate the model’s
capabilities. In addition, we also observe that CODEM (9.6K) with a 7B parameter size, can hold its ground against WizardCoder 15B and CodeFuse 15B in certain downstream tasks, even with fewer parameters. This further concluded that CODEM possesses remarkable advantages compared to its baselines. Overall, CODEM leverages ability matrix to reduce redundancy among multiple training scenarios, and to develop a versatile model using limited training data. This offers insightful guidance for constructing concise instruction data.

heading: 4.3 Conjecture Verification
text: We implement CODEM based on two conjectures as mentioned in Section 2.2: (1) code LLMs can generalize; (2) code LLMs can imitate. We will explore the validity of them via experiments.

heading: 4.3.1 Can Code LLMs Generalize?
text: We would like to verify whether code LLMs possess generalization capabilities under different training scenarios. For the sake of comprehensiveness of our experiments, in addition to the scenarios provided by HumanEvalPack, we also select a broader range of scenarios. Those new scenarios cover more programming languages of the code generation task (C, TypeScript, PHP, Bash, Racket, Haskell, SQL, HTML, and XML), more coding tasks (code translation, test case generation, code commenting, code-related questing answering), and even more domains (math and natural language). For each scenario, we create the corresponding dataset (see Appendix A and B for more details). We fine-tune StarCoder 7B on each dataset separately, and then evaluate each fine-tuned model on MultiPL-E and HumanEvalPack.
Table 1 and Table 2 respectively display the evaluation results on MultiPL-E and HumanEvalPack of the models trained on various scenarios. It can be observed that code LLMs can generalize across different languages, tasks, and even domains. As an example, SFT-JS, trained on JavaScript, achieves an absolute improvement of 13.1% on pass@1 for the Python code generation task, compared to its base model StarCoder 7B, and even exceeds SFTPython. We also observe that training on structured query languages (SFT-SQL) and markup languages (SFT-HTML and SFT-XML) can also yield performance improvements in general-purpose programming languages like Python and Java. For instance, SFT-HTML leads to an absolute improvement of 6.8% on pass@1 for C++ code generation. Further-
more, we discover that different tasks can generalize to each other. For example, SFT-CR, trained on the Python code repair task, achieves an absolute improvement of 6.8% on pass@1 for C++ code generation. By analyzing the results in Table 1, our findings reveal that CG and CR can always improve each other substantially, compared to CE. One plausible reason is that the essence of CG and CR lies in generating code, whereas CE involves understanding code and producing natural language output. More surprisingly, instruction datasets of non-coding domains can also bolster code LLMs in coding. SFT-NLQA and SFT-Math improve the JavaScript code generation performance by 11.8% and 6.8% on pass@1, respectively. That might be due to the enhancement in the model’s fundamental abilities, e.g. natural language understanding (Rozière et al., 2023) and reasoning (Dong et al., 2023). Overall, our findings consistently affirm the generalization of code LLMs, thereby solidifying the foundation of CODEM.

heading: 4.3.2 Can Code LLMs Imitate?
text: We design several ablation experiments to demonstrate whether code LLMs possess the imitation capability. The default setting of CODEM-CG#Py covers the Python row and the CG column, as stated in Figure 2. We intentionally exclude an intersection point of the Python row and the CG column from the training corpus of CODEM-CG#Py (abbreviated as w/o CG×Py in Table 1). Upon our conjectures, this will lead to a drop in CODEMCG#Py’s performance on all evaluation tasks: without the training instances of CG×Py scenario, the model can not reach those composed abilities that are not explicitly learnable through imitation. By observing the results in Table 1, compared to CODEM-CG#Py, excluding the CG×Py corpus indeed leads to a decline in performance, further confirming the correctness of our conjectures. Furthermore, we remove the CG×Java corpus from the training corpus of CODEM-CG#Py (w/o CG×Java), which results in the absence of any Java-related data. The results in Table 1 indicate that such the setting only has a negative impact on Java-related tasks, without affecting others. This observation also justifies our conjectures. Additionally, we exclude the Python CE corpus (w/o CE×Py) and observe the same pattern. In summary, the experimental findings confirm the imitation capability of code LLMs and further underscore CODEM’s effectiveness.

heading: 4.4 Closer Analysis
text: Data Scale CODEM takes the advantage of using less data to yield more versatility. To demonstrate CODEM’s advantage, we plot the performance of CODEM and its baseline SFT-Mixed across various data scales on HumanEvalPack, as shown in Figure 3 (a). We observe that, at any data scale, various versions of CODEM consistently outperform SFT-Mixed. For example, with a dataset scale of 19.2K, CODEM-CE#Java outperforms SFT-Mixed by 7.2% in pass@1, achieving performance on par with the 48K version of SFT-Mixed. This observation demonstrates CODEM’s advantage in terms of data scale. Additionally, with sufficient data, CODEM can achieve peak performance comparable to that of SFT-Mixed. This indicates that with CODEM, the model can seamlessly combine singledimensional abilities within ability matrix, achieving uncompromising performance on composed abilities without explicit learning. Compared to SFT-Mixed, CODEM can converge to its peak performance more rapidly, where the former reaches its peak at 100K and the latter at 28.8K, further underscoring CODEM’s advantage.
Data Quality We intend to explore the robustness of CODEM and its baseline with respect to data quality. In Figure 3 (b), we deliberately introduce noise for CODEM (4.3K) and SFT-Mixed (9.6K), by creating data pairs with inconsistent instructions and responses, at different noisy levels from 0% to 100% at 10% intervals. We observe that at most noisy levels, CODEM outperforms its baseline. Also, as the noisy level increases, CODEM tends to stabilize around a 20% pass@1 rate, whereas SFT-Mixed continues to drop to 12.5% and is still on a downward trend. This exhibits CODEM’s greater robustness to noise data.
Data Redundancy In section 4.2, compared to SFT-Mixed, we claim that CODEM reduces data redundancy by selecting one row and one column
from 6×3 matrix. To prove this, we select not only one row (language) and column (task) from Figure 1 but also m rows and n columns2, where m ∈ {1, 2, 3, 4, 5, 6} and n ∈ {1, 2, 3}. We summarize the experimental results in Figure 4. We observe that increasing the selected scenarios (m>1 and n>1) does not yield performance gain, compared to CODEM (m=1 and n=1). For example, the pass@1 is 35.8% for m=6, n=2, compared to 35.9% for m=1, n=1. This suggests there exists redundancy in the training data among the 18 scenarios, further underscoring CODEM’s advantages.
2We randomly select rows and columns in the experiments.
Base Models CODEM can be built upon different base models, such as StarCoder (1B, 3B, 7B, 15B), CodeLlama (7B, 13B, 34B), and DeepSeekCoder (1B, 7B, 33B). We thus plot the results of CODEMCG#Py (4.3K) and its baseline SFT-Mixed (9.6K) in Figure 5. We observe that CODEM consistently improves the average pass@1 of various base models on HumanEvalPack, each achieving an improvement of over 10%. For instance, CODEM-CG#Py brings an absolute improvement of 19.5% for StarCoder 7B in the average pass@1. Notably, in all settings, CODEM with merely 4.3K training data can compete with SFT-Mixed with 9.6K, demonstrating the efficiency of CODEM. In this paper,
CODEM’s base model defaults to StarCoder, due to its fully open-source pre-training data, unlike CodeLlama and DeepSeekCoder.

heading: 5 Related Work

heading: 5.1 Code Large Language Models
text: Codex (Chen et al., 2021) with 12-billion parameters is able to solve Python programming problems automatically. This remarkable success triggered a significant buzz in both the academic and industrial realms. Followed by Codex, a plenty of code LLMs are released, including AlphaCode (Li et al., 2022), PaLM-Coder (Chowdhery et al., 2022), CodeGen (Nijkamp et al., 2023), InCoder (Fried et al., 2023), CodeT5 (Wang et al., 2021), PanGu-Coder (Christopoulou et al., 2022), PyCodeGPT (Zan et al., 2022), SantaCoder (Allal et al., 2023), CodeGeeX (Zheng et al., 2023a), StarCoder (Li et al., 2023), CodeLlama (Rozière et al., 2023), phi-1/1.5/2 (Gunasekar et al., 2023), CodeFuse (Di et al., 2023), and DeepSeekCoder (Guo et al., 2024). These above models are trained on a large-scale code corpus and achieve impressive code generation performance. Recent works (Ouyang et al., 2022; Zhang et al., 2023a) have witnessed the instruction tuning technique that can teach LLMs how to follow instructions. In the realm of code generation, WizardCoder (Luo et al., 2023), PanGu-Coder2 (Shen et al., 2023), CodeLlama-instruct (Rozière et al., 2023), Phind (Name, 2023), and DeepSeekCoderinstruct (Guo et al., 2024) also harness this technique to unlock their code related potential. In this paper, we fine-tune these off-the-shelf models on our crafted multi-lingual multitasking instruction data, to derive a versatile and powerful model.

heading: 5.2 Instruction Data
text: Instruction fine-tuning can unlock the potential of LLMs. In this process, constructing what kind of instruction data is a highly fascinating research topic (Zhang et al., 2023a; Zhao et al., 2023). Some studies (Zhou et al., 2023; Cao et al., 2023; Chen et al., 2023) claim that high-quality instruction data will yield significant performance improvements. Beyond data quality, some efforts also highlight the importance of data diversity (Bukharin and Zhao, 2024; Kapania et al., 2023). Recent works also propose some tricks for training LLMs from the perspective of data composition (Dong et al., 2024) and training sequence (Wang et al., 2023; Guo et al., 2024). Unlike prior studies, this paper aims to investigate how to enable code LLMs to achieve more versatility performance with less data, on multiple tasks across multiple languages.

heading: 6 Conclusion and Future Work
text: In this paper, we propose CODEM, which aims to empower code LLMs with powerful multilingual multitasking capabilities using less training data by leveraging ability matrix. The matrix assisting CODEM divides the model’s intrinsic abilities into two dimensions: languages and tasks, then guides the construction of instruction training data. Extensive experiments on HumanEvalPack and MultiPL-E demonstrate the effectiveness and superiority of CODEM. Furthermore, we validate two conjectures of CODEM: generalization and imitation, and obtain many insightful findings. In our future work, we would like to delve into code LLMs to uncover more effective and intriguing data construction strategies.
Limitations
While this paper showcases numerous intriguing findings on the generalization of code LLMs, it also poses a few limitations as follows:
• CODEM constructs instruction data from two dimensions: programming languages and tasks, with the expectation that the two dimensions can be better blended together to empower a more versatile code LLM. Technically speaking, CODEM is also capable of handling additional dimensions, such as data domains and data structures, which will be explored in our forthcoming research.
• To ensure the consistency of data origin and quality, as well as the fairness of the experiments, the majority of our experiments are conducted using GPT-3.5-generated data. But employing CODEM in practice might also involve data from other sources, potentially posing threats to our approach. Also, we do not verify the correctness of content generated by GPT-3.5, which is in line with previous works (Luo et al., 2023; Shen et al., 2023).
• Our work, including training, inference, and API requests to OpenAI’s GPT-3.5-turbo, requires a high cost in computational resources. Therefore, we will open-source our efforts to foster the rapid advancement of this field.

heading: Acknowledgements
text: This research was supported by the National Key Research and Development Program of China, under Grant No. 2022ZD0120201 - “Unified Representation and Knowledge Graph Construction for Science Popularization Resources”.

heading: A Prompts for Requesting OpenAI’s GPT-3.5-turbo
text: To collect the training datasets for our selected scenarios where each instance is an instructionresponse pair (an example is presented in Figure 6), we craft two types of prompts to request GPT-3.5turbo: the instruction request prompt for instruction generation, and the response request prompt for requesting a corresponding response. The prompt
formats differ between different languages, tasks, and domains.
For instruction request prompts, which are presented in Appendix Table 3, we typically ask OpenAI’s GPT-3.5-turbo to craft a new task based on given seed tasks and optional additional information. The placeholder {seed task} in the prompt templates corresponds to a seed task3; {TabFact topic} corresponds to a brief topic randomly selected from the dataset TabFact4; {language} corresponds to a specified programming language.
For response request prompts, which are presented in Appendix Table 4, we typically utilize the obtained instructions to request a corresponding answer from GPT-3.5-turbo. The placeholder {programming task} in the prompt templates corresponds to an instruction obtained via instruction requests or retrieving from the existing dataset (e.g., SFT-Python); {code snippet} corresponds to a suitable code snippet, which can be retrieved from the already crafted dataset (e.g., SFT-Python).
3The seed task is CodeAlpaca mentioned in Section 3. 4https://huggingface.co/datasets/tab_
fact

heading: B Scenario Selection for Generalization
text: Verification
As mentioned in Section 3 and 4.3.1, in this paper, we have crafted instruction data for diversified training scenarios, covering a total of 15 programming languages, 7 code-related tasks, and 3 domains. This section will explain the reasons behind the selection of these scenarios.
15 Programming Languages We meticulously select 15 languages, including Python, C, C++, JavaScript, TypeScript, PHP, Go, Rust, Bash, Java, Racket, Haskell, SQL HTML, and XML. To derive a diversified set of training scenarios, these selected languages cover a broad range of language features, considering aspects of programming paradigm, type system, memory management, etc., as shown in Appendix Table 6. In addition, we also take into account the share of these languages in the pre-trained corpus5 of our base model, ranging from 0% to 15% all over. For each of the 15 languages, we have created an instruction dataset of code generation correspondingly (see Appendix A for more details).
7 Tasks We select 7 code-related tasks: code generation, code explanation, code repair, code translation, test case generation, code commenting, and code-related questing answering. We select these tasks because they represent a comprehensive range of skills required in software development and maintenance (Zheng et al., 2023b; Zhang et al., 2023b). For each of the 7 tasks, we have created multiple language versions of instruction data, including Python, JavaScript, Java, Go, C++, and Rust (see Appendix A for more details).
3 Domains We select 3 domains including code, math, and natural language. We choose the latter two domains based on the assumption that arithmetic reasoning and natural language understanding may also bolster the code-related capabilities of code LLMs (Rozière et al., 2023; Name, 2023). As for the instruction dataset of two non-code domains, we directly extract 9.6K data pairs from the GSM8K (Cobbe et al., 2021) and WizardLM (Xu et al., 2023) datasets.
5https://huggingface.co/datasets/ bigcode/the-stack
C Implementation Details
C.1 Training and Inference
We fine-tune StarCoder using PyTorch (Paszke et al., 2019), transformers (Wolf et al., 2019), and
DeepSpeed (Rajbhandari et al., 2020) with FP16 enabled. During instruction tuning, we set the batch size to 8, the epoch to 4, the max length to 1024, the warmup ratio to 0.03, the gradient accumulation steps to 16, the save steps to 10, and the
learning rate to 2e-5 with cosine scheduler. When fine-tuning on one training dataset, we report the results of the last checkpoint. In our experiments, all results are truncated to one decimal place. The number of samples in MultiPL-E for Python, C, C++, JavaScript, TypeScript, PHP, Go, Rust, Bash, Java, and Racket are 161, 162, 161, 161, 159, 161, 154, 156, 158, 158, and 161, while HumanEval-
Pack uniformly contains 164 programming problems for various evaluation scenarios.
We use the same prompt for training and inference for each task. To align the forms across diverse tasks, we design similar prompts for each task, while we also employ distinct descriptions in the prompt as a prefix of the task instruction to differentiate them. As shown in Appendix Table 7,
we utilize instructions generated by GPT-3.5-turbo as the content of ### Instruction section for most tasks. In the case of SFT-TestCase, we also provide the function signature in the prompt for the models’ reference. In the case of SFT-Comment, we provide a code snippet without any code annotation, as well as a constant prompt: “Please add appropriate comments for the given code snippet.”
C.2 Other Details
(1) To ensure that the data generated by GPT3.5-turbo meets our requirements, we manually review them. It is noteworthy that we remove those HTML code responses where there are embedded JavaScript scripts. (2) HumanEvalPack (Muennighoff et al., 2023) offers two versions of evaluation sets for the code repair task, including HumanEvalFixDocs and HumanEvalFixTests. We opt for the former to conduct our exper-
iments. (3) Among the competitive models evaluated in our study (Table 1 and Table 2), CodeLlama (Rozière et al., 2023) and CodeFuse (Di et al., 2023) have multiple versions. The specific versions we used are CodeLlama-7b-hf6 and CodeFuse-StarCoder-15B7.
6https://huggingface.co/codellama/ CodeLlama-7b-hf
7https://huggingface.co/codefuse-ai/ CodeFuse-StarCoder-15B

heading: Future Work
text: In our future work, we would like to delve into code LLMs to uncover more effective and intriguing data construction strategies. Limitations
While this paper showcases numerous intriguing findings on the generalization of code LLMs, it also poses a few limitations as follows:
• CODEM constructs instruction data from two dimensions: programming languages and tasks, with the expectation that the two dimensions can be better blended together to empower a more versatile code LLM. Technically speaking, CODEM is also capable of handling additional dimensions, such as data domains and data structures, which will be explored in our forthcoming research. • To ensure the consistency of data origin and quality, as well as the fairness of the experiments, the majority of our experiments are conducted using GPT-3. 5-generated data. But employing CODEM in practice might also involve data from other sources, potentially posing threats to our approach. Also, we do not verify the correctness of content generated by GPT-3. 5, which is in line with previous works (Luo et al. , 2023; Shen et al. , 2023). • Our work, including training, inference, and API requests to OpenAI’s GPT-3. 5-turbo, requires a high cost in computational resources. Therefore, we will open-source our efforts to foster the rapid advancement of this field