AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data
Authors: Zifan Song
Yudong Wang
Wenwei Zhang
Kuikun Liu
Chengqi Lyu
Demin Song
Qipeng Guo
Hang Yan
Dahua Lin
Kai Chen
Cairong Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Code Generation
Instruction Fine-tuning
Hindsight Relabeling
Primary area: natural_language_processing
TL;DR: We propose to harmonize multi-source data for instruction tuning Code LLMs and develop AlchemistCoder series, outperforming all models of the same size (6.7B/7B) and rivaling or even surpassing larger models (15B/33B/70B/ChatGPT).

Abstract: Open-source Large Language Models (LLMs) and their specialized variants, particularly Code LLMs, have recently delivered impressive performance. However, previous Code LLMs are typically fine-tuned on single-source data with limited quality and diversity, which may insufficiently elicit the potential of pre-trained Code LLMs. In this paper, we present AlchemistCoder, a series of Code LLMs with enhanced code generation and generalization capabilities fine-tuned on multi-source data. To achieve this, we pioneer to unveil inherent conflicts among the various styles and qualities in multi-source code corpora and introduce data-specific prompts with hindsight relabeling, termed AlchemistPrompts, to harmonize different data sources and instruction-response pairs. Additionally, we propose incorporating the data construction process into the fine-tuning data as code comprehension tasks, including instruction evolution, data filtering, and code review. Extensive experiments demonstrate that AlchemistCoder holds a clear lead among all models of the same size (6.7B/7B) and rivals or even surpasses larger models (15B/33B/70B), showcasing the efficacy of our method in refining instruction-following capabilities and advancing the boundaries of code intelligence. Source code and models are available at https://github.com/InternLM/AlchemistCoder.

AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data
Zifan Song 1.+
Yudong Wang2*
Wenwei Zhang
Kuikun Liu?
Chengqi Lyu²
Demin Song"
Qipeng Guo
Hang Yan
Dahua 2,n
Kai Chen Cairong Zhao
'Tongii University Shanghai Al Laboratory
MMLab, The Chinese University of Hong Kong
*HKGAI under InnoHK
ht https://internln. gi thub. io/hlchemiiisssee rtooe
Abstract
Open-source Large Language Models (LLMs) and their specialized variants, partic- ularly Code L.LMs, have recently delivered impressive performance. However, pre- vious Code LLMs are typically fine-tuned on single-source data with limited quality and diversity, which may insufficiently elicit the potential of pre-trained L.MMs. In this paper, present AlchemistCoder, series of Code LLMs with enhanced code generation and generalization capabilities fine-tuned on multi-source data. To achieve this, we pioneer to unveil inherent conflicts among the various styles and qualities in multi-source code corpora and introduce data-specific prompts with hindsight relabeling, termed AlchemistPrompts, to harmonize different data sources and instruction-rese respoosee pairs. Additionally, we propose incorporating the data construction process into the fine-tuning data code comprehension tasks, including instruction evolution, data filtering, and code review. Extensive experi- ments demonstrate that AlchemistCoder holds a clear lead among all models of the same size (6.7B/7B and rivals or even surpasses larger models (15B/33B/70B) showcasing the efficacy of our method in refining citrucction-ollowiiii capabilities and advancing the boundaries of code intelligence. Source code and models are available htt thtp::////ihubb.....//
thub. com/ Intp/nnteennn.. h/wlchemm/m/cmmimcicmmaca stCoder
80
Uamel te
33,5%
Uaraa-Chat-00
Improsvmentt
Introduction
49.9%
Cocolimme Pythen-345
Impreveneet,
HumanEval Score (Pa5s@1)
Closed-source Large Language Models (LLMs)
Dieplrek Coden intmued- 2B
176.0%
like ChatGPT and GPT-4 [33 34 exhibit im-
60
DeepSeek Coden iatruct-300
pressive code intelligence by learning on large-
Mapesta' ni
Mypostos CL.7
50
scale and diverse code corpus, which also ben-
McchmnistCader-d--0
efits many other applications, such as math
MechemtstCoder
reasoning (6]. embodied control (25] and
Anferiar perjermance Froce
alwarty wiving villi-uneee dltto
agent [46]. Since open-source LLMs 40) still
88
30
lag behind closed-source LLMMS 34 in this field,
there has been growing interest in investigating
the acquisition of code capabilities by develop- ing specialized Code LLMs [35] 14).
Base Model
10
30
50
60
70
80
MBPP Score (Pass@1)
The training of Code LLMs mainly goes through pre-training and fine-tuning stages [35 Pioneer
Figure 1: Performance scatter plot (top right better) of open-source models on mainstream code benchmarks, HumanEval and MBPP. Our AlchemistCoder series demonstrates astonishing performance across all open-source Code LLMss. 38th Conference on Neural Information Processing Systems (NcurIPS 2024).
Equal contributions + Corresponding author

---

(a) Open-Source Datu Integration
(b) Open-Source Data Evolution
Instruction Tuning
oper-source
open-source
dedupiication
Harmonized Duta
muti source
oustomized
Fow
comprohersion
data
Prompe
dats
Instruction:
Fine the shortast parn from one sottcc to ooher watto graps mare the code ttabsutaiio the requeament
Resgonse:
baklnar, fondtgraph, serreets
R listropliaae vie made
Unicker cods
Pretrained LLMs
alle then dysoeet
atparithe that
qrequality data
dets tormatting
Imulttpla choies)
CFS4
3. Sappertrr
Response: Paynon atttaa reg.iemmnt....
multissource
code
(c) AlchemistPrompr Customization
(d) Code Comprchension Task Construction
AlchemistCoder
Overview for developing AlchemistCoder series. We first integrate high-quality open-source data (a) and conduct data evolution based on them (b). Then, we adopt Alchemist to harmonize their inherent conflicts (c) and construct code comprehension data (d). We use a mix of these data to fine-tune various pre-trained LLMs to obtain our AlchemistCoder models.
Figure 2:
works 32
have amassed extensive code data for pre-training, while recent open-source models 30 44 highight the effectiveness of high-quality or targeted code fine-tuning datasets. Despite these advancements, current fine-tuning methods mainly rely on specific type of code- related question-answering dataset, unlike the pre-training stage that integrates code-related corpus from various sources [35). Such discrepancy indicates that the fine-tuning data may lack the necessary diversity to fully stimulate the capabilities of base models, resulting in limited performance, generalization, and robustness.
22 36
To overcome the limitations in quality and diversity within single-source data, we pioncer to explore integrating multi-source data for Code LLM fine-tuning. However, this is non-trivial paradigm and blindly integrating multi-source data can potentially lead to inferior performance (e.g., the DirectlyMix-L-78 model in Fig. D. To track this, unveil inherent conflicts in multi-source code corpora, including conflicting code language requirements and response styles. Inspired by hindsight relabeling [1 48) we propose to design data-specific prompts to harmonize the inherent conflicts for multi-source data integration, better eliciting the performance of base models. We term this form of prompts as Mohppss inspired by the power and definition of Alchemists:
" "Alchemist: Someone Who Transforms Things for the Better." Merriam Webster
Specifically, we first integrate several open-source code datasets and conduct instruction evolu- tion [30) based on some of them (Fig. 2 a. b)). As shown in Fig. (c.) for instruction-respome pairs of different sources,
adopt one LLM to generate AlchemistPrompts that accurately and explicitly describe the characteristics as requirements of the response to enrich the instructions. In-depth, the efficacy of TichemistPrompts is twofold: 1) Harmonization between different data sources: AlchemistPromprs generated from the same LLM have similar styles and can bridge the style differences between sources, while the introduction of Alchemist prrpppoettttttezeed data, accounting for only 5%, achieves a balance between data diversity and domain gaps; 2) Harmonization within instruction-responnee pairs: As fine-grained and data-specific prompts, Alchemist rommpss designed to augment instructions with specific programming languages, algorithm concepts, and other code- related information involved in responses, which can refine the alignment within instruction respoonse pairs and enhance the instruction-folloving abilities of line-tuned models.
Apart from the conventional problem- data, argue that the progression of code data (e.g., data evolution, cleaning, and quality evaluation) reflects higher-level capabilities and offers valuable insights for the enhancement of Code LLMs. Consequently, we delineate the construction of data into three integral tasks for training: instruction evolution, data filtering, and code review (see Fig. 2 (d)), facilitating enhanced code comprehension capabilities.
We conduct extensive experiments with various base models
40 and develop the instruction- tuned AlchemistCoder series. As shown in Fig. D on two mainstream code benchmarks, HvmanEval

---

Instruction:
Instruction:
Instruction (Source #182):
Change the color of the toxt blue.
Find the minimum two given numbers,
Replace all words with letters or more that start with vowel and and with consonant, with in the givon toxt.
Inout:
Input:
The
Output: miniman number is
"The quick brows Fox jumps over the Iacy dag""
Response:
Response:
achiove this
use ESS. Here's an ecomple of how you can
change the color the bluw, you can
"*Reasoning"""
Format**
miniuum givan numbers, we
"The quick brown fox jumps the basy dog"
Surel Here
Response (Source Wt]:
with the specifed replacements
use an statement, We will compare
Responsc (Source #2):
esty
.blue text [ color: Baan;
RTM..
rumbers, saaig the smaler value to variable called
"minimum, pum and "*Coce""*
Hore's a Python Tunction that ochieves this.
Pythax
del replace Jonce (baet:
</xtylee
class="blur the quick brow foxe/s>
if else:
Pytles
"a (o Mr
taxt)
clement, the wil be deppoyed in blue color.
applying
paragraph
#intmu number rued pránt("The
einiiwe mumberrrrr FAMR
prdntireplace wtt.seeeeeeeeeeeeeeeeeeee (ordd((act))
11790 basy
Inherent Confllct:
ainisun
Inherent Corpliet:
Mising Language Declanation. The instruction specify programming language, while
"*Eeccution Resurts:**
Respatse farmat Dicrepancyy For simber intrrct lom,
The minimum numberibbi é
Source #1 provices the result directly, while Source provides the implmmnnation code, which may lead confusion in understanding the instr intructions.
the response assumes HTMLLCSS and provides an implementation accordingly.
Inherent Confiict:
Format Exception. the resporse introduces
AlchemistPrompte pruttttiio Instruction: working
notebook fonmat specified instruction.
in this tasi,, you are required to create a Python program that scans through a given text and reploces spec fied words.
Huchembbrroonphcuuttto
Instruction (Source #2):
HTML document
Alchemiitrropp-ee Propproo-p--tttieeedde Instruction: Complete simple task notebook format, including reasoning, coda, and esecution results. Find the minimum two given numbers. ss input: 4,5 sa Output: The minimum number 4
and the permissions modify the CSS styles associated with
Rep aoc all words with letters or more that stort with vowel, and end with consonant, with the givin text.
Change the color of the coot to blue. atl Input: Text: The quick brown
"The quick brown for
the dog"
Figure 3: Examples of inherent confliets (e.g., various styles and quality) within multi-source code corpora. By applying Alchemist AlchemissPrompp-cuttnn rroppi-cuttomized instructions that are more consistent with the responses, the diversity from multiple sources can be effectively managed and utilized, thereby improving the quality of our fine-tuning data and the instruction-followin capabilities of the fine- tuned models.
and MBPP, AlchemistCoder holds clear lead among all models of equivalent size (6.7B/7B), and rivals or even surpasses larger models (15B/33B/70B), demonstrating harmonized and formidable code capabilities, Furthermore, we delve into the effectiveness of AlchemistPromee they alleviate the misalignment between instructions and responses within the data. Remarkably, AlchemistPrompe allow the code corpus to also significantly improve the general capability of Code L.Mss, as demonstrated by the improvements on MMLU, BBH, and GSM8K. Our main contributions are summarized as follows:
and discern that
Our work pioneers to integrate multi-source data for Code LLM fine-tuning to overcome the limitations of quality and diversity inherent in single-source data.
We unveil inherent conflicts within multi- code corpora and introduce Alchemist- Prompts, revealing the power of hindsight tuning for code generation, aiming to harmonize the confficts among sources and bridge the alignment within instruction-response pairs.
We propose incorporate data construction process into the fine-tuning data and design code comprehension tasks, including instruction evolution, data filtering. and code review, endowing comprehensive code capabilities.
Extensive ablation and analytical studies confirm the efficacy of our key concepts for enhancing the diversity, quality, and cost-effectiveness of Code LLM fine-tuning data. Through instruction tuning on various base models, we develop the AlchemistCoder series, surpassing all Code LLMs of the same size on a wide spectrum of code benchmarks.
2 Method
To more comprchensively elicit the capability of the base LLMs, we first construct multi-source data for fine-tuning ($ 2.T). which is harmonized by Alchemistrropps to take ( Code comprehension tasks are also constructed to further improve the performance 2.3). We also discuss
the details and statistics of the filtered and harmonized multi-source data in $ $2.4
2.1 Multi-source data construction
To fully elicit the capability of code LLMs,, we first collect the fine-tuning data from multiple sources (Fig. (la) and adopt the instruction evolution 3D) to improve the complexity of the instructions (Fig. 2()).. However, integrating multi-source data for instruction tuning poses challenges. Naturally,
3

---

one code-related question can be solved by different coding languages with various algorithms or response styles (e.g.. with or without reasoning). When naively combing data curated by different developers with different LLMMs, the model tends to learn to answer similar questions with different coding languages and response styles, as depicted Fig. g On the one hand, learning diverse responses may elicit different capability aspects of the base models. On the other hand, since the learned responses to similar instructions often diverge due to implicit human intentions, the LLMs the directly mixed data (e.g.. we
tend to be unaligned (to our expectation) after the fine-tuning cannot expect which coding language the LLMs will use in real-world applications), resulting in inferior performance. Therefore, directly mixing multi-source data is not promising solution and can be detrimental.
2.2 AlchemistPrompt
To harmonize the inherent conflicts within multi-source data, propose to customize data-specific prompts called AlchemistPrompps, (Fig. (())). inspired by the concept of hind-
AlchemistPrompt Generation
Here Il sample number 1) sets of Instructions-rssonnse paired coding samples. Each set begins with [SAMPLE BEGIN] and ends with [SAMPLE END]. Please read them carefully and complete the TASK.
sight 48 Specifically, we employ GPT-
4 34 to play the role of Alchemist and design the prompt as illustrated in Fig. F to obtain Al- themistPrompts. For instance, for instruction of 'Write code to find the shortest path from one vertex all other vertices in a graph', if the response involves Python code of Bellman- Ford algorithm with dynamic programming, we would expect to customize the instruction with an AlchemistPromph of Please generate Python code for the following task and attempt to use the concept of Dynamic Programming'.
[SAMPLE 1 BEGIN]
<I Instrrcctoon |>:
{ [Alchemistromppl )
[[ sample instruction
<| Response po
il sample response }}
[SAMPLE 1 END]
TASK:
Please [AlchemistPrompt J appropriate content (within words) to make the Instructions clearer and better match the
NOTE (very Important):
analyze (including not limited to programming languages, algorithm concepts, and code characteristics) and briefly refiect the results in [AlchemistProppt ) 2. You are not allowed to modily anything other than the contents within [AlchemistPrompt J.
1. you find code in the response,
For the selection of data customized by AlchemistPrompis, we calculate the differ- ences in perplexities of generating responses with/without given instructions, called Con- ditional Perplexity Discrepancy (CPD). Then, we selectively chose data with higher CPD values for AlchemistPrompp harmonizations. We treat CPD as an indicator of how data affects the complexity of model-generated re- sponses under given conditions (i.e., instructions), and its calculation formula is CPD = Perplexity(conditiona conditiona__nssruuctionn instruction + response) Perplexity (respons)). The level of CPD re-
3. You are asked to generate only one AlchemistPrompt J. which should be universally applicable all the sample number samples.
Figure 4: Detailed prompt designed for generating data-specific AlchemistPrompts.
flects the impact of the conditional instruction on the complexity of the generated response. Specif- ically, a high CPD indicates that the perplexity of the generated response significantly increases under the presence of conditional instruction, which usually reflects a poor alignment between the instruction and the response, the instruction may be unclear or not specific enough, or insufficient contextual information, thereby increasing the difficulty of model response generation. By analyzing high CPD values, we can identify cases where instructions and responses are poorly aligned and more effectively optimize data quality.
The adjustments to data made by AlchemistPromphs are relatively minor and well-calibrated. Our abla- tion study indicates that the optimal performance can be achieved by incorporating AlchemistPrompee into only 5% samples, striking a balance between the diversity and domain gap resulting from the fusion of multi-source data. Crucially, by retrospectively analyzing responses and reinterpreting them as alternate goals, the AichenisPrrmppts serve to elevate the condition/goal of data. This hindsight integration [ [88 26 allows for more nuanced and adaptive learning process, enhancing not only the models' comprehension of data but also refining celtructioo-oolloiiii capabilities.
2.3 Code comprehension task
The existing training datasets for Code L.LMs 24 4 39
primarily focus on the code generation task consisting of programming problems and their corresponding code solutions. However,

---

contend that beyond this, the process of constructing code data demonstrates higher-level abilities. Consequently, we devise three code comprehension tasks relevant data construction, including instruction evolution, data filtering, and code review (Fig. dd).
Instruction evolution. Inspired by the concept of instruction evolution 45
30
we employ GPT-3.5
33 to construct instruction evolution task data, which entails augmenting the requirements for instructions and providing detailed explanations for programming tasks. Integrating the instruction evolution task aids the model in discerning the disparities before and after evolutions, thereby deepening the comprehension of programming requirements, code complexity, task decomposition. and other code-related concepts.
Data filtering. We identify four categories of low- low-quality data from multiple sources: (a) responses that are excessively short and lack code, (b) code that fails to compile, (c) code with poor clarity, and (d) code that does not adhere the requirement in the instruction regarding its organization in function form. Each instruction in the data filtering task presents the model with a low-quality sample and prompts the model to classify it into one of the four categories. The data filtering task entails recycling the filtered-out data by offering counterexamples, thereby assisting the model in generating fewer low-quality responses.
Code review. In this task, we require the model to review a piece of code and assign scores between O and 10 for correctness and clarity separately. Additionally, the model is expected to provide suggestions for code improvement and present the refined code. To obtain higher-quality data, we utilize GPT-4 34 generate code reviews and select cases that are more representative, particularly those with average correctness and clarity scores exceeding 8 or falling below 6. Simultaneously, focus instances where aspect exhibits severe deficiencies, iee, the score of correctness or clarity is equal to or below 4.
2.4 Data cleaning and decontamination
In practice we have established a set of filtering rules enhance our data cleaning and purification procedures. These rules involve excluding samples based on various criteria, such as response length (either too short or too long), absence of code or insufficient code content, non-compilable code, code failing test cases (pertinent to certain samples), responses structured in notebook form, and instances with excessive textual descriptions preceding the code. After conducting an extensive series of validation experiments, we conclusively decide to eliminate low-quality data meeting either of the following conditions: (a) responses that are excessively brief and lack code. Such responses typically offer direct answers to the instructions, neglecting both the code solution and explanatory annotations. Additionally, these samples frequently present overly simplistic questions in the instructions; (b) code solutions that are non-compilable or fail test cases (relevant to specific samples).
Concurrentyy, following [3]. we employ N-gram similarity, cosine distance of code embeddings. and edit distance of code syntax trees to calculate the similarity between training data and samples in HumanEval and MBPP. We subsequently discard samples through this process of data filtering and deduplication, resulting in the removal of approximately 6% of the dataset.
2.5 Harmonized AlchemistCoder dataset
Our AlchemistCoder dataset (~200M tokens) comprises four types of multi-source data, encom- passing open-source datasets and three types of data constructed by us. Specifically, (a) open- source datasets including
[10], CodeExercise-Pyttee codcalpaca-vl (39), (b) EvolCode data generated from gpt-3.5-turbo following [30], (c) data cus-
Instruct-Code-800e
[9), and evol-
tomized by tithemistPromppse and (d) data of the code comprehension tasks (i.e., instruction evolu- tion, data filtering, and code review).
We visualize the distributions of data sources and programming languages using two circular graphs in Fig. Concurrently, Fig. rrpports a distribution of text description lengths and code lines. Compared to CodeAlpaca 4 and OOS-INSTRUCT 44, our AlchemisiCoder dataset presents a notably diverse distribution and maintains moderate overall text description and code lengths, benefiting significantly from the integration of multi-source data along with AlchemistP Prompts and code comprehension tasks. This is instrumental in contributing to a comprchensive evolution of code capability.
5

---

Comproteisbe (3.75)
AicheniscProngt
Length of Text Descripttin
600
RroCaale fod [N)
Python
500
(63,095)
300
124
Jask 590
200
twarfrrist 9%%)
(6
00SINSTRUCT AichemistCoder
Codle antot trtles (a.a))
(8.290)
C. Cae
pa 25
100
Lines of Cade
00
(N)
80
40
Figure 5: Data distribution analysis of our Al- chemistCoder dataset. The outer and inner cir- cular diagrams respectively display the distribu- tions of data composition and programming lan- guages. Data from pechhmmistroomppe and code comprehension tasks, constituting only 8% of the total data, plays crucial role in harmonizing and polishing the fine-tuning data.
CodeAipace
00S R&S TRUCT AichemistCoder
Figure 6: Comparative distribution text de- scription lengths (top) and code lines (bottom). Our dataset contains high-quality samples with more diverse distributions.
Experiments
In this section, we report results various benchmarks of code generation and conduct ablation experiments. Furthermore, we present analytical studies provide more in-depth demonstration of the efficacy of our AlchemistCoder.
3.1 Benchmarks and implementation details
Benchmarks. We adopt six code benchmarks: HumanEval (5), HumanEval+ [27], HumanEval-
[4), MBPP (2), MBPP+ (27), and DS-1000 (21]. In addition, access three mainstream
benchmarks (MMLU [15]. BBH (37). and GSM8K (8) to evaluate generalization abilities. All
evaluation and benchmark details be found in Appendix $D
Baselines. We compare with the following competitive baselines. Closed-Souree Models: GPT-3,5-
Turbo (33 and GPT-4-Turbe 34. Open-Souree Models: Llama 2 40). CodeLlama [35], StarCoder
22.. WizardCoder
30 DeepSeek-Codee [14), and Magicoder 44))
Supervised fine-tuning. We adopt Llama-2-7B. CodeLlama--Pthonn-255 and DeepSeek- Coder-Basse
6.7B the base models and fine-tune all the base models for epochs using NVIDIA A100-80GB
GPUs. We set the initial learning rate, minimum learning rate, and optimizer warmup steps and at le-4, 6e-6, and 15, respectively. use Adam optimizer (28) and choose batch size with sequence length of 8192.
3.2 Evaluation on code generation task
Results on python code generation. We first access HumanEval and MBPP to evaluate the capability of the AlchemistCoder series for Python code generation. These benchmarks necessitate models to generate code based on the function definitions and subsequently pass the test cases. Models are evaluated zero-shot on HumanEval and 3-shot on MBPP. The comprehensive comparisons Tab. and Fig. demonstratee the impressive capabilities of AlchemistCoder models. From the results, AlchemistCoder- attains a remarkable performance boost of 42.7% and 28.4% pass scores on HumanEval and MBPP respectively, compared to Llama 2-7B. Notably, Alchemis/Coder- the pass scores to 79.9% and 77.0% on these benchmarks, holding an overall improvement of 33.3%. Morcover, our AlchemistCoder series with 7B parameters outperforms larger models (e.g..
elevates

---

Table I: Results of pass @ HumanEval (HumanEval+) and MBPP (MBPP+) benchmarks. We
report the results of HvmanEval and MBPP consistently from the EvalPlus (27 and the bold scores denote the best performance among models of the same size.
Model
Params
Base Model
HumanEval (+) MBPP (+) Average (+)
Closed-source Models
GPT-o.5-Turbo
72.6 (65.9)
81.7 (69.4) (67.7)
GPT-4-Turho
85,4 (81.7)
83.0 (70.7)
84.2 (76.2)
Open-source Models
Llama 2-Chat
40
70B
Llama
31.7 (26.2)
52.1 (38.6)
41.9 (32.4)
Codel Jamm--ytttoonn
65
70B
Llama 2
57.9 (50.0)
72.4 (52.4)
65.2 (51.2)
loseelamm-Instrucc
35
70B
CodeLlama
65.2 (58.5)
73.5 (55.1)
69.4 (56.8)
CodeL Python
35
34B
Llama 2
51.8 (43.9)
67.2 (50.4)
59.5 (47.2)
WizardCoder-CL
300
34B
Clde Python
73.2 (56.7)
(51.9)
73.2 (54.3)
DeepScek- SopSeek-CCode--Ingg Coser-Instrect
14
33B
DeepSeck-Cooe Code-- Basc
78.7 (67.7)
78.7 (59.7)
78.7 (63.7)
StarCoder 22
15B
(33.5)
55.1 (43.4)
44.6 (38.5)
RodeLLama-PPthonr
35
13B
Llama
42.7 (36.6)
61.2 (45.6)
(41.1)
WizardCoder-SC 30
15B
StarCoder
51,9 (45.7)
61.9 (44.9)
56.9 (45.3)
Llama
40
7B
14.0 (10.4)
26.1 (17.5)
(14.0)
StarCoder
22
7B
24.4 (21.3)
33.1 (29.2)
28.8 (25.3)
Came-.laman
35
7B
Llama 2 CodeLlama-Py
(33.5)
(4.6))
47.7 (38.0)
WizardCoder-CL
30
7B
48.2 (42.1)
56.6
52.4
DeepSeek-Coder-Bll
6.7B
47.6
75.2
(47.6)
Magicoder-CLI 44
7B
CodeLlama- CodeLLama-Pythoor Python
(49.4)
64.2
62.3 (47.8)
MagicoderS-CI. CLCL.
7B
CodeLlamm--P Jaam- lamaa-Pyhhn
70.7 (60.4)
64.41)
(54.8)
Magieoder-DS
44
6.7B
DeepSeck- -oder- Basc
(55.5))))
75..
71.0
14
6.7B
DeepSeck- Coder-Caaar
(69.5)
72.7
73.3 (62.6)
MagicoderS-DS.
6.7B
DeepSeek- Ceeekk-CoeerBaaaa
(65.2)
75.7 (56.1)
(60.7)
Alchemis(Coder-L (ours)
7B
Llama 2
(52.4)
54.5 (49.6)
55160))))
Alchemis(Coder-CI. (ours)
7B
CodeLlama-PP Python
74.4 (68.3)
68.5 (55.1)
71.5 (61.7)
AlchemistCoder-DS (ours)
6.7B
DeepSeck DeepSeck-Coder Coder Basc
79.9 (75.6)
77.0 (60.2)
78.5 (67.9)
Table 2: Results of pass@ on HumanEval-X.
Table Pass@1 results models with 6.7B/7B parameters DS-1000, pd, np, tf, sp, skl, torch, and plt represent Pandas, Numpy, Tensorflow, Scipy, Sklearn, Pytorch, and Matplotlib, respectively.
We present the multilingual code capabilities of our AlchemistCoder with the respective base models and competitors (6. (6.7B/
Model
Python
C++
Go
Java
JS
Aag
Model
pd
np
tf
sp
skl
torch
plt
All
Llama 2
14.0
11.0
6.1
11.0
14.0
11.2
Llama 2
2.4
7.3
6.7
6.6
2.6
1.5
7.7
5.0
Codel.lama
31.7
27.4
12.8
25.6
32.9
26.1
CodeLlama
12.0
17.8
12.2
17.0
AlchemistCoder-L
45.1
37.1
AlchemissCader-..
13.4
22.7
31.1
11.3
25,2
29.0
20.2
CodeLlama-Pyitee
37.8
33.5
30,5
39.6
35.4
35.4
CodeLlana Python
16.2
MagiculerS CL.
68.3
47.6
39.6
34.8
57.9
49.6
Magicoder'S CL
25.1
40.9
35.6
29.3
36.5
38.2
$1.0
36.7
MichemisCoderr-C
74.4
53.1
42.7
64.0
52.4
57.3
AlchemistCoder-C
40.3
DeepSeek Coder Base
47.6
45.1
38.4
56.1
43.9
46.2
DeepSock-Code Caaesck-Cdde-Baaee Base
21.3
35.0
26.7
23.6
34.8
25.0
34.8
28.7
MagicoderS-DS
72.6
63.4
51.8
70.7
66.5
65.0
MagicoderS-DS
CichemisiCoder-DS
79.9
62.2
59.8
72.0
68.9
68.6
AlchomisConder-DS
32.0
51.7
44.5
33.1
38.4
33.8
49.8
40.5
WizardCoder-CL- and bridging the performance gap between closed-source and open-source models.
and rivals with GPT-3.5-Turbo, significantly
Results multilingual code generation. We compare the pass@) I accuracy of the base models and the corresponding fine-tumed AlchemistCoder models on Humaneval-X X 49). The results presented in Tab. demonstratee that the AlchemistCoder series exhibits great improvements (exceeding 50%) for multilinuaal code generation, delivering comprehensive code capabilities.
Results on code generation for data science. We further conduct the evaluation of data science code completion on DS-1000 21 According to Tab. 3 AlchemistCoder models exhibit a notable improvement of up to 19.2% in overall performance compared to the base models. Particularly,

---

Average Score (HumanEval MBPP
75
75
20.4% Improvement
73
73
19.4% Improvement
71
71
69
69
67
67
65
65
63
63
61
61
10.3% Improvement
8.9% Imprevement
59
59
HumanEval (Pass@1)
HumanEval (Pass 1]
57
MBPP (Pass@1)
57
MBPP (Pass@1)
55
55
0%
5%
10%
15%
20%
0%
5%
10%
15%
20%
AlchemistPrompt Ratio
AlchemistPrompt Ratio
Figure Ablation study on the proportion Alchemiss/rrmpp/-cusoniiies data conducted on AlchemistCoder- CL-7B. Left: Augment the original data. Right: Replace the original data.
Table 4: Ablation study the effectiveness of multi multi-source source harmonization (i.e., Multi-soure Inte- gration, Data Decontamination, and AlchemistPrompl Harmonization) and code understanding tasks (i.e., Instruction Evolution Task, Data Filtering Task, and Code Review Task) for the AlchemistCoder CL.7B model, evaluated on HumanEval and MBPP benchmarks.
Multi-source
AlchemistPrompt
Instruction Evolution Task
HumanEval (Pass@
Integration
Decontamination
Harmonization
Filtering Task
Review Task
(Pass@1) 1)
37.8
$7.6
54.6 (16.81)
57.9 (0.31)
(5.21)
(0.31)
(2.0
63.4 (5.21)
71.3 (0.71)
65.8 (2.4%)
73.8 (2.51)
67.7 (1.99)
74.4 (0.67)
68.5 (0.81)
AlchemistCoder- CL achieves an astonishing overall accuracy of 40.3% with relatively better perfor- mance in all libraries, demonstrating powerful capabilities data science workflows.
3.3
Ablation study
The Recipe of AlchemistPrompts. As illustrated in Sec. 2.2, peshemisttrrompl can further align the instructions and responses of data samples and harmonize the domain gap between multiple sources. Code data from different sources may vary significantly in language style and content, including question types, code style, presence of comments, test cases, etc. Therefore, multi-source data mixing is a doublc-edged sword: it provides necessary diversity but can also bring large domain gaps. Adding concise corpus generated from the same Alchemist model (i.e., AlchemistPrompss with similar language styles) to small amout of data can effectively bridge this gap while maintaining diversity. To find the appropriate recipe of AlchemistPrompts that maintains balance between data diversity and domain gap, we conduct ablation experiments on the proportion (0% to 20%) of data customized by AlchemistPromppts We adopt two settings: (a) augment the original data with its customized variant and report the results of fine-tuning for epochs replace the original data and report the results of fine-tuning for the same steps (i.e., keeping number of tokens used consistent). As shown in Fig. 7 AlchemistCoder is particularly enhanced when the proportion of customized data increases from 1 to 5%, and nearly peaks in performance at 5%. Thus, introduce AlchemistPromee gain and the generation cost. Additionally, both two strategies effectively enhance the performance and validate the efficacy of our approach. To push the limit of AlchemistCoder. augmentation strategy in our performance experiments. In addition, we present detailed experimental results from the multi-source integration and harmonization process in Appendix to offer more in in-depth demonstration of the AlchemistPromph efficacy as data complexity scales.
Lada--lama-
(b)
into 5% of the training set to balance the performance
employ the
Efficacy of the code comprehension tasks. We conduct ablation study on the key components of the code comprehension tasks to ascertain their individual contributions to the overall performance. As reported in Tab, compared to the baselines (the first and second rows), the model demonstrates

---

(Raw]
Conditional Permaaaeerpiiee perplexity Discrepancy
Instruction (AchemistPrampt)
0.0
1.0
Density
0.8
-1.0
-1.5
0.4
Instruction (Rtw])
-2.0
Instruction (Akhemisshrommt)
Response
-1.5
-1.0
-0.5
0.0
0.5
Aug À O Distance: Aug * Distance: 2.67
Conditional Perplexity Discrepancy
UMAP Visualization of Instruction/Response
Figure 8: In-depth analysis of the efficacy from AlchemistPrompts. Left: Kernel Density Estimation of Conditional Perplexity Discrepancy. Right: UMAP visualization of 10 instruction/rresponse groups.
enhanced performance on both benchmarks following the incremental incorporation of code com- prehension task data. Notably, the improvement (5.1% regard to the pass@) metric) is particularly remarkable on MBPP. This indicates the significant contribution of all code comprehension tasks to furthering programming capabilities.
3.4 Analytical study
AlchemistPrompt harmonize the discrepancy between instructions and responses. To in-depth verify the efficacy of AichemistPrompt we calculate the Conditional Perplexity Discrepancy (CPD, refer Sec. [.2p values of our fine-tuning data harmonized by AlchemistPromphe i.e., the difference between (exeilycoooiiionalll instruction + response) and Perplexity(respons) value quantifies the difficulty change in generating responses before and after adding specifie inputs (e.g., instructions) to the model (the smaller the value, the easier it becomes). Specifically, we adopt the instructions before and after customization by AlchemistPrompp
The CPD
for comparison, and provide the Kernel Density Estimation of CPD in Fig. g Clearly. the latter (green) gains smaller overall CPD values, indicating that AlchemistPromppe are beneficial for prediction and can provide effective contextual information. Furthermore, we randomly select 10 groups of these samples and use UMAP 31 to map their feature representations into 2-D space the right of Fig. [8] From the fact that the solid lines are generally shorter than the dashed lines, our AlchemistPromprs can harmonize the discrepancy between instructions and responses, leading to higher-quality data for attaining improved dsstuutiionfollowing ability.
AlchemistCoder models are better generalists. To further analyze the comprehensive capabilities of our AlchemistCoder, we conduct evaluations on more diversified benchmarks, including MMLU [1] for multitask language understanding, BBH (37) for comprehensive reasoning, and GSM8K for mathematical ability. The results are presented in Tab. 5 and illustrate that the AlchemistCoder models exhibit an overall performance increase of 6.4%, 13.6%. and 14.5% over the base models Llama 2, Maaeelllm--Python and DeepSeek-Code Base, respectively. Notably, CodeL Python presents inferior performance on these benchmarks relative to Llama 2, indicating the discrepancy between natural language processing and code capabilities of open-source models. Such divergence can be attributed to "catastrophic forgetting" [11 29) 20], often occurring when fine-tuning is exclusively concentrated on domain-specific data. By leveraging harmonized multi-source data, our AlchemistCoder series models achieve enhanced reasoning abilitics, better instruction instruction-follooiing following abilitics, and improved context understanding, which contribute develop better generalists.
Error case analysis. To meticulously dissect the improvements brought by our method, we provide an analysis of eтror cases HumanEval and MBPP. We compare models before and after the introduction of AlchemistPrompts and code understanding task data. The bar chart shown Fig. 9 (top) indicates that these two types of key data help to better handle compilation errors (i.e., Erro,,Error, NameError, and ValueErrorr, and eliminate the occurrence no code written in the responses. On the other hand, the results of Fig. (bottom) on MBPP suggest that the Alchemisi series incorporated with these two types of data attains stronger programming logic, as evidenced by the clear reduction in the 'Wrong Answer' еπтог cases.

---

@Syetaxiinor BAsserionEEoo @hhnnetmo BVilkefrro
Table 5: Results of models (6.7B/7B on vari- ous benchmarks, including MMLU for multitask language understanding, BBH for comprehensive reasoning, and GSM8K for mathematical ability.
Aklemittüoder
Akhemiscodee
Achemastzdee CL*
AMheriatCederd
30%
30%
50%
Model
MMLU
GSM8K
Avg
W'Wrong
Ilama 2
41.1
34.6
16.8
30.8
CodeLlama
31.5
42.7
14.4
29.5
AthhmannCodee
AlchemistCoder--
43,9
42.7
25.0
37.2
AthemistCader CL
CodeLlama-Pylin
26.1
26.7
6.6
19.8
MagicoleerS Cl.
33.0
41.5
18.8
31.1
AlchemistCoder-CI.
42.1
39.3
20.2
33,9
Figure 9: Analysis of error case proportions on HumanFval (top) and MBPP (bottom). repre-
DeepSeck-Coer--Bak
34.0
12.8
22.0
22.9
MagicoderS- DS
34.4
43.8
14.3
30.8
sents the models fine-tuned without Alchemist- Prompts and the code comprehension task data.
AlchemisCCnder.DS
38.5
45.6
28.0
37.4
4 Related Work
focus on collecting massive amounts of code data to develop pretrained Code LLMs. Recent efforts 30
Code large language models. Early researches 32 22
47 44 are dedicated to fine- tuning these pretrained models with specific instructional data to further the coding abilities. For instance, WizardCoder [30] and Magicoder 44 construct their instruction tuning datasets based on CodeAlpaca [4]] and the stack [8] dataset, respectively. In this work, we develop the AlchemistCoder series by instruction tuning optimized multi-source data instead of single-category data in previous methods, endowing astonishing and harmonized code capability.
Instruction tuning. Instruction tuning aims to enhance LLMs via fine-tuning pre-trained LLMs using samples of instruction/response pairs. Obtaining high-quality data for instruction tuning is typically challenging and extensive works have been dedicated to this endeavor. For instance, Alpaca (38 employs lelf-instruct [42 to generate instruction-followwe introduces Evol-Instruct and transforms the instruction data into more complex variants. In addition to Evol-untt we also incorporate the data construction process itself as a form of data into the
demonstrations. WizardI.M L [45)
training. Moreover, although previous works [16 43 41 23 utilize multiple fine-tuning datasets, we harmonize multi-source data at a fine-grained level.
Learning from hindsight. The concept of learning from hindsight (26 has been explored in goal-conditioned learning [2). Hindsight Experience Replay (HER) is designed re-label rewards and facilitate learning from sparse feedback. Korbak et al. [19] study the influence of human preferences during pre- gre-training, showing improved performance when models are aligned with human preferences. Previous work primarily serves as an alternative to RLFT, utilizing HER to leverage (suboptimal) historical data for model learning. We focus on harmonizing the inherent conflicts within multi-source data through hindsight, to fully tap into the potential of base models.
5 Conclusion
In this paper, we propose an effective framework for integrating multi-souree data fine-tune Code LLMs, addressing the limitations in quality and diversity inherent within single-source dataset. This a non-trivial paradigm and we pioneer to unveil inherent conflicts in multi-source code corpora. To resolve this challenge, we innovatively design data-specific AlchemistPromee Proeps, inspired by hindsight relabeling. Additionally, we make the first effort integrating the data construction process as code comprehension tasks into the training process. These key concepts enhance the diversity, quality, and cost-effectiveness of code fine-tuning data, facilitating the development of the AlchemistCoder series models with significantly improved and comprehensive coding capabilities.
6
Acknowledgments
This work is supported by National Natural Science Fund of China (62076184, 62473286) part by Shanghai Natural Science Foundation (22ZR 1466700). Besides, this project is funded in part by the Hong Kong Generative Al Research and Development Center (HKGAI) under the Innovation and Technology Commission (ITC)'s InnoHK. Dahua Lin is a PI of HKGAI under the InnoHK.
10

---

References
[1] Marcin Andrychowicz. Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbecl, and Wojeicch Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[2] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint a00: 07733, 2021.
[3] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, al. Internim2 technical report. arXiv preprint arXiv:2403. 17297, 2024.
[4] Sahil Chaudhary. Code alpaca: An iustruction-followee Ilama model for code generation, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint orXiv:2107.033
2021.
[6] Wenhu Chen, Xueguang Ma, Xinyi Wang. and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. CoRR, abs/2211. 0112211.1120 1. 12588, 2022.
[7] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2344.05222 2023.
[8] Karl Cobbe, Vineet Kosaraju. Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek. Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 10. 14168, 2021.
[9] codefuse ai. The codecxercise- -72k dataset. httpos//wnggingfeeeec///aatttttt SetExxercise-yythone 27k 2023.
[10] codefuse ai. The evolinstrutcode dataset. http:///wuggirreeeeeo///aatttttt
Avol-instruction 66k 2023.
[11]
Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang. Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv preprint arXiv:23 10.05492, 2023.
[12] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.0783 2022.
[13] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Tendoro Mendes, Allie Del Giomo, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, al. Textbooks are all you need. arXiv preprint arXiv:2306.1 11644, 2023.
[14] Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, al. Deepscek-coder: When the large language model meets programming the rise of code intelligence. arXiv preprint arXiv:2401. 14196, 2024.
[15] Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song. and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
[16] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in changing climate: Enhancing Im adaptation with tulu 2. arXiv preprint Xrv: 107002 2023.
[17] Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, volume 2, pages 1094-8. Citeseer, 1993.
[18] Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. The stack: 3 th of permissively licensed source code. arxiv preprint 2rXiv:2211.15533 2022.
[19] Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pages 17506-17533. PMLR, 2023.
[20] Suhas Kotha, Jacoh Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic forgetting in language models via implicit inference. arXiv preprint arXiv:2309. 10105, 2023.
[21] Yuhang Lai, Chengxi Li, Yiming Wang. Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference Machine Learning, pages 18319 1834-.8355... PMLR, 2023.
[22] Raymond Li, Loubna Ben Allal, Yangtian Zi Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06661 2023.
11

---

[23] Siyuan Li. Weiyang Jin, Zedong Wang, Fang Wu, Zicheng Liu, Cheng Tan, and Stan Z Li. Semireward: A general reward model for semi-supervised learning. arXiv preprint arXiv:2310.0301, 2023.
[24] Yujia Li. David Choi, Junyoung Chung. Nate Kushman, Julian Schrittwieser, Rémi Leblond. Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378866244:109221099 2022.
[25] Jacky Liang. Wenlong Huang. Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In ICRA, pages 9493-9500.
IEEE, 2023.
[26] Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of hindsight aligns language models with feedback. arXiv preprint arXiv:2302.02677. 3, 2023.
[27] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat- gpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.012/0 2023.
[28] Ilya Loshchilov and Frank Hutter.
Decoupled
weight decay regularization. arXiv preprint
[29] Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirieal study of catastrophic forgetting in large language models during continual fine-tuming. arXiv preprint arXiv:2308. 08747, 2023.
[30] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306. 08568, 2023.
[31] Leland McInnes, John Hcaly, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint wrXiv:1802.03426, 2018
[32] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang. Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203. 13474, 2022.
[33] OpenAL Chatgpt: Optimizing language models for dialogue. http:://opnna..co/bbbogcccaaaaa 2022.
[34] OpenAl. Gpt-4 technical report. Technical report, 2023.
[35] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla. Itai Gat, Xiaoqing Ellen Tan, Yossi Adi Jingyu Liu. Tal Remez, Jérémy Rapin, et al. Code Ilaaa: Open foundation models for code. arXiv preprint arXiv:2308.12900 2023.
[36] Demin Song, Honglin Guo, Yunhua Zhou, Shuhao Xing, Yudong Wang, Zifan Song, Wenwei Zhang, Qipeng Guo, Hang Yan, Xipeng Qiu, al. Code needs comments: Enhancing code Ilms with comment augmentation. arXiv preprint arXiv:2402.1300
2024.
[37] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung. Aakanksha Chowdhery, Quoe V Le, Fd H Chi, Denny Zhou, al. Challenging big-hench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.00 10.09261, 2022.
[38] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following Ilama model, 2023.
[39]
thehlackcat
The evolved code alpaca dataset. 2023.
https://hugginnfface. http://uuggnnffaeee.cc/o///tssss
[40] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09988 2023.
[41] Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Advancing open-source language models with mixed-quality data, arXiv preprint ar09v:2309. /235, 2023.
[42] Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint ar 22v: 105660 2022.
[43] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah Smith, Iz Beltagy, al. How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.0413 2023.
[44] Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02.120 2023.
[45] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng. Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint mrXv:2304.122444, 2023.
12

---

[46] Ke Yang, Jiateng Liu, John Wu, Chaoqi Yang. Yi R. Fung, Sha Li, Zixuan Huang, Xu Cao. Xingyao Wang. Yiquan Wang, Heng Ji, and Chengxiang Zhai. If LLM is the wizard, then code is the wand: A survey how code empowers large language models to serve as intelligent agents. CoRR, abs/2401 .00812, 2024.
[47] Zhaojian Yu, Xin Zhang, Ning Shang. Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation. arXiv preprint wixv:2312.14187. 2023.
[48] Tianjun Zhang, Fangehen Liu, Justin Wong, Picter Abbcel, and Joseph Gonzalez. The wisdom of hindsight makes language models better instruction followers. arXiv preprint arXiv:232..05066 2023.
[49] Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufci Xue, Zihan Wang, Lci Shen, Andi Wang. Yang Li, et al. Codegeex: A pre-trained model for cade generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303. 17566 2023.
13

---

Appendix
In the Appendix sections, we discuss limitations ( ethical considerations and broader impacts ($ additional experimental results ($C). benchmark and evaluation details ($D), and more details of our AlchemistCoder fine-tuning data (sE).
A Limitations
Currently, GPT-4 holds an advantage generating high-quality responses, and thus has been chosen as our Alchemist model. Compared to methods that heavily rely on GPT-4 to generate entire new datasets, have been striving to minimize our dependence generate data from scratch, we optimize a small amount of data. In the ablation experiments shown in Figure we have verified that achieving optimal performance only requires using GPT-4 generate Jichemistrrompe for 5% of the data. Furthermore, the generation tasks we designed only require very short responses (less than 50 words), significantly reducing the token usage of GPT-4. Despite these efforts, the generation of AlchemistP is still a significant cost. We will explore fine-tuning open-source models achieve the free generation of AlchemistPrompts in the future. For data bias, Our Alchemis(Prompp mitigate biases. For example, the model has a bias towards responding with Python code when the programming language is not specified, the inclusion of programming language declarations in Alchemist Prompts helps to alleviate this bias. We have not delved into the aspect of data bias yet and will explore in the future.
GPT-4, Instead of using GPT-4 to
enhance the instruction-followe
abilities of models, which can potentially
B Ethical Considerations and Broader Impacts
We use publicly available datasets, benchmarks, and models for training and evaluation, free from any possible harm toward individuals or groups. The generated data are relevant code-related tasks and no personal identification information is involved. Furthermore, we adopt ChatGPT to polish the writing and assist with language. For broader impacts, AlchemistCoder enhances code generation and generalization through multi-source fine-tuning, promising improved software development efficiency, democratization of programming, and educational benefits. However, also raises concerns about malicious use, intellectual property issues, and skill degradation. To ensure the responsible release of AlchemisiCodee models, we will implement controlled access, provided usage guidelines, and engaged with the research community, thereby mitigating the risks of misuse or dual-use.
C
Additional Experimental Resluts
C.1 Details of fine-tuning tokens
In Tab.
A1
we provide details of the training corpus used for fine-tuned Code LLMs.
C.2 Data complexity and multi-source integration
Our research demonstrates that integrating data from multiple sources significantly increases data complexity and diversity, as evidenced by the broader distributions of code and description lengths shown in Figure 6 of the manuscript. While this integration facilitates the model's 's ability to learn richer feature representations, also heightens the demands on the model to manage inputs varying styles, formats. and quality. AlchemistCoder addresses this challenge by introducing Alchemist which help conduct harmonizations across various data sources and within instruction-responee pairs. To offer more in-depth demonstration of the AlchemistPromee rompts efficacy as data complexity scales, we present detailed experimental results from the multi-source integration and harmonization process in Tab. A2
D Benchmark and Evaluation Details
D.I HumanEva/HHumatEaal-
14

---

Table A1: Pass results on HumanEval (HumanEval+) and MBPP (MBPP+) benchmarks. The column of FT Tokens (Source) specifies the tokens and origin of fine- fine-tuning (FT) data used to train each model.
Model
Params
Base Model
FT Tokens (Source)
HumanEval
MBPP
Average
Closed-source Models
GPT-3.5-Turbo
72.6 (65.9)
81.7 (69.4)
77.2 (67.7) 84.2 (76.2)
GPT-4-Turbo
85.4 (81.7)
83.0 (70.7)
Open-source Models
Llama 2-Chat
70B
Llama 2
31.7 (26.2)
52.1 (38.6)
41,9 (32.4)
CodeLlama-Python
70B
Llama 2
57.9
72.4
65.2 (51.2)
CodeLlame-Instruct
70B
CodeL.lmaa
(Llama Generation)
(58.5)
73.5 (55.1)
69.4 (56.8)
CodeLLama Python
34B
Liama
(43.9)
67.2 (50.4)
59.5 (47.2)
WizarCCoder-CLL
34B
CodeLlana-Python
(GPT Generation)
73.2 (56.7)
73.2 (51.9)
73.2 (54.3)
DecpScck-Code--nee
33B
DeepSeek-Codereee
2B (GitHub Crawling)
78.7 (67.7)
78.7 (59.7)
78.7 (63.7)
StarCoder
15B
34.1 (33.5)
55.1 (43.4)
44.6 (38.5)
Codel lama- Python
13B
Ilama 2 StarCoder
42.7 (36.6)
61.2 (45.6)
52.0 (41.1)
WizunlCoder-SC
15B
80M (GPT Generation)
51.9 (45.7)
61.9 (44.9)
56.9 (45.3)
Llama 2
7B
14.0(10.4)
26.1 (17.5)
20.1 (14.0)
StarCoder
24.4 (21.3)
33.1 (29.2)
28.8 (25.3)
CodeLlama-Pythoon
Ilama 2
(33.5)
57.6 (42.4)
47.7 (38.0)
WizardCoder-CL.
7B
CodeL-lma--Pyhh
80M (GPT Generation)
48.2 (42.1)
56.6 (42.4)
52.4 (42.3)
DeepSeek-
BerrSeee--Cdee-BBae Coder-Base
6.7B
47.6 (41.5)
70.2 (53.6)
58.9 (47.6)
Magicoder-CU CL
CodeLlans: Python
90M (GPT Generation)
60.4 (49.4)
64.2 (46.1)
62.3 (47.8)
MagicoderS CL
CodeLlama- Python
(GPT Generation)
(60.4)
(49.1)
69.6 (54.8)
Magicoder-DS
6.7B
DeepSeek-Codes Carekk-CodeBBBBee
90M (GPT Generation)
66.5 (55.5)
75.4 (55.6)
(55.6)
DecpScck-Coder-nnst
6.7B
DeepSeek-Codee Coder-Bae
2B (GitHub Crawling)
73.8 (69.5)
72.7 (55.6)
73.3 (62.6)
MagicoderS-DS
6,7B
DeepSeck- Codec---ac
240M (GPT Generation)
76.8 (65.2)
75.7 (56.1)
76,3 (60.7)
AlchemisCCderr-.. (ours)
Ilama 2
200M (GPT Harmonization)
56.7 (52.4)
54.5 (49.6)
55.6 (51.0)
AlchemistCoder--I. (ours)
Codel Python
Harmonization)
(68.3)
(55.1)
71.5 (61.7)
Alchemis(Coder-DS (ours)
6.7B
DeepSeek Coder. Base
200M (GPT Hamonization)
79.9 (75.6)
77.0 (60.2)
78.5 (67.9)
Table A2: Ablation study on the efficacy of multi-source integration and AlchemistPrompt harmo- nizations, evaluated on the HumanEval (Pass @ 1) and MBPP (Pass @ 1) benchmarks.
Method
HumanEval
MBPP
Baseline (Llama2-7B)
14.0
26.1
I-source data fine-tuniae mieittunng
18.3
29.0
AlchemistPrompt harmonizations
22.6 (4.31)
30.2 (1.21)
2-source data fine-tuning
35.4
30.6
AlchemistPrampt harmonizations
39.0 (3.61)
32.8 (2.21)
3-source data toe-ttnnng
37.8
35.4
AlchemistPrampi harmonizations
43.9 1f.11)
(5.41)
4-source data fine-tuning
40.2
42.2
AlchemistPrompt harmonizations
55.1 (14.91)
49.4 (7.21)
HumanEval (5) and HumanEval+ 27 are benchmarks for assessing LLMs' code generation. focusing on functional correctness. HumanEval+ expands on HumanEval by sig- nificantly increasing test cases through EvalPlus, using LLM and mutation strategies for more rigorous evalua- tion. This approach reveals performance drops in models like GPT-4 and ChatGPT against challenging tests, em- phasizing the need for diverse test scenarios to accurately evaluate LLMs coding abilities. For evaluation on Hu- manEval and Hva+, we adopt the prompt designed for HumanEvall www///aaa/Evaaaaaaa tasks shown in Fig. Al Fol-
Prompt
Below an instruction that describes task, Write response that appropristely completes the request.
Complete following python function.
(instruction:
Here the completed function:
Figure A1: Prompt used to evaluate on HumanEval and HvlanEval+.
15

---

lowing prior works 49 44], we use the greedy decoding
strategy and focus on comparing the pass @ metric.
D.2 MBPP/MBPP+
The MBPP (Mostly Basic Python Programming) benchmark (2 consists of around 1,000 Python challenges, crowd-sourced test basic programming skills, including fundamentals and standard library use. Aimed at beginners, each challenge offers a description. solution, and three tests for verifying solution accuracy. MBPP+ [27] is an extension of the MBPP benchmark, utilizing a subset of hand-verified problems from MBPP-ssaitizze to ensure tasks are well-defined and unambiguous. For the evaluation on MBPP and MBPP+, we adopt the three-shot prompt shown in Fig. A2
D.3 HumanEval-X
HumanBval-X [49 is a comprehensive benchmark that assesses the capabilities of code generation models across multiple programming languages, including Python, C++, Java, JavaScript, and Go. It consists of 820 meticulously created data samples, each accompanied by test cases, making it an invaluable resource for evaluating and improving multilingual code generation models. The benchmark aims to provide insights into the models' proficiency in solving diverse coding challenges and their aceuracy in generating functionally correct code in different languages. For evaluation on HumanEval-X. XX, we do not use specific prompts and follow the original test prompts.
D.4 DS-1000
The DS-1000 benchmark 21 adapts 1000 different data science coding problems each with unit tests from StackOverflow and checks both execution semantics and surface-form constraints. These realistic problems are drawn from seven popular data science libraries in Python, including Matplotlib (plt). NumPy (np), Pandas (pd), SciPy (scp), Scikit-Learn (sk), Py PyTorch (py), and TensorFlow (tf). DS- 1000 has two modes: completion and insertion, and here we only evaluate completion, as the basic ModeLlamm--yythnr does not support insertion. For evaluation on DS-1000, we do not use specific prompts and follow the original test prompts.
D.5 MMLU
The Massive Multitask Language Understanding (MMLU) benchmark [15] is an evaluation frame- work designed to measure the depth and breadth of knowledge that LLMs possess. It accomplishes this by testing these models across 57 varied tasks in both zero-shot and few-shot scenarios. The tasks encompass wide array of topics, including basic math, American history, computer science, law, and more, challenging the models to leverage their acquired knowledge to solve complex problems. MMLU seeks to emulate the multifaceted way which human knowledge and problem-solving skills are assessed, offering comprehensive gauge of a model's ability to understand and apply information across multiple domains. For evaluation on MMLU, we do not use specific prompts and follow the original test prompts.
D.6 BBH
The BIG-Bench Hard (BBH) Benchmark [37] is specialized evaluation framework tailored to rigorously test the capabilities of LLMs. This benchmark targets a selection of tasks that have historically proven challenging for LLMs, focusing on areas where models typically do not exceed average human performance. The BBH Benchmark aims to push the boundaries of what LLMs can achieve by emphasizing complex reasoning, deep understanding, and nuanced interpretation, setting high bar for model development and performance evaluation. For evaluation BBH, we do not use specific prompts and follow the original test prompts.
D.7 GSM8K
The GSM8K (Grade School Math 8,000) benchmark 8 serves as rigorous evaluation framework for testing the mathematical problem-solving prowess of L.Mss. This benchmark comprises a dataset of 8,500 diverse and high-quality math word problems the grade school level, designed to challenge
16

---

Prompt
Question:
You expert Python programmer, and here
your task:
[few_she shot instruction)
Your code should pass these tests:
Code Review Data Generation
ffew_shee shott case
You are asked to act as professional code reviewer and your task to professionally and accurately review the content and scorn. The involved programming languages include but are not limited Python, Cr+, Java, JavaScript, HTMI,, Haskell, SQI, Ct, PHP.
Your code should start with [BEGIN] and end
with [DONE] tag.
Answer:
[few_shot__re respooses
Question:
adhere following
requirements:
You are expert Python programmer, and here
Correct: No syntax and logir errors. The implementations should follow given function
your task:
{ few shot instruction
Clarity:
Variables should have meaningful names. The arguments and should type
Your code should pass these tests:
values
few shot case
Your cade should start with [BEGIN] and end
(important):*
[DONE] tag.
1. Please evaluste and score the code from the aspects of 'corrett' and 'darity',
Answer: { feww [few_shot_a] wtl__shooosppotee response
to following output [Correct]
Question:
syour (ranges [Clarity]
syour reviews
You are an expert Python programmer, and here
your task:
cyour score (ranges from 0 to 10)>, eyour review>
[few_sh new__hot__uee instrucicttt
[Suggestions]
Your code should pass these tests:
cyour additional suggestions for further ingrovemento
{ few_ssoot_ case
[Refined Code]
Your code should start with [BEGIN] tag and end [DONE] tag.
<The refined code according to your suggestions [DO NOT modify the function declarations appearing in the codel>
Answer:
there is no code the given content, please answer with [N/A].
[few_she www_shot_rreppnsse
DO NOT respond with content that outside of the specified output format.
Question:
You are an expert Python programmer, and here
your
Given Content: cade from multi
{ [few_shot shot instruction }
Your cade should pass tests: { few shot
case }
Figure A3: Detailed prompt designed for generating code review data.
Your cade should start with [BEGIN] tag and end with [DONE] tag.
Answer:
Figure A2: Three-shot prompt used evalu- ate MBPP and MBPP+.
LLMs with tasks necessitating advanced, multi-step reasoning abilities. GSM8K's primary aim is to gauge how well these models can parse, understand, and solve math problems, thereby offering comprehensive measure of their capacity for logical reasoning and mathematical computation. By incorporating such specialized benchmark, researchers can better understand the extent which LLMMs can mimic human-like reasoning in solving complex mathematical scenarios. For evaluation on GSM8K, we do not use specific prompts and follow the original test prompts.
E
AlchemistCoder Dataset Details
E.1 AlchemistPrompt
We provide two samples of AlchemistPrompts in Fig. A4 and Fig. A5
E.2
Code comprehension task data
For instruction evolution task data, we provide two samples in Fig. A6 and Fig. A7 For data filtering task data, we provide two samples in Fig. A8 and Fig. A9 For code review task data, we design prompt as illustrated in Fig. A3 to obtain high-quality code review task data and we provide two
samples in Fig. A10 and Fig. ATT
17

---

Instruction
Design function that finds the greatest common divisor (GCD) of two given numbers: 10, 20.
Response
Sure, here is a Python function that uses the Euclidean algorithm to find the GCD of two numbers:
def gcd(a, b): while b i- 8: 9, b, return abs(a)
print(gco(10,, 26)) Output: 10
This function keeps replacing with and with "a b' (the remainder of 'a b') until 'b" is '0', at which point 'a' the greatest
common divisor. So, the GCD of 10 and 20 would be 10.
You call the above function with any numbers you want find the GCD of.
AlchemissPrompp//uuttoiiid Instruction
Please provide Python code example for the following task including the expected output after the code execution. Design function that finds the greatest common divisor (GCD) of given numbers: 10, 20.
Figure A4: Example of AlchemistPrompts.
Instruction
Compare two strings and output true if they anagram. The input strings can contain both lowercase and uppercase letters.
Response
is antnaaaanssttrngg,, string?)):
string1 strenng..lowr(()
string2 string2.lower(()
string1 stringg. ", "")
string2 repineg..
list1 1ist(stringt)
list(string2)
list1.sart() list2.sort()
strinsttnggg-- "'.jot(llstt)) sorted_strings ".ooin(1ist2)
Conpare sorted strings if sorted_string stringl as sorted_string2:
else: return False
AlddeenisPrompp---usteized Instruction
Write Python function to check if two input strings are anagrams of each other, considering both lower and uppercase letters. An anagram is word or phrase formed by rearranging the letters of different word or phrase. Compare two strings and output true if they are anagram. The input strings can contain both lowercase and uppercase letters.
Figure A5: Example #2 of AlchemistPrompts.
18

---

Instruction
Please increase the difficulty of the given programming test question bit. You can increase the difficulty using, but not limited to, the following methods: the original problem can solved with only a few logical steps, please add more reasoning steps.
Given Prompt:
Construct SQL query add new column called "Gender" the table "Employees" with values "Male" and "Female", but also make sure that the values for the "Gender" column are randomly assigned for each employee.
Response
Construct an SQL query to add a new column called "Gender" to the table "Employees" with values "Male" and "Female", but also make sure that the values for the "Gender" column are randomly assigned for each employee. Additionally, ensure that the ratio of male to female employees is approximately 1:1.
Figure A6: Example #1 of instruction evolution task data.
Instruction
Please increase difficulty of the given programming test question a You increase the difficulty using, but not limited to, the following methods: If the original problem can be solved with only a few logical steps, please add more reasoning steps.
Given Prompt:
Iterate over a list and count the number of occurrences of each item. The list may contain duplicates and non-integer elements. Return dictionary with the count of each item. The input list can have maximum length of 1000.
Response
Iterate list and count the number of occurrences of each item. The list may contain duplicates and non-integer elements. However, now you need to also handle nested lists and dictionaries within the main list. You should recursively iterate through the nested data structures and count the occurrences of each item. Return a dictionary with the count of each item. Additionally, consider optimizing the code for performance, as the input list can have maximum length of 10,000.
Figure A7: Example #2 of instruction evolution task data.
19

---

Instruction
Below is code task and its corresponding solution. Please utilize your understanding of code to carefully review this solution.
Task:
Before we proceed, let's briefly talk about Python list indexing. Python uses zero-based indexing, which means the first element of a list at index The list 'a' here a mutable sequence of integers, and we are going to modify it using for loop.
Consider a list a = [1,3,2,2,3,4]* What would be the output of the following code?
[1,3,2,2,3,4]
for i in a[i] a(i -1]
Solution:
The correct output of the given code would be: [3,
The above solution considered to be of *low quality*. Choose from the following four options what category this solution belongs to:
a. The solution is too short (less than 20 words) and does not contain code.
b. There are code compilation errors in the solution.
c. The clarity of the code in the solution is too poor.
d. The solution contains multiple sections of code and is not organized into modularized functions.
Your choice (answer only the choice without the reason):
Response
a. The solution is too short (less than 20 words) and does not contain code.
Figure A8: Example #1 of data filtering task data.
20

---

Instruction
Below is a code task and corresponding solution. Please utilize your understanding of code to carefully review this solution.
Task:
Write function that takes as input a tuple of numbers (t_1,..,t[(N+1) and returns tuple length N where the i-th element of the tuple equal to t_i t_(i+1). The function should also handle negative numbers and zeros, and return an error message if the input is not a tuple.
Solution:
Here is a function implementation in Python:
def suple_produue (input_ (input_tuple) tuple):
1f not isinstance tttpt,,uple,,_ tuple)
return "Erron: Input is not tuple"
return (input_tuple(ii) * input_tuple(1. 1] for in range((ee(amput__tulle 1))
print(tuple Sacrlll___oduu((_____ (1, 2, 3, 4, 5))
Outputs: (2, 6, 12, 20)
print(tuple cectttule__rouu(((((e 2, -3, -5))
Outputs: (-2, -6, -12, -20)
The above solution is considered to be of *low quality*. Choose from the following four options what category this solution belongs to:
a. The solution is too short (less than 20 words) and does not contain code.
b. There are code compilation errors in the solution.
c. The clarity of the code the solution is too poor.
d. The solution contains multiple sections of code and organized into modularized functions.
Your choice (answer only the choice without the reason):
Response
b. There are code compilation errors in the solution.
Figure A9: Example #2 of data filtering task data.
21

---

Instruction
The prompt for generating data is omitted.
Given Content:
You can use the generate_json function to get JSON string:
import json
process process_value valuuevalue):
isinstance str):
return value. upper()
elif isinstance (value, int):
return value elif ssinstannce(value list):
return [process valuueel)mmntt) for element value] elif isinstance (value, dict):
return (key.upper(): process process_valuevval)) for key, val value.items())
else:
return value
def (iteeattejiso(diciioonrr))))
modified_diad sediffieddditttonayyy process process_valuse value((iictionrry)
droon.dups(teee (modi (nddiiieddddiiitinonr
return json_string
Response
[Correct]
9. The code is correct. It correctly processes the values according to the specified rules and returns the expected ISON string. [Clarity]
8. The code is clear and easy to understand. The function names are descriptive and the code follows a logical flow. [Suggestions]
would be helpful include type annotations for the function arguments and return values. This would make the code more clear and easier to understand. [Refined Code]
import json
from typing import Any, Dict, List, Union
def process_ (valee: Union[str, int, List[Any], Dict[str, Any]]) Union[str, int, List[Any], Dict[str, Any]]: (sinstanceevallue, str):
return upper()
elif isinstance(value, int):
return value elif isinstance(value, list):
return valuse_eeeueeeeeme for element in value]
elif isinstance(value, dict):
return ukey.upper(): sicess_valueevarr for key, val in value,items(
else:
return value
def generate_ (snndiiiiicn. Dict[str, Any]) -> str: modified dictionary process (diie((i)tionary) ison_string dumps (modiified_dcctionrry) return json_string
Figure A10: Example #1 of code review task data. For clarity, the prompt for generating data is omitted.
22

---

Instruction
The prompt for generating data is omitted.
Given Content:
The error in the code is an incorrect indentation when printing the indentation spaces. The number of spaces printed should based on the line number, the code mistakenly indents by fixed number of spaces. This leads misalignment and missing characters in some lines. Additionally, there are missing asterisks and periods in the inner loop, which causes the alternating pattern to be incorrect. By fixing the indentation and adding the missing characters, the tower of stars will be displayed correctly.
def print_star star__twer():
for i in range(1, 11):
indentation
print(" + i, end-"") Indentation should be based line number for j in range(1, 1*3):
if j == 8:
print('.. end="") Erron: Missing asterisk
else:
print('*', end="") Error: Missing period
print("")
Response
[Correct]
3. The code has syntax and there are logic errors that cause the tower of stars to be displayed incorrectly. [Clarity]
8. The code uses meaningful variable names and easy understand. [Suggestions]
The code could benefit from adding type annotations the function and its arguments.
The logic errors could be fixed by adjusting the indentation and adding the missing characters the inner loop.
[Refined Code]
def print_ste star (onnt_starrtoor()) -> None:
for i in range(1, 11): print("
end="")
for j in range(1, 1*3):
if j % 2 -- 0:
print('.', end="!)
else:
print('"', end-"')
print()
In the refined code, the indentation has been fixed and the missing characters have been added. Additionally, type annotations have been added to the function and arguments to improve clarity.
Figure All: Example #2 of code review task data. For clarity, the prompt for generating data is omitted.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly outline the primary contributions and scope the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Our work focuses on experimental research to advance the boundaries of code intelligence and does not include theoretical results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have fully disclosed the datasets, models, and experimental procedures used in detail. Benchmarks and implementation details can be found in Sec. 3.1 Appendix $D More details of our Alchemis(Coder fine-tuning data are provided in Appendix $
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have provided code with sufficient instructions in our supplemental material. We have released our code, data, and AlchemistCoder series models at htttss:/iittern)) gi thub. wat.:oooAllcheee www.chhetoosttooocccoccoccccoooe stCoder)
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: for more details.
/ //nips.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have specified all the training and test details. Benchmarks and implemen- tation details can be found in Sec. and Appendix $D More details of our AlchemistCoder fine-tuning data are provided in Appendix S
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper provides extensive information about the statistical significance of experiments. The factors of variability discussed, including the impact of different data sources and the harmonization process using AlchemistPrompts.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The experiments compute resources have been discussed in Sec. 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:////eeerrrsscccpppl////ssts
Answer: [Yes]
Justification: We fully follow the ethics guidelines NeurIPS and the ethical considerations have been discussed in Appendix
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discuss both potential positive societal impacts and negative societal
impacts of the work performed in Appendix B
Guidelines:
The answer NA means that there is no societal impact of the work performed.
27

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
The safeguards have been discussed in Appendix $B
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use publicly available datasets, benchmarks, and models for training and evaluation, free from any possible harm toward individuals or groups.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper thcode. com commdaaasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The paper introduces new models that well-documented. Documentation includes details about the dataset, code, and model, such as training procedures, licenses, limitations, and consent from data sources. We have released code, data, and Alchemist- gi thub. io/Alchemi stCoder'
Coder series models at htttp:://ntteere
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "sSWiEpGHuI", "reviewer": "Reviewer_q6UB", "summary": "This paper introduces AlchemistCoder, a series of code language models fine-tuned on multi-source data. The authors propose using \"AlchemistPrompts\" to harmonize inherent conflicts in multi-source code corpora and incorporate code comprehension tasks into the training process. The resulting models show improved performance on various code generation benchmarks compared to baseline models of similar size.", "questions": "1. Can you provide more insights into the relative contributions of data diversity and the harmonization process to the overall performance improvement? Are there scenarios where one factor might be more important than the other?\n2. How does the efficiency and effectiveness of the AlchemistPrompt approach scale with increasing model size and dataset complexity? Are there any computational or performance bottlenecks that might limit its applicability to very large language models or extremely diverse multi-source datasets?", "limitations": "The authors have addressed some limitations of their work, such as the potential for hallucination in response to obscure queries. However, there are several other important limitations that could be more thoroughly discussed:\n1. Scalability: The paper focuses on models with 6.7B/7B parameters. There's limited discussion on how the AlchemistPrompt approach and code comprehension tasks would scale to much larger models or more diverse datasets. The computational costs and potential performance bottlenecks for scaling up are not addressed.\n2. Data Bias: The process of selecting and harmonizing multi-source data could potentially introduce or amplify biases present in the original datasets. The paper doesn't thoroughly address how these biases are identified, mitigated, or might impact the model's outputs.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses an important challenge in fine-tuning code language models by utilizing multi-source data, which could potentially lead to more robust and versatile models.\n2. Empirical results demonstrate significant improvements over baseline models on several code generation benchmarks, including HumanEval, MBPP, and DS-1000. The authors provide detailed ablation studies and analyses to validate the effectiveness of their proposed methods, including the impact of AlchemistPrompts and code comprehension tasks.", "weaknesses": "1. The core concept of AlchemistPrompts lacks substantial novelty. It appears to be a variation of existing instruction evolution techniques, such as the Evol-Instruct.\n2. While the performance improvements are notable, the overall approach of using multi-source data and harmonizing it is not fundamentally new in the field of language model fine-tuning.\n3. The paper compares AlchemistCoder models (6.7B/7B parameters) with larger models, including 15B, 33B, and 70B models. While the results show that AlchemistCoder outperforms or rivals these larger models on several benchmarks (as seen in Table 1 and Figure 1), the paper does not provide a comprehensive analysis of the factors contributing to this performance gain. It would be valuable to have a more in-depth discussion on how much of this improvement can be attributed to the proposed method versus other potential factors such as the efficiency of smaller models or the quality of the base model used. This additional analysis would strengthen the paper's claims about the effectiveness of the AlchemistCoder approach relative to simply scaling up model size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1&W2: The novelty of the proposed method.**\n- Actually, our AlchemistPrompts are entirely different from existing instruction evolution techniques in several aspects:\n  - **Different designed goals**: Instruction evolution techniques are designed to \"expand into a richer and more complex set of instructions,\" whereas the goal of AlchemistPrompts is to harmonize multi-source data and instruction-response pairs. In short, the former aims at \"**expansion**\", while the latter focuses on \"**harmonization**\".\n  - **Different operational methods**: Instruction evolution techniques start from an initial set of instructions **only** and typically involve multiple rounds of **iterative** generation for **all** data. In contrast, AlchemistPrompts use only a **small** amount of data, require **both** instructions and responses, and only need to generate **once**. Additionally, instruction evolution techniques require generating **new** responses, whereas our AlchemistPrompts do **not**.\n  - **Different practical effects**: In terms of **diversity**, instruction evolution techniques significantly expand the data, while AlchemistPrompts suppress excessive diversity in multi-source data by incorporating corpora with similar language styles. Regarding **controllability**, instruction evolution techniques generate data with reference to a broad evolutionary direction, resulting in relatively high randomness, while AlchemistPrompts focus on fine-grained alignment within individual instruction-response pairs, thereby providing stronger controllability. For **data alteration**, AlchemistPrompts only insert concise corpus into the instructions, whereas instruction evolution techniques make substantial changes to both instructions and responses.\n- Additionally, our AlchemistPrompts explore a novel application of hindsight relabeling in the domain of Code LLM fine-tuning, which differs fundamentally from previous methods. Specifically:\n  |**Method**|**Designed Purpose**|**Relabeled Object**|**Generation Mode**|**Relabeling Period**|**Experience Source**|\n  |-|:-:|:-:|:-:|:-:|:-:|\n  |Previous Methods [a,b,c,d]|Alignment for Preferences|Conditional Goal|Handcrafting/Scripting|Postprocessing| Human|\n  |AlchemistPrompts (ours)|Harmonization for Multi-source Data|Data Instruction|LLM Generation|Preprocessing|LLM|\n- More importantly, our core contributions lie in pioneering the application of multi-source data **in the field of Code LLM fine-tuning** and we are **the first to unveil inherent conflicts in multi-source code corpora**. To achieve this, we propose logically progressive components of our method: AlchemistPrompts and code comprehension tasks. These key ideas introduce new insights into improving prompts and designing instruction fine-tuning tasks, enabling our fine-tuning data to be **more diverse, higher quality, and lower cost** for empowering AlchemistCoder with promising and comprehensive code capabilities.\n\n#### **Reference**\n[a] Andrychowicz M, et al. Hindsight experience replay. NeurIPS 2017.\n\n[b] Li A, et al. Generalized hindsight for reinforcement learning. NeurIPS 2020.\n\n[c] Packer C, et al. Hindsight task relabelling: Experience replay for sparse reward meta-rl. NeurIPS 2021.\n\n[d] Korbak T, et al. Pretraining language models with human preferences. ICML 2023.\n\n**W3: Additional analysis on the effectiveness of the proposed method.**\n- Thanks for your considerate suggestions! For additional analysis on the effectiveness of our method, please refer to our response to Reviewer fYHy W1.\n\n**Q1: More insights into the relative contributions of data diversity and the harmonization process.**\n- Code data from different sources may vary significantly in language style and content, including question types, code style, presence of comments, test cases, etc. Therefore, multi-source data mixing is a double-edged sword: it provides necessary diversity but can also bring large domain gaps. Adding concise corpus generated from the same Alchemist model (i.e., AlchemistPrompts with similar language styles) to a small amount of data can effectively bridge this gap while maintaining diversity.\n- In terms of balancing diversity and harmonization, for single-source data, the relatively lacking diversity is more important; for multi-source data, diversity is naturally introduced during mixing, so harmonizing conflicts becomes more crucial.\n\n**Q2&L1: Scalability & Data Complexity**\n- Thanks for your insightful advice! For scalability, due to limited computational resources, we focus on smaller models that are more popular and practical in the Code LLM field. For larger models, we will discuss this in our limitations and will open source our fine-tuning data to contribute to the Code LLM community, welcoming developers to apply it to larger models.\n- In terms of data complexity, we assume that an increase in data complexity implies the inclusion of more diverse data sources. We deconstruct this into a higher degree of (possibly excessive) diversity and more varied quality. In such a scenario, the harmonization effect introduced by our method becomes increasingly critical, which may imply a greater need for a higher proportion of AlchemistPrompts to achieve optimal performance. As for extreme cases, this essentially means fine-tuning towards generalist models rather than specialized ones (e.g., Code LLMs), where the bottleneck tends to be the balance of various capabilities, i.e., like a seesaw.\n\n**L2: Data Bias**\n- Thank you for emphasizing the aspect of data bias. Our AlchemistPrompts enhance the instruction-following abilities of models, which can potentially mitigate biases. For example, if the model has a bias towards responding with Python code when the programming language is not specified, the inclusion of programming language declarations in AlchemistPrompts helps to alleviate this bias. We have not delved into this yet and will discuss it in our limitations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iEBjSgAHAX", "reviewer": "Reviewer_fYHy", "summary": "This work improves upon past work developing code LLMs with a focus on intervention on the data used to instruction tune the models. Specifically, the key insight in this work is that past works have usually relied on single source data for fine-tuning, but this can come at a drawback of quality and diversity. To reduce these issues, the authors use multi-source data. This in turn leads to a challenge where the same question can elicit multiple responses, calling for the need for AlchemistPrompts that \"harmonize\" the sourced. A second source of gains for the AlchemistCoder models series comes from construction of code comprehension tasks. Various experiments are performed to demonstrate the method is very effective in achieving performance as good as models from a larger model size.", "questions": "1. Can you point me to this study: \"Our ablation study indicates that the optimal performance can be achieved by incorporating AlchemistPrompts into only 5% of all the samples, striking a balance between the diversity and domain gap resulting from the fusion of multi-source data.\"\n2. Is code comprehension data augmentation a novel contribution of this work?", "limitations": "The generation of AlchemistPrompts relies heavily on GPT-4", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. Clear problem identification: I liked that the authors identified a problem in existing literature training code LLMs pertaining to single source code data. They performed subsequent steps to mitigate this gap with single-source data, then harmonizing it.\n2. Strong results: The alchemistcoder model series punches much above models of the same parameter size, as evident from results in Table 1 and Figure 1.\n3. Wide range of evaluations: The authors test their model capabilities on both code generation tasks, and standard benchmarks, and the performance gains stay across the board.\n4. Analytical experiments on data composition: I liked the analysis in Figures 5, 6 and 8 giving more perspective to the reader on the differences between AlchemistCoder data and data from past work.", "weaknesses": "1. Missing ablations: Many of the comparisons are nor iso-compute (different models trained for different time). I believe an important aspect of this work is to clearly show model performance w.r.t. various ablations performed: 1. Multi-source 2. +Harmonization. 3. +Code comprehension.\n2. Python-based evals: Even though multi-source data is used, all the evaluations are limited to python based evaluation. I would have liked to see more analysis on how the multi-source data impacts model performance in other languages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Missing ablations.**\n- Thanks for your meticulous suggestions! We provide a new version of Table 1 in the global response PDF, which includes details of the training corpus used for fine-tuned Code LLMs.\n- Here, we reorganize the following table based on Table 4 to more clearly demonstrate the impact of multi-source data, harmonization, and code comprehension:\n  |**Method**|**HumanEval (Pass@1)**|**MBPP (Pass@1)**|\n  |-|:-:|:-:|\n  |Baseline (CodeLlama-Python-7B)|37.8|57.6|\n  |+ Multi-source data (w/o data decontamination)|54.6|57.9|\n  |+ Multi-source data (w data decontamination)|59.8|58.2|\n  |+ Harmonizations (AlchemistPrompts)|72.0|63.4|\n  |+ Code comprehension (Instruction Evolution)|71.3|65.8|\n  |+ Code comprehension (Data Filtering)|73.8|67.7|\n  |+ Code comprehension (Code Review)|**74.4**|**68.5**|\n- Due to inherent conflicts, the improvement brought by directly mixing multi-source data is limited (refer to the second/third row of the table above and DirectMix-L-7B in Figure 1). We believe that the key efficacy of our method lies in optimizing the data to possess the following characteristics conducive to effective model training:\n  - **Balance between diversity and source gap**: Multi-source mixing may bring necessary diversity but harmful domain gaps. Adding concise corpus generated from the same Alchemist model (i.e., AlchemistPrompts with similar language styles) can effectively bridge this gap.\n  - **Better alignment within instruction-response pairs**: AlchemistPrompts supplements instructions with details about programming languages, algorithm concepts, and code characteristics corresponding to the responses, representing an optimization of instruction/response alignment specific to coding capabilities. As shown on the right side of Figure 8, the instructions customized by AlchemistPrompts are closer to the responses in the feature space. Meanwhile, as shown on the left side of Figure 8, Conditional Perplexity Discrepancy (CPD) quantifies the difficulty change in generating responses before and after adding specific inputs (e.g., instructions) to the model (the smaller the value, the easier it becomes). The generally smaller CPD values after adding AlchemistPrompts reflect the facilitating effect of AlchemistPrompts on improving model performance.\n  - **Reflecting code comprehension**: Instruction evolution task data, which communicates the relationship between data before and after evolution, addressing a gap in existing work using instruction evolution data. Data filtering and code evaluation task data, provide the model with a deep understanding of both high-quality and low-quality code.\n- In addition, we have explored which data features are harmful to model training (refer to the fourth row of the table above and lines 162-171 of the manuscript):\n  - Responses that are overly concise and devoid of code. These answers usually provide straightforward replies to the instructions, overlooking both the code solution and explanatory annotations. Moreover, these instances often contain overly simplistic questions in the instructions.\n  - Code solutions that are either non-compilable or do not pass test cases (pertaining to particular samples).\n\n**W2: Python-based evals.**\n- Indeed, similar to existing work, we focus on Python due to its popularity and the wealth of benchmark resources available. We have provided results on multilingual code generation (refer to Table 2 in the manuscript, including C++, Go, Java, and JavaScript) and results on general benchmarks (refer to Table 5 in the manuscript, including MMLU for multitask language understanding, BBH for comprehensive reasoning, and GSM8K for mathematical ability). These extensive experimental results demonstrate that our AlchemistCoder series models deliver comprehensive code capabilities and reasoning abilities. \n- Additionally, each single source data itself contains multiple programming languages, not just one language per source, making it challenging to separately ablate the corpus for each language. Under the efficacy of multi-source integration, our fine-tuning data includes nearly 60% of various programming languages (refer to Figure 5 in the manuscript, including C, Java, HTML, SQL, etc.), aiming to contribute more universally powerful open-source models to the Code LLM community.\n\n**Q1: Clarification of the ablation study.**\n- Sorry if that statement was confusing. Code data from different sources may vary significantly in language style and content, including question types, code style, presence of comments, test cases, etc. Therefore, multi-source data mixing is a double-edged sword: it provides necessary diversity but can also bring large domain gaps. Adding concise corpus generated from the same Alchemist model (i.e., AlchemistPrompts with similar language styles) to a small amout of data can effectively bridge this gap while maintaining diversity. Besides, the inclusion of 5% AlchemistPrompts also represents a balance between cost and performance.\n\n**Q2: Is code comprehension data augmentation a novel contribution?**\n- Yes, we pioneer novel code comprehension tasks for fine-tuning Code LLMs. Meanwhile, we also aim to provide new insights into the construction of fine-tuning data: the data construction process itself reflects higher-level capabilities and can guide model training.\n\n**L1: The Reliance on GPT-4.**\n- Our method does indeed rely on Alchemist models (i.e., GPT-4), and we have discussed this in section A of limitations. Compared to methods that heavily rely on GPT-4 to generate entire new datasets, we strive to minimize our dependence on GPT-4 and only generate concise AlchemistPrompts for 5% of the data.\n- More importantly, this urges us to continue advancing toward our goal: exploring techniques to optimize large models and bridging the gap between open-source and closed-source models. And we will open source our fine-tuning data to contribute to the Code LLM community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HBITJ58hHg", "reviewer": "Reviewer_yLAq", "summary": "The paper presents a series of Code Large Language Models (LLMs) named AlchemistCoder, which are fine-tuned on multi-source data to enhance code generation and generalization capabilities. The authors address the limitations of previous Code LLMs that were typically fine-tuned on single-source data, which lacked diversity and quality, by introducing AlchemistPrompts, data-specific prompts generated through hindsight relabeling to harmonize different data sources and improve instruction-response pairs. Additionally, they incorporate the data construction process into fine-tuning as code comprehension tasks, including instruction evolution, data filtering, and code review. Extensive experiments demonstrate that AlchemistCoder outperforms models of the same size and rivals or surpasses larger models, showcasing its efficacy in refining instruction-following capabilities and advancing code intelligence.", "questions": "For different portions of AlchemistPrompts other than 5%, does the general increasing trend of accuracy on multi-source data in Table 4 maintained?", "limitations": "See weaknesses.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is clear and well-written.\n2. The contribution is simple, clear, and easy to be adopted. (Use GPT4-generated prompts to harmonize the domain gap)\n3. The evaluation justifies the effectiveness of the proposed method. (The ablation study explains the effectiveness of each step.)\n4. The improvement is impressive.", "weaknesses": "I'm not fully convinced that the improvement is from \"multi-source\" data. AlchemistPrompts possibly introduced high-quality data from GPT-4. Table 4 also confirmed that the most gain is from that 5% AlchemistPrompt.\nIf this is the reason, the comparison with other models weaker than GPT4 is not completely fair.\nIt also means the proposed method is weak in relying on a strong model to supervise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Concerns of AlchemistPrompts.**\n- Thanks for your insightful concerns! To fairly compare with other models, we provide a new version of Table 1 in the global response PDF, which includes details of training corpora sources for fine-tuned Code LLMs. Compared to methods that heavily rely on GPT-4 to generate entire new datasets, we strive to minimize our dependence on GPT-4 and only generate AlchemistPrompts for 5% of the data. **More importantly, existing Code LLM methods usually use strong models (e.g., GPT-4) to directly generate code for fine-tuning, whereas we do not. The AlchemistPrompts we generate are concise textual descriptions that do not include code (refer to Figures A4 and A5 in the manuscript), fundamentally differing from other methods in both their intended goals and practical effects.** To sum up, we do not rely directly on the code capabilities of strong models for supervision and our method achieves more effective optimization of fine-tuning data with greater diversity, higher quality, and lower cost for empowering AlchemistCoder to obtain promising and comprehensive code capabilities. \n- Additionally, we highlight the importance of combining AlchemistPrompts with multi-source data to achieve exceptional performance and we present more examples in Figure R1 of the global response PDF that demonstrate how our method harmonizes inherent conflicts in multi-source data. This is fundamentally different from fine-tuning on a single-source high-quality dataset. The synergy between AlchemistPrompts and multi-source data sets AlchemistCoder apart from other models and demonstrates a new path for enhancing code LLMs. Our contributions also lie in introducing new insights into improving prompts and designing instruction fine-tuning tasks to develop better open-source models: 1) The design philosophy and utility of AlchemistPrompts (please refer to our response to Reviewer q6UB W1); 2) The data construction process itself reflects higher-level capabilities and can guide model training.\n- Indeed, GPT-4 holds advantages in code and general capabilities and this urges us to continue advancing towards our goal: exploring techniques to optimize large models and bridging the gap between open-source and closed-source models. We will open source our fine-tuning data to contribute to the Code LLM community and explore the expansion of the proposed method to other LLM tasks. \n\n**Q1: For different portions of AlchemistPrompts other than 5%, does the general increasing trend of accuracy on multi-source data in Table 4 maintained?**\n- Yes, although 5% of AlchemistPrompts significantly contribute to performance improvement, other proportions (1–20%) of AlchemistPrompts also maintain a similar trend of enhancement. It is important to note that code data from different sources may vary greatly in language style and content, including question types, code style, presence of comments, test cases, etc. Therefore, mixing multi-source data has a double-edged sword effect: it provides necessary diversity but may also introduce significant domain gaps. To effectively bridge this gap while maintaining diversity, adding concise corpus generated by the same Alchemist model (i.e., AlchemistPrompts with similar language styles) to a small amount of data can effectively address this issue. Additionally, AlchemistPrompts are beneficial for refining the alignment within instruction-response pairs to enhance the instruction-following abilities of fine-tuned models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2DOGdtLBF5", "reviewer": "Reviewer_FM86", "summary": "This paper presents an improved pipeline for training LLMs for code generation. Their method incorporates prompt modifications using hindsight tuning to modify prompts to better align with the associated code. In addition, they introduce improved data filtering and additional training tasks which they find give moderate additional performance boosts. Models trained using their pipeline show better generation abilities than existing LLMs and code LLMs on existing benchmarks and retain better ability on natural language tasks", "questions": "1. Is the 5% of data modified by AlchemistPrompter chosen at random or targeted? If they were chosen randomly, was the experiment repeated? Finally, if they were chosen randomly, is there a chance that using a heuristic to choose them could improve performance further?\n\n2. Why should we expect AlchemistCoder models to be better generalists in NL tasks when the diversity they have seen is mostly in the form of code data? Though there is more diversity in this dataset, it is still domain specific, which makes the catastrophic forgetting hypothesis seem unlikely to me. Do you perform any further experiments on this?", "limitations": "The authors explain adequately.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The method is relatively simple (though somewhat costly) and appears to give performance boosts across tasks. Improving the performance of Code LLMs in ways beyond scaling up data is a valuable contribution to the community and this method has demonstrated utility. The authors compare to a variety of models and demonstrate the efficacy of their method on a variety of model sizes and families. Additionally, the authors give good ablation studies on what elements in their method contribute to this performance boost.", "weaknesses": "The main weaknesses of the paper come in lack of clarity. If these points can be clarified, I would be inclined to raise my score:\n\n1. The writing in this paper was not especially clear to me, particularly when it came to motivating introducing this method. When contrasting with prior work, it would help to give examples of the type of low code quality and diversity that has been observed. More concerningly, because these details were not clearly explained, it was not clear to me how this dataset improved upon prior work.\n\n2. The selection criteria are unclear for what samples that were chosen to be improved using AlchemistPrompts. If it is only a random 5% of the data that needs to be improved, it is unclear to me why performance benefits do not continue for higher percentages. On the other hand, if the data is selected more specially for this, it should be made clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Concerns of the presentation.**\n- Thank you for your valuable advice on improving our presentation! Our motivation stems from the following two considerations: 1) Existing Code LLM pre-train methods typically use multi-source data, while Code LLM fine-tuning methods focus on developing high-quality single-source datasets; 2) The current open-source Code LLM community already has various high-quality single-source fine-tuning data. To better utilize existing data resources and contribute to the Code LLM community, we are pioneering the integration of multi-source data for Code LLM fine-tuning.\n- Additionally, we have provided more examples in the global response PDF that demonstrate how our method harmonizes inherent conflicts in multi-source data. We will also refine our presentation in the latest version of the manuscript.\n\n**W2&Q1: Concerns about the selection of data customized by AlchemistPrompt.**\n- Sorry for the confusion. We calculate the Conditional Perplexity Discrepancy (CPD, refer to lines 248-252 of the manuscript) and **selectively chose data with higher CPD values for AlchemistPrompts harmonization**. Conditional perplexity discrepancy is an indicator of how data affects the complexity of model-generated responses under given conditions (i.e., instructions), and its calculation formula is $CPD = Perplexity(instruction + response) − Perplexity(response)$. The level of CPD reflects the impact of the conditional instruction on the complexity of the generated response. Specifically, a high CPD indicates that the perplexity of the generated response significantly increases under the presence of a conditional instruction, which usually reflects a poor alignment between the instruction and the response, the instruction may be unclear or not specific enough, or insufficient contextual information, thereby increasing the difficulty of model response generation. By analyzing high CPD values, we can identify cases where instructions and responses are poorly aligned and more effectively optimize data quality. As deeply analyzed in Figure 8 of the manuscript, AlchemistPrompts can effectively harmonize the discrepancy between instructions and responses.\n\n**Q2: Why should we expect AlchemistCoder models to be better generalists in NL tasks?**\n- Indeed, similar findings are also present in existing work in some other fields. Based on our insights, we attribute this phenomenon to the following three points:\n  - **Enhanced reasoning abilities**: Coding abilities themselves reflect reasoning capabilities. Additionally, multi-source integration significantly increases the diversity of fine-tuning data, which often includes detailed analyses, reasoning, and explanatory annotations, all of which contribute to improved model reasoning abilities.\n  - **Better instruction-following abilities**: AlchemistPrompts supplements instructions with details about programming languages, algorithm concepts, and code characteristics corresponding to the responses, representing an optimization of instruction/response alignment specific to coding capabilities. Thus, AlchemistPrompts can refine the alignment within instruction-response pairs and enhance the instruction-following abilities of fine-tuned models.\n  - **Improved context understanding**: As shown in Figure A4~A11 of the manuscript, both our AlchemistPrompts and code comprehension tasks can provide training for the context-understanding capabilities of models.\n- In summary, our fine-tuning data includes a substantial amount of natural language descriptions in addition to code snippets, leading to the improvements from fine-tuning that are not limited to coding abilities. The enhancements in the three abilities mentioned above also benefit tasks MMLU for multitasking language understanding, BBH for comprehensive reasoning, and GSM8K for mathematical reasoning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear all, 


We appreciate the reviewers for valuable feedback remarking our work has "simple" method (**Reviewer yLAq & FM86**) and "strong" efficacy with "impressive improvements" (**Reviewer q6UB & fYHy & yLAq & FM86**), provides "wide range of evaluations, detailed ablation studies and analyses" (**Reviewer q6UB & fYHy & yLAq & FM86**), "identifies and addresses a important challenge of utilizing multi-source data for Code LLM fine-tuning" (**Reviewer q6UB & fYHy**), and is "clear and well-written" (**Reviewer fYHy & yLAq**). We have responded to all the concerns point by point and additional details mentioned in the response will be **synced** to the revised version of our manuscript.

The core innovation of our AlchemistCoder lies in proposing an effective framework for integrating multi-source data for Code LLM fine-tuning to overcome the limitations in quality and diversity inherent within a single-source dataset. **This is a non-trivial paradigm in the field of Code LLM fine-tuning and we are the first to unveil inherent conflicts in multi-source code corpora**. To resolve this challenge, we innovatively design data-specific AlchemistPrompts, inspired by hindsight relabeling. Additionally, we make the first effort to integrate the data construction process as code comprehension tasks into the training process. These key concepts facilitate the **enhancement of the diversity, quality, and cost-effectiveness** of our fine-tuning data, thereby enabling the development of the AlchemistCoder series models with significantly enhanced and comprehensive coding capabilities. Our contributions can be summarized as:
- **AlchemistPrompts**: Designed as data-specific prompts for harmonizing inherent conflicts in multi-source data and mitigating instruction/response misalignment at a fine-grained level.
- **Code Comprehension Tasks**: Sourced from data construction process, consisting of instruction evolution, data filtering, and code review.
- **Harmonized Multi-source Data**: Instruction tuned on 200M tokens, including 6 types of high-quality data.
- **Superior Model Performance**: Surpassing all the open-source models of the same size (6.7/7B), and rivaling or even beating larger models (15B/33B/70B/ChatGPT) on 6 code benchmarks.
- **Advanced Generic Capabilities**: Demonstrated by the significant improvements on MMLU, BBH, and GSM8K.

Once again, we really appreciate the supportive feedback and strongly believe that these reviews have strengthened the work.  


Sincerely,

Authors of Paper #10234

Meta-review: The paper presents a series of Code LLMs called AlchemistCoder with an emphasis on multi-source data. In order to accommodate data sources with different styles, the authors design data-specific prompts with highsight relabeling. The empirical performance is strong despite that the authors focus on the models at around 7B parameters. The reviewers had questions about the 5% generated data but they were addressed by the authors.
Decision: Accept (poster)