AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data

Authors: zifan  song
yudong  wang
wenwei  zhang
kuikun  liu
chengqi  lyu
demin  song
qipeng  guo
hang  yan
dahua  lin
kai  chen
cairong  zhao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 15:07:15

Abstract: Open-source Large Language Models (LLMs) and their specialized variants, particularly Code LLMs, have recently delivered impressive performance. However, previous Code LLMs are typically fine-tuned on single-source data with limited quality and diversity, which may insufficiently elicit the potential of pre-trained Code LLMs. In this paper, we present AlchemistCoder, a series of Code LLMs with enhanced code generation and generalization capabilities fine-tuned on multi-source data. To achieve this, we pioneer to unveil inherent conflicts among the various styles and qualities in multi-source code corpora and introduce data-specific prompts with hindsight relabeling, termed AlchemistPrompts, to harmonize different data sources and instruction-response pairs. Additionally, we propose incorporating the data construction process into the fine-tuning data as code comprehension tasks, including instruction evolution, data filtering, and code review. Extensive experiments demonstrate that AlchemistCoder holds a clear lead among all models of the same size (6.7B/7B) and rivals or even surpasses larger models (15B/33B/70B), showcasing the efficacy of our method in refining instruction-following capabilities and advancing the boundaries of code intelligence. Source code and models are available at https://github.com/InternLM/AlchemistCoder.

Main paper: 
 claims: 
 Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The abstract and introduction clearly outline the primary contributions and scope of the paper. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
method: 
 To more comprehensively elicit the capability of the base LLMs, we first construct multi-source data for fine-tuning ( § 2.1), which is harmonized by AlchemistPrompts to take effect( § 2.2). Code comprehension tasks are also constructed to further improve the performance( § 2.3). We also discuss the details and statistics of the filtered and harmonized multi-source data in § 2.4.To fully elicit the capability of code LLMs, we first collect the fine-tuning data from multiple sources (Fig. 2(a) ) and adopt the instruction evolution [30] to improve the complexity of the instructions (Fig. 2 (b)). However, integrating multi-source data for instruction tuning poses challenges. Naturally, one code-related question can be solved by different coding languages with various algorithms or response styles (e.g., with or without reasoning). When naively combing data curated by different developers with different LLMs, the model tends to learn to answer similar questions with different coding languages and response styles, as depicted in Fig. 3 . On the one hand, learning diverse responses may elicit different capability aspects of the base models. On the other hand, since the learned responses to similar instructions often diverge due to implicit human intentions, the LLMs tend to be unaligned (to our expectation) after the fine-tuning on the directly mixed data (e.g., we cannot expect which coding language the LLMs will use in real-world applications), resulting in inferior performance. Therefore, directly mixing multi-source data is not a promising solution and can be detrimental.Here are {{ sample number }} sets of instructions-response paired coding samples. To harmonize the inherent conflicts within multi-source data, we propose to customize data-specific prompts called AlchemistPrompts, (Fig. 2(c )), inspired by the concept of hindsight [1, 48] . Specifically, we employ GPT-4 [34] to play the role of an Alchemist and design the prompt as illustrated in Fig. 4 to obtain Al-chemistPrompts. For instance, for an instruction of 'Write code to find the shortest path from one vertex to all other vertices in a graph', if the response involves Python code of a Bellman-Ford algorithm with dynamic programming, we would expect to customize the instruction with an AlchemistPrompt of 'Please generate Python code for the following task and attempt to use the concept of Dynamic Programming'.For the selection of data customized by AlchemistPrompts, we calculate the differences in perplexities of generating responses with/without given instructions, called Conditional Perplexity Discrepancy (CPD). Then, we selectively chose data with higher CPD values for AlchemistPrompt harmonizations. We treat CPD as an indicator of how data affects the complexity of model-generated responses under given conditions (i.e., instructions), and its calculation formula is CPD = Perplexity(conditional_instruction + response) -Perplexity(response). The level of CPD reflects the impact of the conditional instruction on the complexity of the generated response. Specifically, a high CPD indicates that the perplexity of the generated response significantly increases under the presence of a conditional instruction, which usually reflects a poor alignment between the instruction and the response, the instruction may be unclear or not specific enough, or insufficient contextual information, thereby increasing the difficulty of model response generation. By analyzing high CPD values, we can identify cases where instructions and responses are poorly aligned and more effectively optimize data quality.The adjustments to data made by AlchemistPrompts are relatively minor and well-calibrated. Our ablation study indicates that the optimal performance can be achieved by incorporating AlchemistPrompts into only 5% samples, striking a balance between the diversity and domain gap resulting from the fusion of multi-source data. Crucially, by retrospectively analyzing responses and reinterpreting them as alternate goals, the AlchemistPrompts serve to elevate the condition/goal of data. This hindsight integration [1, 48, 26] allows for a more nuanced and adaptive learning process, enhancing not only the models' comprehension of data but also refining instruction-following capabilities.The existing training datasets for Code LLMs [24, 4, 39, 30, 44] primarily focus on the code generation task consisting of programming problems and their corresponding code solutions. However, we contend that beyond this, the process of constructing code data demonstrates higher-level abilities. Consequently, we devise three code comprehension tasks relevant to data construction, including instruction evolution, data filtering, and code review (Fig. 2(d) ).Instruction evolution. Inspired by the concept of instruction evolution [45, 30] , we employ GPT-3.5 [33] to construct instruction evolution task data, which entails augmenting the requirements for instructions and providing detailed explanations for programming tasks. Integrating the instruction evolution task aids the model in discerning the disparities before and after evolutions, thereby deepening the comprehension of programming requirements, code complexity, task decomposition, and other code-related concepts.Data filtering. We identify four categories of low-quality data from multiple sources: (a) responses that are excessively short and lack code, (b) code that fails to compile, (c) code with poor clarity, and (d) code that does not adhere to the requirement in the instruction regarding its organization in function form. Each instruction in the data filtering task presents the model with a low-quality sample and prompts the model to classify it into one of the four categories. The data filtering task entails recycling the filtered-out data by offering counterexamples, thereby assisting the model in generating fewer low-quality responses.Code review. In this task, we require the model to review a piece of code and assign scores between 0 and 10 for correctness and clarity separately. Additionally, the model is expected to provide suggestions for code improvement and present the refined code. To obtain higher-quality data, we utilize GPT-4 [34] to generate code reviews and select cases that are more representative, particularly those with average correctness and clarity scores exceeding 8 or falling below 6. Simultaneously, we focus on instances where one aspect exhibits severe deficiencies, i.e., the score of correctness or clarity is equal to or below 4.In practice, we have established a set of filtering rules to enhance our data cleaning and purification procedures. These rules involve excluding samples based on various criteria, such as response length (either too short or too long), absence of code or insufficient code content, non-compilable code, code failing test cases (pertinent to certain samples), responses structured in notebook form, and instances with excessive textual descriptions preceding the code. After conducting an extensive series of validation experiments, we conclusively decide to eliminate low-quality data meeting either of the following conditions: (a) responses that are excessively brief and lack code. Such responses typically offer direct answers to the instructions, neglecting both the code solution and explanatory annotations.Additionally, these samples frequently present overly simplistic questions in the instructions; (b) code solutions that are non-compilable or fail test cases (relevant to specific samples).Concurrently, following [13] , we employ N-gram similarity, cosine distance of code embeddings, and edit distance of code syntax trees to calculate the similarity between training data and samples in HumanEval and MBPP. We subsequently discard samples through this process of data filtering and deduplication, resulting in the removal of approximately 6% of the dataset.Our AlchemistCoder dataset (∼200M tokens) comprises four types of multi-source data, encompassing open-source datasets and three types of data constructed by us. Specifically, (a) opensource datasets including Evol-Instruct-Code-80k-v1 [10] , CodeExercise-Python-27k [9] , and evolcodealpaca-v1 [39], (b) EvolCode data generated from gpt-3.5-turbo following [30] , (c) data customized by AlchemistPrompts, and (d) data of the code comprehension tasks (i.e., instruction evolution, data filtering, and code review).We visualize the distributions of data sources and programming languages using two circular graphs in Fig. 5 . Concurrently, Fig. 6 reports a distribution of text description lengths and code lines. Compared to CodeAlpaca [4] and OOS-INSTRUCT [44] , our AlchemistCoder dataset presents a notably diverse distribution and maintains moderate overall text description and code lengths, benefiting significantly from the integration of multi-source data along with AlchemistPrompts and code comprehension tasks. This is instrumental in contributing to a comprehensive evolution of code capability. Our dataset contains high-quality samples with more diverse distributions.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We have discussed the limitations of our work in Appendix §A. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experiments: 
 In this section, we report results on various benchmarks of code generation and conduct ablation experiments. Furthermore, we present analytical studies to provide a more in-depth demonstration of the efficacy of our AlchemistCoder.Benchmarks. We adopt six code benchmarks: HumanEval [5] , HumanEval+ [27] , HumanEval-X [49] , MBPP [2] , MBPP+ [27] , and DS-1000 [21] . In addition, we access three mainstream benchmarks (MMLU [15] , BBH [37] , and GSM8K [8] ) to evaluate generalization abilities. All evaluation and benchmark details can be found in Appendix §D.Baselines. We compare with the following competitive baselines. Closed-Source Models: GPT-3.5-Turbo [33] and GPT-4-Turbo [34] . Open-Source Models: Llama 2 [40] , CodeLlama [35] , StarCoder [22] , WizardCoder [30] , DeepSeek-Coder [14] , and Magicoder [44] .Supervised fine-tuning. We adopt Llama-2-7B, CodeLlama-Python-7B, and DeepSeek-Coder-Base-6.7B as the base models and fine-tune all the base models for 2 epochs using 32 NVIDIA A100-80GB GPUs. We set the initial learning rate, minimum learning rate, and optimizer warmup steps and at 1e-4, 6e-6, and 15, respectively. We use Adam optimizer [28] and choose a batch size of 2 with a sequence length of 8192.Results on python code generation. We first access HumanEval and MBPP to evaluate the capability of the AlchemistCoder series for Python code generation. These benchmarks necessitate models to generate code based on the function definitions and subsequently pass the test cases. Models are evaluated in zero-shot on HumanEval and 3-shot on MBPP. The comprehensive comparisons in Tab. 1 and Fig. 1 demonstrate the impressive capabilities of AlchemistCoder models. From the results, AlchemistCoder-L attains a remarkable performance boost of 42.7% and 28.4% pass@1 scores on HumanEval and MBPP respectively, compared to Llama 2-7B. Notably, AlchemistCoder-DS elevates the pass@1 scores to 79.9% and 77.0% on these benchmarks, holding an overall improvement of 33.3%. Moreover, our AlchemistCoder series with 7B parameters outperforms larger models (e.g., Table 1 : Results of pass@1 on HumanEval (HumanEval+) and MBPP (MBPP+) benchmarks. We report the results of HumanEval and MBPP consistently from the EvalPlus [27] and the bold scores denote the best performance among models of the same size. WizardCoder-CL-34B and CodeLlama-Instruct-70B) and rivals with GPT-3.5-Turbo, significantly bridging the performance gap between closed-source and open-source models.Results on multilingual code generation. We compare the pass@1 accuracy of the base models and the corresponding fine-tuned AlchemistCoder models on Humaneval-X [49] . The results presented in Tab. 2 demonstrate that the AlchemistCoder series exhibits great improvements (exceeding 50%) for multilingual code generation, delivering comprehensive code capabilities.Results on code generation for data science. We further conduct the evaluation of data science code completion on DS-1000 [21] . According to Tab. AlchemistCoder-CL achieves an astonishing overall accuracy of 40.3% with relatively better performance in all libraries, demonstrating powerful capabilities in data science workflows.The Recipe of AlchemistPrompts. As illustrated in Sec. 2.2, AlchemistPrompts can further align the instructions and responses of data samples and harmonize the domain gap between multiple sources. Code data from different sources may vary significantly in language style and content, including question types, code style, presence of comments, test cases, etc. Therefore, multi-source data mixing is a double-edged sword: it provides necessary diversity but can also bring large domain gaps. Adding concise corpus generated from the same Alchemist model (i.e., AlchemistPrompts with similar language styles) to a small amout of data can effectively bridge this gap while maintaining diversity. To find the appropriate recipe of AlchemistPrompts that maintains a balance between data diversity and domain gap, we conduct ablation experiments on the proportion (0% to 20%) of data customized by AlchemistPrompts. We adopt two settings: (a) augment the original data with its customized variant and report the results of fine-tuning for 2 epochs on CodeLlama-Python-7B; (b) replace the original data and report the results of fine-tuning for the same steps (i.e., keeping the number of tokens used consistent). As shown in Fig. 7 , AlchemistCoder is particularly enhanced when the proportion of customized data increases from 1% to 5%, and nearly peaks in performance at 5%. Thus, we introduce AlchemistPrompts into 5% of the training set to balance the performance gain and the generation cost. Additionally, both two strategies effectively enhance the performance and validate the efficacy of our approach. To push the limit of AlchemistCoder, we employ the augmentation strategy in our performance experiments. In addition, we present detailed experimental results from the multi-source integration and harmonization process in our Appendix §C to offer a more in-depth demonstration of the AlchemistPrompts efficacy as data complexity scales.Efficacy of the code comprehension tasks. We conduct an ablation study on the key components of the code comprehension tasks to ascertain their individual contributions to the overall performance. As reported in Tab. 4, compared to the baselines (the first and second rows), the model demonstrates enhanced performance on both benchmarks following the incremental incorporation of code comprehension task data. Notably, the improvement (5.1% regard to the pass@1 metric) is particularly remarkable on MBPP. This indicates the significant contribution of all code comprehension tasks to furthering programming capabilities.AlchemistPrompts harmonize the discrepancy between instructions and responses. To in-depth verify the efficacy of AlchemistPrompts, we calculate the Conditional Perplexity Discrepancy (CPD, refer to Sec. 2.2) values of our fine-tuning data harmonized by AlchemistPrompts, i.e., the difference between Perplexity(conditional_instruction + response) and Perplexity(response). The CPD value quantifies the difficulty change in generating responses before and after adding specific inputs (e.g., instructions) to the model (the smaller the value, the easier it becomes). Specifically, we adopt the instructions before and after customization by AlchemistPrompts for comparison, and provide the Kernel Density Estimation of CPD in Fig. 8 . Clearly, the latter (green) gains smaller overall CPD values, indicating that AlchemistPrompts are beneficial for prediction and can provide effective contextual information. Furthermore, we randomly select 10 groups of these samples and use UMAP [31] to map their feature representations into a 2-D space in the right of Fig. 8 . From the fact that the solid lines are generally shorter than the dashed lines, our AlchemistPrompts can harmonize the discrepancy between instructions and responses, leading to higher-quality data for attaining improved instruction-following ability.AlchemistCoder models are better generalists. To further analyze the comprehensive capabilities of our AlchemistCoder, we conduct evaluations on more diversified benchmarks, including MMLU [15] for multitask language understanding, BBH [37] for comprehensive reasoning, and GSM8K [8] for mathematical ability. The results are presented in Tab. 5 and illustrate that the AlchemistCoder models exhibit an overall performance increase of 6.4%, 13.6%, and 14.5% over the base models Llama 2, CodeLlama-Python, and DeepSeek-Coder-Base, respectively. Notably, CodeLlama-Python presents inferior performance on these benchmarks relative to Llama 2, indicating the discrepancy between natural language processing and code capabilities of open-source models. Such divergence can be attributed to "catastrophic forgetting" [11, 29, 20] , often occurring when fine-tuning is exclusively concentrated on domain-specific data. By leveraging harmonized multi-source data, our AlchemistCoder series models achieve enhanced reasoning abilities, better instruction-following abilities, and improved context understanding, which contribute to develop better generalists.Error case analysis. To meticulously dissect the improvements brought by our method, we provide an analysis of error cases on HumanEval and MBPP. We compare models before and after the introduction of AlchemistPrompts and code understanding task data. The bar chart shown in Fig. 9 (top) indicates that these two types of key data help to better handle compilation errors (i.e., SyntaxError, NameError, and ValueError), and eliminate the occurrence of no code written in the responses. On the other hand, the results of Fig. 9 (bottom) on MBPP suggest that the AlchemistCoder series incorporated with these two types of data attains stronger programming logic, as evidenced by the clear reduction in the 'Wrong Answer' error cases. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We have provided code with sufficient instructions in our supplemental material. We have released our code, data, and AlchemistCoder series models at https://internlm. github.io/AlchemistCoder. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
related work: 
 Code large language models. Early researches [5, 32, 22] focus on collecting massive amounts of code data to develop pretrained Code LLMs. Recent efforts [30, 47, 44] are dedicated to finetuning these pretrained models with specific instructional data to further the coding abilities. For instance, WizardCoder [30] and Magicoder [44] construct their instruction tuning datasets based on CodeAlpaca [4] and the stack [18] dataset, respectively. In this work, we develop the AlchemistCoder series by instruction tuning on optimized multi-source data instead of single-category data as in previous methods, endowing astonishing and harmonized code capability.Instruction tuning. Instruction tuning aims to enhance LLMs via fine-tuning pre-trained LLMs using samples of instruction/response pairs. Obtaining high-quality data for instruction tuning is typically challenging and extensive works have been dedicated to this endeavor. For instance, Alpaca [38] employs self-instruct [42] to generate instruction-following demonstrations. WizardLM [45] introduces Evol-Instruct and transforms the instruction data into more complex variants. In addition to Evol-Instruct, we also incorporate the data construction process itself as a form of data into the training. Moreover, although previous works [16, 43, 41, 23] utilize multiple fine-tuning datasets, we harmonize multi-source data at a fine-grained level.Learning from hindsight. The concept of learning from hindsight [26] has been explored in goal-conditioned learning [17, 12] . Hindsight Experience Replay (HER) [1] is designed to re-label rewards and facilitate learning from sparse feedback. Korbak et al. [19] study the influence of human preferences during pre-training, showing improved performance when models are aligned with human preferences. Previous work primarily serves as an alternative to RLFT, utilizing HER to leverage (suboptimal) historical data for model learning. We focus on harmonizing the inherent conflicts within multi-source data through hindsight, to fully tap into the potential of base models. 
conclusion: 
 In this paper, we propose an effective framework for integrating multi-source data to fine-tune Code LLMs, addressing the limitations in quality and diversity inherent within a single-source dataset. This is a non-trivial paradigm and we pioneer to unveil inherent conflicts in multi-source code corpora. To resolve this challenge, we innovatively design data-specific AlchemistPrompts, inspired by hindsight relabeling. Additionally, we make the first effort of integrating the data construction process as code comprehension tasks into the training process. These key concepts enhance the diversity, quality, and cost-effectiveness of code fine-tuning data, facilitating the development of the AlchemistCoder series models with significantly improved and comprehensive coding capabilities. Table A2 : Ablation study on the efficacy of multi-source integration and AlchemistPrompt harmonizations, evaluated on the HumanEval (Pass@1) and MBPP (Pass@1) benchmarks. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: The safeguards have been discussed in Appendix §B.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We use publicly available datasets, benchmarks, and models for training and evaluation, free from any possible harm toward individuals or groups.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: The paper introduces new models that are well-documented. Documentation includes details about the dataset, code, and model, such as training procedures, licenses, limitations, and consent from data sources. We have released our code, data, and Alchemist-Coder series models at https://internlm.github.io/AlchemistCoder.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
33.3% improvement: 
 Inferior performance from directly mixing multi-source data Closed-source Large Language Models (LLMs) like ChatGPT and GPT-4 [33, 34] exhibit impressive code intelligence by learning on largescale and diverse code corpus, which also benefits many other applications, such as math reasoning [6] , embodied control [25] , and agent [46] . Since open-source LLMs [40] still lag behind closed-source LLMs [34] in this field, there has been growing interest in investigating the acquisition of code capabilities by developing specialized Code LLMs [35, 14] .The training of Code LLMs mainly goes through pre-training and fine-tuning stages [35] . Pioneer . Then, we adopt AlchemistPrompt to harmonize their inherent conflicts (c) and construct code comprehension data (d). We use a mix of these data to fine-tune various pre-trained LLMs to obtain our AlchemistCoder models.works [5, 32, 22, 36, 3] have amassed extensive code data for pre-training, while recent open-source models [30, 44] highight the effectiveness of high-quality or targeted code fine-tuning datasets. Despite these advancements, current fine-tuning methods mainly rely on a specific type of coderelated question-answering dataset, unlike the pre-training stage that integrates code-related corpus from various sources [35] . Such a discrepancy indicates that the fine-tuning data may lack the necessary diversity to fully stimulate the capabilities of base models, resulting in limited performance, generalization, and robustness.To overcome the limitations in quality and diversity within single-source data, we pioneer to explore integrating multi-source data for Code LLM fine-tuning. However, this is a non-trivial paradigm and blindly integrating multi-source data can potentially lead to inferior performance (e.g., the DirectlyMix-L-7B model in Fig. 1 ). To track this, we unveil inherent conflicts in multi-source code corpora, including conflicting code language requirements and response styles. Inspired by hindsight relabeling [1, 48] , we propose to design data-specific prompts to harmonize the inherent conflicts for multi-source data integration, better eliciting the performance of base models. We term this form of prompts as AlchemistPrompts, inspired by the power and definition of Alchemists:"Alchemist: Someone Who Transforms Things for the Better." --Merriam Webster Specifically, we first integrate several open-source code datasets and conduct instruction evolution [30] based on some of them (Fig. 2(a, b )). As shown in Fig. 2 (c), for instruction-response pairs of different sources, we adopt one LLM to generate AlchemistPrompts that accurately and explicitly describe the characteristics as requirements of the response to enrich the instructions. In-depth, the efficacy of AlchemistPrompts is twofold: 1) Harmonization between different data sources: AlchemistPrompts generated from the same LLM have similar styles and can bridge the style differences between sources, while the introduction of AlchemistPrompt-customized data, accounting for only 5%, achieves a balance between data diversity and domain gaps; 2) Harmonization within instruction-response pairs: As fine-grained and data-specific prompts, AlchemistPrompts are designed to augment instructions with specific programming languages, algorithm concepts, and other coderelated information involved in responses, which can refine the alignment within instruction-response pairs and enhance the instruction-following abilities of fine-tuned models.Apart from the conventional problem-solution data, we argue that the progression of code data (e.g., data evolution, cleaning, and quality evaluation) reflects higher-level capabilities and offers valuable insights for the enhancement of Code LLMs. Consequently, we delineate the construction of data into three integral tasks for training: instruction evolution, data filtering, and code review (see Fig. 2 (d)), facilitating enhanced code comprehension capabilities.We conduct extensive experiments with various base models [40, 35, 14] and develop the instructiontuned AlchemistCoder series. As shown in Fig. 1 , on two mainstream code benchmarks, HumanEvalInstruction: Merge the elements of two given arrays into one, but ensure that the merged array contains only the common elements between the two arrays. ## Input: [1, 2, 3, 4] and [3, 4, 5, 6] ## Output: [3, 4] Response (a): Here is an example implementation in JavaScript:function mergeArrays(array1, array2) { let mergedArray = [];for (let i = 0; i < array1.length; i++) { if (array2.includes(array1[i])) { mergedArray.push(array1[i]); } } return mergedArray; } const array1 = [1, 2, 3, 4] ; const array2 = [3, 4, 5, 6] ; const mergedArray = mergeArrays(array1, array2); console.log(mergedArray); // Output: [3, 4] ``B y using this algorithm, you can merge two arrays and obtain a new array containing only the common elements between them.Change the color of the text to blue. ## Input: Text: The quick brown foxTo change the color of the text to blue, you can use CSS. Here's an example of how you can achieve this: By applying the "blue-text" class to the paragraph element, the text will be displayed in blue color.Missing Language Declaration. The instruction does not specify the programming language, while the response assumes HTML/CSS and provides an implementation accordingly.Assume we are working with a HTML document and have the permissions to modify the CSS styles associated with it. Change the color of the text to blue. ## Input: Text: The quick brown fox <style> .blue-text { color: blue; } </style> <p class="blue-text">The quick brown fox</p> Before you begin, please make sure you know how to create a function in JavaScript and use the 'includes' method to check if an array contains a specific element.Find the minimum of two given numbers. ## Input: 4, 6 ## Output: The minimum number is 4*Notebook Format:* **Reasoning:** To find the minimum of two given numbers, we can use an if-else statement. We will compare the two numbers, assign the smaller value to a variable called "minimum_number", and print out it. **Code:** **Execution Results:** The minimum number is 4Format Exception. The response introduces a notebook format not specified by the instruction.Complete the simple task in notebook format, including reasoning, code, and execution results. Find the minimum of two given numbers. ## Input: 4, 6 ## Output: The minimum number is 4 num1, num2 = 4, 6 if num1 < num2: minimum_number = num1 else: minimum_number = num2 print("The minimum number is", minimum_number)Replace all words with 5 letters or more that start with a vowel, and end with a consonant, with "*****" in the given text. "The quick brown fox jumps over the lazy dog"Sure! Here is the text with the specified replacements: "The quick brown fox jumps ***** the lazy dog"Here's a Python function that achieves this: Response Format Discrepancy. For similar instructions, Source #1 provides the result directly, while Source #2 provides the implementation code, which may lead to confusion in understanding the instructions. Figure 3 : Examples of inherent conflicts (e.g., various styles and quality) within multi-source code corpora. By applying AlchemistPrompt-customized instructions that are more consistent with the responses, the diversity from multiple sources can be effectively managed and utilized, thereby improving the quality of our fine-tuning data and the instruction-following capabilities of the finetuned models.and MBPP, AlchemistCoder holds a clear lead among all models of equivalent size (6.7B/7B), and rivals or even surpasses larger models (15B/33B/70B), demonstrating harmonized and formidable code capabilities. Furthermore, we delve into the effectiveness of AlchemistPrompts and discern that they alleviate the misalignment between instructions and responses within the data. Remarkably, AlchemistPrompts allow the code corpus to also significantly improve the general capability of Code LLMs, as demonstrated by the improvements on MMLU, BBH, and GSM8K. Our main contributions are summarized as follows:• Our work pioneers to integrate multi-source data for Code LLM fine-tuning to overcome the limitations of quality and diversity inherent in single-source data. • We unveil inherent conflicts within multi-source code corpora and introduce Alchemist-Prompts, revealing the power of hindsight tuning for code generation, aiming to harmonize the conflicts among sources and bridge the alignment within instruction-response pairs. • We propose to incorporate data construction process into the fine-tuning data and design code comprehension tasks, including instruction evolution, data filtering, and code review, endowing comprehensive code capabilities. • Extensive ablation and analytical studies confirm the efficacy of our key concepts for enhancing the diversity, quality, and cost-effectiveness of Code LLM fine-tuning data. Through instruction tuning on various base models, we develop the AlchemistCoder series, surpassing all Code LLMs of the same size on a wide spectrum of code benchmarks. 
 
 Appendix: Baseline (Llama2-7B) HumanEval [5] and HumanEval+ [27] are benchmarks for assessing LLMs' code generation, focusing on functional correctness. HumanEval+ expands on HumanEval by significantly increasing test cases through EvalPlus, using LLM and mutation strategies for more rigorous evaluation. This approach reveals performance drops in models like GPT-4 and ChatGPT against challenging tests, emphasizing the need for diverse test scenarios to accurately evaluate LLMs' coding abilities. For evaluation on Hu-manEval and HumanEval+, we adopt the prompt designed for HumanEval/HumanEval+ tasks shown in Fig. A1 . Fol-lowing prior works [49, 7, 44] , we use the greedy decoding strategy and focus on comparing the pass@1 metric.The MBPP (Mostly Basic Python Programming) benchmark [2] consists of around 1,000 Python challenges, crowd-sourced to test basic programming skills, including fundamentals and standard library use. Aimed at beginners, each challenge offers a description, solution, and three tests for verifying solution accuracy. MBPP+ [27] is an extension of the MBPP benchmark, utilizing a subset of hand-verified problems from MBPP-sanitized to ensure tasks are well-defined and unambiguous.For the evaluation on MBPP and MBPP+, we adopt the three-shot prompt shown in Fig. A2 .HumanEval-X [49] is a comprehensive benchmark that assesses the capabilities of code generation models across multiple programming languages, including Python, C++, Java, JavaScript, and Go. It consists of 820 meticulously created data samples, each accompanied by test cases, making it an invaluable resource for evaluating and improving multilingual code generation models. The benchmark aims to provide insights into the models' proficiency in solving diverse coding challenges and their accuracy in generating functionally correct code in different languages. For evaluation on HumanEval-X, we do not use specific prompts and follow the original test prompts.The DS-1000 benchmark [21] adapts 1000 different data science coding problems each with unit tests from StackOverflow and checks both execution semantics and surface-form constraints. These realistic problems are drawn from seven popular data science libraries in Python, including Matplotlib (plt), NumPy (np), Pandas (pd), SciPy (scp), Scikit-Learn (sk), PyTorch (py), and TensorFlow (tf). DS-1000 has two modes: completion and insertion, and here we only evaluate completion, as the basic CodeLlama-Python does not support insertion. For evaluation on DS-1000, we do not use specific prompts and follow the original test prompts.The Massive Multitask Language Understanding (MMLU) benchmark [15] is an evaluation framework designed to measure the depth and breadth of knowledge that LLMs possess. It accomplishes this by testing these models across 57 varied tasks in both zero-shot and few-shot scenarios. The tasks encompass a wide array of topics, including basic math, American history, computer science, law, and more, challenging the models to leverage their acquired knowledge to solve complex problems. MMLU seeks to emulate the multifaceted way in which human knowledge and problem-solving skills are assessed, offering a comprehensive gauge of a model's ability to understand and apply information across multiple domains. For evaluation on MMLU, we do not use specific prompts and follow the original test prompts.The BIG-Bench Hard (BBH) Benchmark [37] is a specialized evaluation framework tailored to rigorously test the capabilities of LLMs. This benchmark targets a selection of tasks that have historically proven challenging for LLMs, focusing on areas where models typically do not exceed average human performance. The BBH Benchmark aims to push the boundaries of what LLMs can achieve by emphasizing complex reasoning, deep understanding, and nuanced interpretation, setting a high bar for model development and performance evaluation. For evaluation on BBH, we do not use specific prompts and follow the original test prompts.The GSM8K (Grade School Math 8,000) benchmark [8] serves as a rigorous evaluation framework for testing the mathematical problem-solving prowess of LLMs. This benchmark comprises a dataset of 8,500 diverse and high-quality math word problems at the grade school level, designed to challenge 3. DO NOT respond with content that is outside of the specified output format.{{ code from multi-source data}} LLMs with tasks necessitating advanced, multi-step reasoning abilities. GSM8K's primary aim is to gauge how well these models can parse, understand, and solve math problems, thereby offering a comprehensive measure of their capacity for logical reasoning and mathematical computation. By incorporating such a specialized benchmark, researchers can better understand the extent to which LLMs can mimic human-like reasoning in solving complex mathematical scenarios. For evaluation on GSM8K, we do not use specific prompts and follow the original test prompts.We provide two samples of AlchemistPrompts in Fig. A4 and Fig. A5 .For instruction evolution task data, we provide two samples in Fig. A6 and Fig. A7 . For data filtering task data, we provide two samples in Fig. A8 and Fig. A9 . For code review task data, we design prompt as illustrated in Fig. A3 to obtain high-quality code review task data and we provide two samples in Fig. A10 and Fig. A11 .Please increase the difficulty of the given programming test question a bit. You can increase the difficulty using, but not limited to, the following methods: If the original problem can be solved with only a few logical steps, please add more reasoning steps.Construct an SQL query to add a new column called "Gender" to the table "Employees" with values "Male" and "Female", but also make sure that the values for the "Gender" column are randomly assigned for each employee.Construct an SQL query to add a new column called "Gender" to the table "Employees" with values "Male" and "Female", but also make sure that the values for the "Gender" column are randomly assigned for each employee. Additionally, ensure that the ratio of male to female employees is approximately 1:1.Please increase the difficu You can increase the diffic If the original problem c more reasoning steps.Iterate over a list and cou contain duplicates and no of each item. The input lisIterate over a list and cou contain duplicates and n handle nested lists and d iterate through the neste item. Return a dictionary optimizing the code for pe of 10,000. st question a bit. e following methods: ogical steps, please add "Gender" to the table ake sure that the values mployee."Gender" to the table ake sure that the values employee. Additionally, oximately 1:1.Iterate over a list and count the number of occurrences of each item. The list may contain duplicates and non-integer elements. However, now you need to also handle nested lists and dictionaries within the main list. You should recursively iterate through the nested data structures and count the occurrences of each item. Return a dictionary with the count of each item. Additionally, consider optimizing the code for performance, as the input list can have a maximum length of 10,000.Please increase the difficulty of the given programming test question a bit. You can increase the difficulty using, but not limited to, the following methods: If the original problem can be solved with only a few logical steps, please add more reasoning steps.Iterate over a list and count the number of occurrences of each item. The list may contain duplicates and non-integer elements. Return a dictionary with the count of each item. The input list can have a maximum length of 1000. Construct an SQL query to add a new column called "Gender" to the table "Employees" with values "Male" and "Female", but also make sure that the values for the "Gender" column are randomly assigned for each employee. Additionally, ensure that the ratio of male to female employees is approximately 1:1.Iterate over a list and contain duplicates a handle nested lists a iterate through the item. Return a dicti optimizing the code f of 10,000.Below is a code task and its corresponding solution. Please utilize your understanding of code to carefully review this solution.Before we proceed, let's briefly talk about Python list indexing. Python uses zero-based indexing, which means the first element of a list is at index 0. The list 'a' here is a mutable sequence of integers, and we are going to modify it using a for loop. Consider a list `a = [1,3,2,2,3,4]`. What would be the output of the following code?The correct output of the given code would be: [3, 1, 3, 2, 2, 3] The above solution is considered to be of *low quality*. Choose from the following four options what category this solution belongs to: a. The solution is too short (less than 20 words) and does not contain code. b. There are code compilation errors in the solution. c. The clarity of the code in the solution is too poor. d. The solution contains multiple sections of code and is not organized into modularized functions.Your choice (answer only the choice without the reason):a = [1,3,2,2,3,4]for i in range(len(a)):a[i] = a[i -1]a. The solution is too short (less than 20 words) and does not contain code.Below is a code ta understanding of iterate through the nested data structures and count the occurrences of each item. Return a dictionary with the count of each item. Additionally, consider optimizing the code for performance, as the input list can have a maximum length of 10,000. Below is a code task and its corresponding solution. Please utilize your understanding of code to carefully review this solution.Write a function that takes as input a tuple of numbers (t_1,...,t_{N+1}) and returns a tuple of length N where the i-th element of the tuple is equal to t_i * t_{i+1}. The function should also handle negative numbers and zeros, and return an error message if the input is not a tuple.Here is a function implementation in Python:The above solution is considered to be of *low quality*. Choose from the following four options what category this solution belongs to: a. The solution is too short (less than 20 words) and does not contain code. b. There are code compilation errors in the solution. c. The clarity of the code in the solution is too poor. d. The solution contains multiple sections of code and is not organized into modularized functions.def tuple_product(input_tuple):if not isinstance(input_tuple, tuple): return "Error: Input is not a tuple" return tuple(input_tuple[i] * input_tuple[i + 1] for i in range(len(input_tuple) -1)) print(tuple_product((1, 2, 3, 4, 5)) # Outputs: (2, 6, 12, 20) print(tuple_product((-1, 2, -3, 4, -5))) # Outputs: (-2, -6, -12, -20) b. There are code compilation errors in the solution. [Correct] 9. The code is correct. It correctly processes the values according to the specified rules and returns the expected JSON string.[Clarity] 8. The code is clear and easy to understand. The function names are descriptive and the code follows a logical flow.[Suggestions] -It would be helpful to include type annotations for the function arguments and return values. This would make the code more clear and easier to understand. The prompt for generating data is omitted.The error in the code is an incorrect indentation when printing the indentation spaces.The number of spaces printed should be based on the line number, but the code mistakenly indents by a fixed number of spaces. This leads to misalignment and missing characters in some lines. Additionally, there are missing asterisks and periods in the inner loop, which causes the alternating pattern to be incorrect. By fixing the indentation and adding the missing characters, the tower of stars will be displayed correctly. [Correct] 3. The code has syntax and there are logic errors that cause the tower of stars to be displayed incorrectly.[Clarity] 8. The code uses meaningful variable names and is easy to understand.[Suggestions] -The code could benefit from adding type annotations to the function and its arguments.-The logic errors could be fixed by adjusting the indentation and adding the missing characters in the inner loop.In the refined code, the indentation has been fixed and the missing characters have been added. Additionally, type annotations have been added to the function and its arguments to improve clarity. def print_star_tower() -> None:for i in range (1, 11) : print(' ' * i, end='') for j in range(1, i*3): if j % 2 == 0: print('.', end='') else:print('*', end='') print()  


Decision: The paper presents a series of Code LLMs called AlchemistCoder with an emphasis on multi-source data. In order to accommodate data sources with different styles, the authors design data-specific prompts with highsight relabeling. The empirical performance is strong despite that the authors focus on the models at around 7B parameters. The reviewers had questions about the 5% generated data but they were addressed by the authors.

Reviews: review: paper_summary: paper_summary: This paper introduces AlchemistCoder, a series of code language models fine-tuned on multi-source data. The authors propose using "AlchemistPrompts" to harmonize inherent conflicts in multi-source code corpora and incorporate code comprehension tasks into the training process. The resulting models show improved performance on various code generation benchmarks compared to baseline models of similar size.
strength_weakness: strength_weakness: 1. The core concept of AlchemistPrompts lacks substantial novelty. It appears to be a variation of existing instruction evolution techniques, such as the Evol-Instruct.
2. While the performance improvements are notable, the overall approach of using multi-source data and harmonizing it is not fundamentally new in the field of language model fine-tuning.
3. The paper compares AlchemistCoder models (6.7B/7B parameters) with larger models, including 15B, 33B, and 70B models. While the results show that AlchemistCoder outperforms or rivals these larger models on several benchmarks (as seen in Table 1 and Figure 1), the paper does not provide a comprehensive analysis of the factors contributing to this performance gain. It would be valuable to have a more in-depth discussion on how much of this improvement can be attributed to the proposed method versus other potential factors such as the efficiency of smaller models or the quality of the base model used. This additional analysis would strengthen the paper's claims about the effectiveness of the AlchemistCoder approach relative to simply scaling up model size.
questions: questions: 1. Can you provide more insights into the relative contributions of data diversity and the harmonization process to the overall performance improvement? Are there scenarios where one factor might be more important than the other?
2. How does the efficiency and effectiveness of the AlchemistPrompt approach scale with increasing model size and dataset complexity? Are there any computational or performance bottlenecks that might limit its applicability to very large language models or extremely diverse multi-source datasets?
limitations: limitations: The authors have addressed some limitations of their work, such as the potential for hallucination in response to obscure queries. However, there are several other important limitations that could be more thoroughly discussed:
1. Scalability: The paper focuses on models with 6.7B/7B parameters. There's limited discussion on how the AlchemistPrompt approach and code comprehension tasks would scale to much larger models or more diverse datasets. The computational costs and potential performance bottlenecks for scaling up are not addressed.
2. Data Bias: The process of selecting and harmonizing multi-source data could potentially introduce or amplify biases present in the original datasets. The paper doesn't thoroughly address how these biases are identified, mitigated, or might impact the model's outputs.
paper_summary: paper_summary: This work improves upon past work developing code LLMs with a focus on intervention on the data used to instruction tune the models. Specifically, the key insight in this work is that past works have usually relied on single source data for fine-tuning, but this can come at a drawback of quality and diversity. To reduce these issues, the authors use multi-source data. This in turn leads to a challenge where the same question can elicit multiple responses, calling for the need for AlchemistPrompts that "harmonize" the sourced. A second source of gains for the AlchemistCoder models series comes from construction of code comprehension tasks. Various experiments are performed to demonstrate the method is very effective in achieving performance as good as models from a larger model size.
strength_weakness: strength_weakness: 1. Missing ablations: Many of the comparisons are nor iso-compute (different models trained for different time). I believe an important aspect of this work is to clearly show model performance w.r.t. various ablations performed: 1. Multi-source 2. +Harmonization. 3. +Code comprehension.
2. Python-based evals: Even though multi-source data is used, all the evaluations are limited to python based evaluation. I would have liked to see more analysis on how the multi-source data impacts model performance in other languages.
questions: questions: 1. Can you point me to this study: "Our ablation study indicates that the optimal performance can be achieved by incorporating AlchemistPrompts into only 5% of all the samples, striking a balance between the diversity and domain gap resulting from the fusion of multi-source data."
2. Is code comprehension data augmentation a novel contribution of this work?
limitations: limitations: The generation of AlchemistPrompts relies heavily on GPT-4
paper_summary: paper_summary: The paper presents a series of Code Large Language Models (LLMs) named AlchemistCoder, which are fine-tuned on multi-source data to enhance code generation and generalization capabilities. The authors address the limitations of previous Code LLMs that were typically fine-tuned on single-source data, which lacked diversity and quality, by introducing AlchemistPrompts, data-specific prompts generated through hindsight relabeling to harmonize different data sources and improve instruction-response pairs. Additionally, they incorporate the data construction process into fine-tuning as code comprehension tasks, including instruction evolution, data filtering, and code review. Extensive experiments demonstrate that AlchemistCoder outperforms models of the same size and rivals or surpasses larger models, showcasing its efficacy in refining instruction-following capabilities and advancing code intelligence.
strength_weakness: strength_weakness: I'm not fully convinced that the improvement is from "multi-source" data. AlchemistPrompts possibly introduced high-quality data from GPT-4. Table 4 also confirmed that the most gain is from that 5% AlchemistPrompt.
If this is the reason, the comparison with other models weaker than GPT4 is not completely fair.
It also means the proposed method is weak in relying on a strong model to supervise.
questions: questions: For different portions of AlchemistPrompts other than 5%, does the general increasing trend of accuracy on multi-source data in Table 4 maintained?
limitations: limitations: See weaknesses.
paper_summary: paper_summary: This paper presents an improved pipeline for training LLMs for code generation. Their method incorporates prompt modifications using hindsight tuning to modify prompts to better align with the associated code. In addition, they introduce improved data filtering and additional training tasks which they find give moderate additional performance boosts. Models trained using their pipeline show better generation abilities than existing LLMs and code LLMs on existing benchmarks and retain better ability on natural language tasks
strength_weakness: strength_weakness: The main weaknesses of the paper come in lack of clarity. If these points can be clarified, I would be inclined to raise my score:

1. The writing in this paper was not especially clear to me, particularly when it came to motivating introducing this method. When contrasting with prior work, it would help to give examples of the type of low code quality and diversity that has been observed. More concerningly, because these details were not clearly explained, it was not clear to me how this dataset improved upon prior work.

2. The selection criteria are unclear for what samples that were chosen to be improved using AlchemistPrompts. If it is only a random 5% of the data that needs to be improved, it is unclear to me why performance benefits do not continue for higher percentages. On the other hand, if the data is selected more specially for this, it should be made clearer.
questions: questions: 1. Is the 5% of data modified by AlchemistPrompter chosen at random or targeted? If they were chosen randomly, was the experiment repeated? Finally, if they were chosen randomly, is there a chance that using a heuristic to choose them could improve performance further?

2. Why should we expect AlchemistCoder models to be better generalists in NL tasks when the diversity they have seen is mostly in the form of code data? Though there is more diversity in this dataset, it is still domain specific, which makes the catastrophic forgetting hypothesis seem unlikely to me. Do you perform any further experiments on this?
limitations: limitations: The authors explain adequately.

Comments: title: Thank you!
Thanks for the detailed response
Please let us know if your concerns have been addressed
Please let us know if your concerns have been addressed
Please let us know if your concerns have been addressed
Thanks for the response
Gentle Reminder
comment: Thank you for raising the rating! Your feedback is invaluable, and we are delighted to make revisions to our paper based on your insights.
Sorry for the late reply. I am willing to raise my score to 6. The comprehensive analysis of data complexity impacts and multi-source integration addresses my concerns.
Dear Reviewer q6UB,

We wish to express our gratitude for your extensive review and supportive feedback. Your feedback of minor questions about data complexity impacts is invaluable to us, and **we are fully committed to thoughtfully incorporating your insights to enhance our paper**. As the discussion phase is nearing its end, we are warmly concerned whether our rebuttal addresses your concerns.

**It would be appreciated if you could raise your score on our paper if we address your concerns**. We thank you again for your effort in reviewing our paper.

Best regards,

Authors of Paper #10234
Dear Reviewer yLAq,

We wish to express our gratitude for your thorough review and positive feedback. Likewise, **we are warmly concerned whether our rebuttal addresses your concerns**. Your feedback is invaluable to us, and we are fully committed to thoughtfully incorporating your insights to enhance our paper.

Once again, thank you for your ongoing support during this review process!

Sincerely,

Authors of Paper #10234
Dear Reviewer fYHy,

We wish to express our gratitude for your extensive review and positive feedback. Likewise, **we are warmly concerned whether our rebuttal addresses your concerns**. Your feedback is invaluable to us, and we are fully committed to thoughtfully incorporating your insights to enhance our paper.

Once again, thank you for your ongoing support during this review process!

Sincerely,

Authors of Paper #10234
Thank you for increasing your score! Your insights have been incredibly helpful, and we are excited to incorporate the changes based on your suggestions into our paper.

Thanks again for your support and valuable input!
Thank you for raising the score and recognizing our response for clarifying your concerns!

Your question about data complexity impacts is insightful and we believe a discussion around this would improve our paper. Data complexity is a fundamental issue in the domain of large models, affecting not only model training and performance but also data processing strategies, model generalization, and resource utilization. Specifically:
- **Data Complexity and Multi-Source Integration**: Our research demonstrates that integrating data from multiple sources significantly increases data complexity and diversity, as evidenced by the broader distributions of code and description lengths shown in Figure 6 of the manuscript. While this integration facilitates the model's ability to learn richer feature representations, it also heightens the demands on the model to manage inputs of varying styles, formats, and quality. AlchemistCoder addresses this challenge by introducing AlchemistPrompts, which help conduct harmonizations across various data sources and within instruction-response pairs. To offer a more in-depth analysis of the AlchemistPrompt efficacy as data complexity scales, we present detailed experimental results from the multi-source integration and harmonization process:

  | **Method**                                               | **HumanEval (Pass@1)** | **MBPP (Pass@1)** |
  |----------------------------------------------------------|:----------------------:|:-----------------:|
  | Baseline   (Llama2-7B)                                   |          14.0          |        26.1       |
  | One-source data   fine-tuning (w data decontamination)   |          18.3          |        29.0       |
  | + Harmonizations   (AlchemistPrompts)                    |       22.6 (4.3 $\uparrow$)       |     30.2 (1.2 $\uparrow$)    |
  | Two-source data   fine-tuning (w data decontamination)   |          35.4          |        30.6       |
  | + Harmonizations   (AlchemistPrompts)                    |       39.0 (3.6 $\uparrow$)       |     32.8 (2.2 $\uparrow$)    |
  | Three-source data   fine-tuning (w data decontamination) |          37.8          |        35.4       |
  | + Harmonizations   (AlchemistPrompts)                    |       43.9 (6.1 $\uparrow$)       |     40.8 (5.4 $\uparrow$)    |
  | Four-source data   fine-tuning (w data decontamination)  |          40.2          |        42.2       |
  | + Harmonizations   (AlchemistPrompts)                    |       55.1 (**14.9** $\uparrow$)      |     49.4 (**7.2** $\uparrow$)    |
  | + Code   comprehensions (i.e., AlchemistCoder-L-7B)      |     **56.7** (1.6 $\uparrow$)     |   **54.5** (5.1 $\uparrow$)  |

- **Data Cleaning and Decontamination**: In the face of complex data, preprocessing steps become crucial. Data cleaning and decontamination can remove noise and irrelevant information, helping the model focus on learning meaningful patterns. More specifically, we have explored which data features are harmful to model training (refer to the fourth row of the table in response to Reviewer fYHy W1 and lines 162-171 of the manuscript):
  - Responses that are overly concise and devoid of code. These answers typically provide straightforward replies to instructions, neglecting both the code solution and explanatory annotations. Additionally, these instances often contain overly simplistic questions in the instructions.
  - Code solutions that are either non-compilable or do not pass test cases (pertaining to specific samples).
- **Model Generalization**: Data complexity also affects the generalization abilities of models. The models need to be adequately trained on complex data to perform well on unseen data. AlchemistCoder series models maintain enhanced code generation and generalization capabilities through multi-source data fine-tuning, thereby improving performance across a range of tasks.
- **Cost and Efficiency Trade-Off**: Processing complex datasets may require more computational resources and time, creating a trade-off between cost and model performance. We strive to achieve significant performance improvements with only generating concise AlchemistPrompts for 5% of the data, thereby finding a balance between cost and efficiency.
Apologies for my late reply. Thank you for the clarification regarding the motivation. The example given in your response PDF of the possible mismatches was helpful, as was the point about how you chose the samples with AlchemistPrompt. Given this, and assuming that these changes are incorporated into the paper, I'm willing to increase my score.
Thanks for the effort. I am willing to raise my score to 5 because the rebuttal effectively clarifies the unique aspects of AlchemistPrompts, distinguishing them from existing instruction evolution techniques. The authors' explanation of the different goals, methods, and practical effects is convincing. While I still have some minor questions about data complexity impacts, the overall approach seems promising for improving Code LLM fine-tuning with multi-source data.
Dear ACs and Reviewers,

We wish to express our gratitude for your endeavors to participate in the review process. Our utmost efforts have been dedicated to resolving the concerns articulated by all reviewers. We sincerely hope that you have had a chance to review our response to your comments. We kindly request that you get back to us as soon as possible since the discussion period is ending soon. We appreciate your time and effort in reviewing our work, and we are happy to provide any further information if required.

Thank you for your ongoing support during this review process!

Sincerely,

Authors of Paper #10234