WizardCoder: Empowering Code Large Language Models with Evol-Instruct
Authors: Ziyang Luo
Can Xu
Pu Zhao
Qingfeng Sun
Xiubo Geng
Wenxiang Hu
Chongyang Tao
Jing Ma
Qingwei Lin
Daxin Jiang
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Code
Instruction Fine-tuning
Primary area: generative models
TL;DR: We present Code Evol-Instruct, enhancing Code LLMs to create novel models WizardCoder.

Abstract: Code Large Language Models (Code LLMs), such as StarCoder, have demonstrated remarkable performance in various code-related tasks. However, different from their counterparts in the general language modeling field, the technique of instruction fine-tuning remains relatively under-researched in this domain. In this paper, we present Code Evol-Instruct, a novel approach that adapts the Evol-Instruct method to the realm of code, enhancing Code LLMs to create novel models, WizardCoder. Through comprehensive experiments on five prominent code generation benchmarks, namely HumanEval, HumanEval+, MBPP, DS-1000, and MultiPL-E, our models showcase outstanding performance. They consistently outperform all other open-source Code LLMs by a significant margin. Remarkably, WizardCoder 15B even surpasses the well-known closed-source LLMs, including Anthropic's Claude and Google's Bard, on the HumanEval and HumanEval+ benchmarks. Additionally, WizardCoder 34B not only achieves a HumanEval score comparable to GPT3.5 (ChatGPT) but also surpasses it on the HumanEval+ benchmark. Furthermore, our preliminary exploration highlights the pivotal role of instruction complexity in achieving exceptional coding performance.

Published as a conference paper at ICLR 2024
WizardCoder: EMPOWERING CODE LARGE LAN- GUAGE MODELS WITH EVOL-INSTRUC
Ziyang Luo2*
Can Xu
Pu Zhao'
Qingfeng Sun'
Xiubo Geng',
Wenxiang Hu'
Chongyang Tao
Jing Ma²
Qingwei Linè
Daxin Jiang'!
1 Microsooft
Hong Kong Baptist University
(cszyluo, uo, ma jing) camp) edu..kuu hk, {caxu, wwwhaaeemiicosoossseee com {gins, xigeng, wenxh, chongyang. tao, qlin
ABSTRACT
Code Large Language Models (Code LLMs), such as StarCoder, have demonstrated remarkable performance in various code-related tasks. However, different from their counterparts the general language modeling field, the technique instruc- tion fine-tuning remains relatively under-researched in this domain. In this paper, we present Code Evoo-Instruct, novel approach that adapts the method to the realm of code, enhancing Code LLMs to create novel models Wiz- ardCoder. Through comprehensive experiments on five prominent code generation benchmarks, namely HumanEval, HumanEval+, MBPP, DS- and MultiPL-E, our models showcase outstanding performance. They consistently outperform all other open-source Code LLMs by significant margin. Remarkably, WizardCoder 15B even surpasses the well-known closed-source L.Mss, including Anthropic's Claude and Google's Bard, the HumanEval and HumanEval+ benchmarks. Additionally, WizardCoder 34B not only achieves a HumanEval score compara- ble GPT3.5 (ChatGPT) but also surpasses on the HumanEval++ benchmark. Furthermore, our preliminary exploration highlights the pivotal role of instruction complexity achieving exceptional coding performance.
INTRODUCTION
Recently, Large Language Models (LLMs) (Brown et al. 22200 OpenAl 2023 Chowdhery et al.
2022
Anil et al. 2023 Hoffmann al., 2022 Rae al., |021| Zeng et al.J 2022] Zhang al.
2022 Touvron et al. 2023a) have garnered immense attention and demonstrated impressive success.
Notably, OpenAl's GPT3.5 (ChatGPT) stands out as a prominent example. These models, through extensive pre-training vast internet data and fine-tuning with detailed instruction data (Ouyang 2022). have achieved tfte-of-thhe-rr (SOTA) zero-shot performance across diverse NLP tasks. This trend also extends the realm of code understanding and generation, where multitude of
ct al.
Code LLMs have emerged (Chen et al. 202 1 a. Li al. 2022: Fried et al. 2022 Nijkamp et al.
2023b Zheng al. 2023 Wang al. 202 2023 Li al. 2023a; Nijkamp et al. 2023a Rozière
ct al. 2023). These models, pre-trained on substantial code data, excel in various code-related tasks,
consistently delivering impressive performance.
In contrast to most previous Code LLMs that primarily focus on the pre-training process, there has been limited exploration of fine-grained instruction tuning in the code domain. The introduction of instruction tuning was initially designed to enhance the generalization capabilities of LMs across different tasks via multitask training (Raffel et al., (2000] Wei
2022) Chung et al. 2022 Aribandi et al. 2022 Sanh et al., 2022 Xu et al. 2022, Khashabi al. (020). OpenAl's InstructGPT (Ouyang et al. 2022), for instance, involved soliciting human annotators to provide explicit instructions to ensure alignment with users' intentions. Similarly, recent works such Alpaca TTooi et al. (023) employed the self-instruct (Wang et al. 2022) method, where GPT3.5 (ChatGPT) generated the
Equal contribution. Work done during the internship at Microsoft.
Corresponding author.

---

Published as a conference paper at ICLR 2024
Dovelop Pyston progront that oreates randon pasdword
Develop a Python program that cruedee random lengh
knogh c2aaaaccees, indluding o leset one uppercase lettrr one digg, specia
Bo 0gAARR-8
Coda
Rosgonas
Resporoe
thot does thic
syfton import rendom
program genenies Pytncil builkie randon
do
mocull::
pyttonn
cal peneratee passwwrddengght
Unwe
Inport elring
saing "5200bt0'""
pontacee
chore string sedi Jedess srinn digle
rancom oe bating eaci Jowaranee)
stinng pussuotion
pontacee cancom chos perinng rights
neggsdegghi)
retur password
Inuureetc Wetheds
passsoont sindomuchis onj ielpooseent)
in nangaslengh 4]
pessword generste (ea neveppneeatt
Easy
Tisk
original proalem,, eaklnn apo memannyy 10 aitiooaaal words
comnenslly hos commion and more apecilic one.
rotum password
randomn Bipposoooer Juat
pessiorr jortpesavent, (uss)
Harder Tesk
paeewass i geownne, parsword ?) pritipasanen
WizardCoder
Open-Source SOTA
60
30
20
Python
Javascript
PHА
Julla
Switt
Figure 1: An illustration of our novel Code Evol-Instruct and the superior pass @ performance of our WizardCoder 34B, outperforming the open-sonee source SOTA (CodeLlama- lama-34B------- 34B--eries as of the date before August 24, 2023) by large margin in 9 different programming languages. The Python score is the mean between HumanEyal and MBPP.
instruction data. Vicuna (Chiang et al. (023) utilized user-shared conversations collected from ShareGPT.ccmm WizardLM (uu et al. (023) introduces the Evol-Instruct method, which involves evolving existing general instruction data to generate more complex and diverse datasets. Drawing inspiration from these previous works in the general domain, work, Code Evol-Instruct, is specifically tailored to the coding domain's 's distinctive characteristics.
In this study, we aim to enhance the capabilities of the SOTA open-source Code LLMs (i.e., StarCoder and CodeLlama), by introducing our novel Code Evol-Instrin The motivation of this fine- aneinnaieed instruction- -tuning method in the code domain is to automatically increase the complexity of code instruction data, so as to make the best of the internal coding ability of the Code LLMs. Our Code Evol-Instruct incorporates several novel methods, including heuristics tailored to coding task features, adversarial sample heuristics, time/space complexity requirements, and evolving stop controls. The whole process includes two steps: initially, we apply our Code Evol-Instruct to evolve basic code instruction data, specifically Code Alpaca (Chaudhary 2023). Subsequently, we fine-tune StarCoder and CodeLlama using our newly generated code instruction-follie followiig training set, resulting in our WizardCoder models.
Figure T and the experimental results obtained from five code generation benchmarks, namely EvaanEval (Chen et al. 20216). HumanEyal+ (Liu et al. 2023), MBPP (Austin et al. 2021), DS- 100 (Lai et al. 2022). and MultiPi.-E (Cassano et al.) 2022), demonstrate that our WizordCoder models outperform all other open-source Code LLMs (before August 24, 2023), achieving state-of-the-arr (SOTA) performance. Remarkably, our WizardCoder 15B even surpasses well-known Anthropic's Claude and Google's Bard in terms of pass rates HumanEval and HumanEyal+. Furthermore, WizardCoder 34B not only achieves a HumanEval score comparable to GPT3.5 (ChatGPT) but also surpasses it on the HumanEyal+ benchmark. Beyond this, our preliminary studies indicate that the complexity of instructions is the key to achieving exceptional coding performance.
The contributions of this work be summarized follows:
2

---

can Published as a conference paper at ICLR 2024
We introduce Code Evol-Instruct, novel instruction fine-tuning approach for code, which enhances the performance of the open-source Code LLMs by a large margin.
We develop WizardCoder models, which surpass all other open-source Code LLMs by a substantial margin in coding tasks. Notably, the 15B version even outperforms the well-known closed-source LLMs, such as Claude, and Bard. The 34B version achieves a HumanEval score comparable to GPT3.5 (ChatGPT) and surpasses it on the HumanEval+ benchmark.
We conduct a preliminary study highlighting the pivotal role of instruction complexity in achieving exceptional coding performance.
RELATED WORK
Large Language Models. Recently, LLMS have demonstrated remarkable achievements across
broad spectrum of tasks. Prominent tech companies have made significant strides in developing highly
proficient LLMss. These include OpenAI's GPT3&4 (Brown al., 2020 OpenAT| 2023). Google's
PaLM (Chowhhery et al. 2022 Anil et al. 2023 and Bard DeepMind's Chinchilla (Hoffmann
al., 2022). and Gopher [Rae et al. 2021), as well Anthropic's 's Claude However, is important
to note that these models are closed-soures and can only be accessed through specific APIs or may
not be accessible at all.
The AI community has witnessed the release of several open-source LLMs, where the model weights are made publicly available. FleutherAl has contributed GPT-NeoX-20RB (Black et al. 2022) and GPT-1-6B (Wang & Komatsuzaki] (2021) Google has released U1.2-20B (Tay et al. 2022) Tsinghua
University has introduced GLM- 130B (Zeng et al. 2022). Meta has released OPT (Zhang et al. 2022) and LLaMAI&2 (Touvron et al. 2023a b). It is worth noting that while these open-source models have made valuable contributions, they generally do not exhibit the same level performance as their slusec-source counterparts.
Large Language Models for Code. Recent studies have introduced a significant number of LLMs for code-related tasks to address the challenges of code understanding and generation. OpenAl has (Microsoft, 2023). Google has proposed PaLM-Coder (Chowdhery et 2022). They perform outstandingly the popular code completion benchmarks, like Evaa (Chen et al. 2021b) and MBPP (Austin et al. 2021). However, these models are closed-source.
unveiled Codex (Chen et al.] 2021a) and Code-Davine
On the other hand, there are several open-source Code LLMs available. Salesforce has introduced CodeGen1&2 (Nijkamp et al.] 2023b ap. CodeT5 (Wang et al. (02I). and CodeT5+ (Wang et al.)
2023). Tsinghua University has contributed CodeGeeX (Zheng et al. (203)) and the BigCode Project
(Rozière et al. 2023). which achieves open-source SOTA performance on several benchmarks. The closely related model, CodeLllam--Istrruc,, refines its performance through the self-instruct method. These models have demonstrated notable advancements in code-related tasks. However, when compared to the SOTA closed-source models, they still lag behind significantly. In contrast to the aforementioned models, our work demonstrates that further training Code LLMs with our Code substantially enhance performance.
has developed StarCoder (Li et al. 2023a Meta has released the Codel.lama-Sert
Instruction Tunnng The primary objective of instruction fine-tuning in its early stages was to enhance the cross-task generalization capabilities of LMs. This was achieved by fine-tuning LMs with substantial corpus of public NLP tasks. T5 (Raffel al. 2020) was among the first models to explore this approach, training on a multitude of supervised text-to-text tasks. Subsequent al.) 2022). TO (Sanh al.) 2022). and UnifiedQA [Kashabi et al. 2020) further expanded the range of tasks to bolster the overall generalization ability of LMs. Notably, ZeroPrompt (Xu et al. (022) and FLAN-T5 (Chung et al. 2022) pushed the envelope by incorporating thousands of tasks in their training pipelines. Across these studies, a consistent finding emerges: fine-tuning LMs with diverse NLP task instructions yields significant performance improvements when applied to new tasks.
works such as FLAN (Wei et al.] 2022), ExT5 (Aribandi
https://barr..coqgee.coo
http:://ww.anchhooicc.ooo
3

---

Published as a conference paper at ICLR 2024
While fine-tuning LMs with diverse NLP tasks has shown promising results. it often falls short in aligning with the intentions of real-world users. OpenAI has pursued a different approach by soliciting human annotators provide a large corpus of human instructions, encompassing diverse forms and a wide range of task types. Building upon this dataset, OpenAl trained its GPT3 (Brown et al. 2020) model to create InstructGPT (Ouyang al. (022), which better aligns with users' inputs. This line of development has even led to the impressive work known as GPT3.5 (ChatGPT), However, is important to note that the dataset and model weights associated with these advancements are not publicly available. Alpaca (Taori et al. 2023) takes different route by adopting the self- instruct method (Wang et al., 2022). leveraging GPT3.5 (ChatGPT) to generate data for training. Vicuna (Chiang et al. 2023) utilizes user-shared conversations collected from ShareGPT.com its models. WizardI.M .M (Xu al. 2023) introduces the Evol-Instruci method, which involves evolving existing general instruction data to generate more complex and diverse datasets. Drawing inspiration from this idea, our work, Code Evol-Ina Instrustt aligning with the distinctive characteristics coding domains, is the first instruction fine-tuning method explicitly designed to enhance Code LLMs.
train
WIZARDCODER: SOTA OPEN-SOURCE CODE LLM
In this section, elaborate on the methodological details of WizardCoder. As illustrated in Figure] !] we first adopt our Code Evol-Instruct to iteratively evolve the Code Alpaca dataset. Subsequently, we fine-tune the pre-trained Code LLMs with the evolved data.
3.1 CODE EVOL-INSTRUCT
Inspired by the Evol-Instruct method proposed by WizardLM Xu et al. (2023). this work attempts to automatically enhance the complexity of code instructions, thereby improving the fine-tuming effectiveness of Code L.MMs. Diverging from the general domain, our methods are meticulously designed to align with the specific characteristics of coding domains. The evolutionary process introduces the following features:
1. Heuristics aligned with coding task features on platforms like LeetCode, strategically increasing the complexity of coding tasks enhance the model's capabilities.
2. Introduction of erroneous code as an adversarial sample, inspired by prior research on attacking pre-trained code models Yang (2022); Jha & Reddy (2022). adds novel and effective method to escalate task complexity.
Introduction of a heuristic emphasizing time and space complexity leverages insights from previous studies Madaan et al. (2023), providing valuable avenue for improving task complexity.
So, the code evolutionary prompt template is as follows:
Prompt for Code Evol- Instructt
Please increase the difficulty of the given programming test question a bit.
You can increase the difficulty using, but not limited to, the following methods: {method)
{question]
Here, {question) represents the current code instruction awaiting evolution, and {method} the type of evolution. The five types used are listed as follows:
4

---

Published as a conference paper at ICLR 2024
Code Evolution Heuristie Methods
Add new constraints and requirements to the original problem, adding approximately 10 additional words.
Replace commonly used requirement in the programming task with a less com- mon and more specific one.
If the original problem can be solved with only few logical steps, please add more reasoning steps.
Provide a piece erroneous code as a reference to increase misdirection.
Propose higher time or space complexity requirements, but please refrain from do- ing so frequently.
3.2 TRAINING WizardCoder
We employ the following procedure train WizandCoder. Initially, we utilize StarCoder 15B (Li et al. 2)23a) and -ada-33lm--34B--PPPa (Rozière et al. (023)) as the foundations and proceed to fine-tune them using the code enstruction-follwwer training set, which was evolved through Code Evol-Instruct. The prompt format for fine-tuning is outlined as follows:
Prompt for Fine- TuninTunig Format
Below is an instruction that describes a task, paired with an input that provides further context. Write response that appropriately completes the request.
### Instruction:
(instruction]
### Response:
To construct the training dataset, we initialized it with the instruction-folloiing dataset called Code Alpaca We iteratively employ the Code Evol-Instruct technique on this dataset consisting around 20k samples produce evolved data. After each round of data evolution, merge the evolved data from all previous rounds with the original dataset to finetune Code LLMs. An external dev set serves the controlled Evol Stop. If the performance drops, we halt the evolution. Appendix C outline the approach employed to prevent data leakage. Additionally, Appendix D showcases some evolved examples for reference.
4 EXPERIMENT
This section begins by providing a comprehensive overview of the baseline models in our experiments. Subsequently, we present the performance of our models five code generation benchmarks:
HumanEval CChnn et al. 2021b). HumanEval+ (Liu et al. 2023). MBPP (Austin et al., 2021). DS-1000 (Lai
2022) and MultiPL-E (Cassano al. 2022).
4.1 BASELINES
Closed-Source Models. Multiple technology companies have successfully developed highly profi- publicly release them. These models referred to as closed-source
cient LLMs while choosing models. For our research, we incorporate a substantial number these models as our baselines.
Specifically, our baselines encompass the following: (1) OpenAl's
( Opee
nAI 2023), Code-Davinci-002 (Microsofi] 2023) Code-Cushman-001 (Microsoft 2023 and
https://github.. hatomm/ahi12200 12801 4/ccea ccacalpaca

---

Published as a conference paper at ICLR 2024
Tests Passed
100
73.2
64.0
01.4
60
53.0
47.5
82.7
35.0
52.3
25-13
18% 17452
20
GM14
WiannEadeer (Chaton)
Woard E50
Clavide Plas
Codettame 34l
Clasda
HarCaater
Insired Codrggen 158
Starcadem Plis
Paleon Anuiract
Vitaman 2.3
duanaro
Figure 2: The percentage of pass rates on the HumanEval and HumanEval+ with single attempt
(greedy decoding). following the EvalPlus leaderboard (Liu et al. 2023)
Codex (Chen et al.) a0 (ii) Google's Bard, PaLM (Anil et al., 2023 PaLM (Chowdh-
ery et al. )022), and LaMDA (Thppiilan et al. 2022); (iii) Google DeepMind's AlphaCode (Li
al. (iv)) Anthropic's Claude; (v) Huawei's PanguCoder2 (Shen et al. 2023); and (vi) Meta's
Umnatural-CodeLLama-- (Rozière et al. 2023).
Open-Source Models. Several open-source LLMs (OSS) have been made available to the Al community, although their performance generally lags behind the closed-source models lot. As part of our research, incorporate significant number of these open-source models our baselines.
Our baselines encompass the following models: InCode Fried et al. (2022), StarCoder and StarCoder-
Plus (Li et al. (2023a). LLaMal&2 (Touvron al.) 2023a b). CodeGen (Nijkamp al., (023b). CodeGeeX (Zheng et al. 2023), CodeT5+<Wangg et al., 2023), ). and CodeLlama (Rozière et al. (023)).
In addition, we also include several models with instructions fine-tuning, including CodeLlama-
Instruct (Rozière et al. 2023). OctoCoder (Muennighoti et al.) 2023), InstructCodeT5+ (Wang et al.
2023
Instruct-Coeggen 16B GGanaco-65B (Dettmers et al. 2003). Falcon-40B- 40B-Instruct (Penedo
et al. 2023) and Vicuna-13B (Chiang et al. 2023). More details can be found in the Appendix
4.2
IMPLEMENTATION DETAILS
The StarCoder and CodeLlama- (aiima-34B-2a 34B-Python serve basic foundation models. OpenAl's gpt3.5- turbo is used to evolve the dataset and generate responses. The evolved dataset consists of approxi- mately 78k samples. To fine-tune the basic models, employ specific configurations, including batch size of 512, sequence length of 2048, 200 fine-tuning steps, 30 warmup steps, a learning rate of 2e-5, Cosine learning rate scheduler, and fp16 mixed precision.
4.3 EVALUATION ON HUMANEVAL, HUMANEVAL+, AND MBPP
HumanEval (Chen et 20216 HumanEval+ Liu al. 2023), and MBPP (Austin al. 202 are key benchmarks in the Code LLM field, featuring diverse Python programming problems validated using test cases. HumanEval comprises 164 problems with an average of 9.6 test cases per problem. HumanEval+ expands the test cases significantly to average of 774.8 per problem. In contrast, MBPP provides 500 test programming problems with three automated test cases each
Comparing with the Closed-Souree Models. Following the same setting of the EvalPlus leader- board (Liu et al. 2023). In Figure 2 we compare our WizardCoder models with the closed-source models, such as GPT4, Claude, and Bard on this Icaderboard. Notably, all models generate code
https: hettps://hugiiii ingiiingaace www.ac/c...oo........ooooooooooo sah 12801 For a fair comparison, we present results for GPT3.5 (TTT3S5CChaGPP)&GGPT using Eval-Plus with the latest OpenAl's APIs (Liu al. 2023 (Figure 2 and OpenAl's report (OpenAl 2023) (Table []. Prompt format details in Appendix
166

---

Published as a conference paper at ICLR 2024
Table Results of pass @ 1(%) on HumanEval and MBPP. We follow the previous works (Chen et al. b02 to generate n=200 samples to estimate the pass @ score of our WizardCoder models with the same set of Hyper-parameteen temperate=0.2, and top. iop.p=0.99. *: our reproduced results.
Model
Params
HumanEval
MBPP
Closed-source models
LaMDA Thoppilan
2022
137B
14.0
AlphaCole La
1.1B
17.1
Pal.M
Chowdlhery
2022
540B
PaLM-CodkY
(C Choowdeeey
2022
540B
36.0
47.0
PaLM
Amil et de R025
Unknown
37.6
50.0
Codex
CChen
2.5B
21.4
Codex (Chen ef
2021
12B
28.8
Code-Cosnmar-ou
(MIKTosoft
2023
Unknown
33.5
45,9
Code Davinci-001 002 Miement
2124
Unknown
47.0
58.1
GPT:3.5 (CharGPT)
TOpenA (OpenA]
2174
Unknown
48.1
52.2
PanguCodar2 (Shen aT
15B
61.6
Unnatural Code Lama
Rozzere et 2023
34B
62.2
61.2
GPT4 OOpenA] 2023
Unknown
67.0
Open-source models
Llama Touvron et
65B
23.7
37.7
Llama2
2045h
70B
29.9
45.0
CodeGen Mono Kapiaampp
2023b
16B
29.3
35.3
CodoCleX Zbeng 2046
24.4
StarCoder Crer al. 21263
15B
33.6
43.6"
CodeT5+ Wang
InstructCodE13 Waig
2023
16B
35.0
OctoCoder (Muenagghofff 2l.
2126
15B
46.2
CodeL_lama Roziere
34B
48.8
55.0
Cadel.lama- Pytnon Knzzere ei
2023
34B
53.7
56.2
(Koziere
WisondCoder
15B
57.3
51.8
WizendCoder
34B
71.5
61.2
solutions each problem utilizing single attempt, and resulting pass rate percentage is reported. To maintain consistency, we employ the same experimental setup by generating answers using greedy decoding and evaluate our WizandCoder models using the provided evaluation codes.
As depicted in Figure 2 our WizardCoder 34B attains the second position in this benchmark, surpassing GPT3.5 (ChatGPT, 64.6 vs. 63.4) on HumanEval+. Our 15B version outperforms Claude- Plus (59.8 vs. 53.0) and Bard (59.8 vs. 44.5). Furthermore, WizardCoder models demonstrate remarkable superiority over other open-source LLMs that undergo instruction fine-tuning.
Comparing with the Open-Source Models. In Table I we conduct a comprehensive comparison of WizardCoder with other open-source models on the HumanEval and MBPP benchmarks. In contrast to the results presented in Figure 2 we adhere to the approach outlined in previous studies Chen et al. (2021b) by generating samples for each problem to estimate the pass @ score. The findings presented in Table I clearly demonstrate that our WizardCoder exhibits a substantial performance advantage over all the open-source models.
4.4 EVALUATION ON MULTI-LANGUAGE CODING
We included comprehensive assessment results across 8 distinct programming languages on the MultiPL-EE benchmarks. These languages encompass Java, JavaScript, C++, PHP, R, Julia, Swift, and Rust. The empirical results, as presented in Table distinctly demonstrate the superior performance of WizardCoder models across all evaluated programming languages, surpassing the SOTA open-souree Code LLMs. This underscores the efficacy of our Code Evol- hol-Instruct method.
4.5 EVALUATION ON DS-1000
The DS-1000 benchmark Lai et al. 2022 comprises 1k distinct data science workflows spanning 7 libraries. assesses the performance code generations against test cases and supports two evaluation modes: completion and insertion. In our experiments, only report insertion scores for
7

---

Published as a conference paper at ICLR 2024
models that support. In Table present pass @ (n=40) results for each library, along with an overall score o Based on these results, our conclusion is that WizardCoder demonstrates a significant superiority over all other models when tackling data science problems on the DS-1000 benchmark.
Table 2: Results of pass@ 1(%) different programming languages on the MultiPL-E (Cassano
et al. 2022) benchmarks. All models are evaluated with the same set of hypeer-aaaaetees:: tempera- ture=0.2, 0.2, top-p=0.95, mex 12, and n=50.
Model
Params
Java
Js
CPP
PHP
R
Julia
Swift
Rust
CodeGen-Multi
16B
22.2
19.2
21.0
8.4
6.5
0
1.3
4.2
CodeGreX
138
19.1
16.9
16.9
13.5
3.9
0.3
7.3
7.9
SodleCushman-001
31.9
31.3
30.6
29.0
11.
22.1
25.2
StarCoderBase
15B
28.5
31.7
30.6
26.8
10.2
21.1
16.7
24.5
StarCoder
15B
22.7
CodeLlama
34B
40.2
41.7
41.4
40.4
22.7
31.4
35.3
38.7
Codel Python
Codel MaooLllma-Isstase
34B
41.5
45.9
41.5
37.0
24.3
32.7
37.6
39.3
WigardCoder
15B
35.8
41.9
39.0
39.3
33.5
34.0
33.7
27.1
WignrdCoder
34B
44.9
55.3
47.2
47.2
39.8
41.5
44.3
46.2
Table 3: Performance WizardCoder 15B and baseline models on DS-1000. models are evaluated with the same set of hyper-parre parameters: temperature=0.., top-p=0.5, max 1engthh11224 Scores are average pass accuracy over 40 samples. Matplotlib (plt) task does not have the right context, so insertion and completion scores are identical.
Format
Model
plt
up
pd
py
scp
of problems:
155
220
291
68
106
115
45
1,000
Completion
InCoder-6B
28.3
4.4
3.1
4,4
2.8
2.8
3.8
7.4
Completion
CodeGen-mono
31.7
10.9
3.4
7.0
9.0
10.8
15.2
11.7
Completion
Code Cushman 001
40.7
21.8
7.9
12.4
11.3
18.0
12.2
18.1
Completion
StarCoder
51.7
29.7
11.4
21.4
20.2
29.5
24.5
26.0
Completion
WawdCoderr
55.2
33.6
16.7
26.2
24.2
26.7
29.2
InCoder-61
4.4
Insertion
StarCoder
51.7
30.8
10.3
21.0
20.2
27.4
20.0
25.4
Insertion
WizarCoder
55.2
35.1
20.4
30.4
28.9
32.3
37.8
32.8
ANALYSIS
Evolution Models and Rounds. In Table GPT- 4 replaces GPT-3.5 for evolved rounds, boosting HumanEval Pass @ 1 scores 73.8 (34B) and 62.2 (15B). Using OSS proves effective, yielding scores of 70.1 (34B) and 55.5 (15B). Despite GPT-4's superior coding perfor- mance (88.4 73.2), the gain in evolved rounds is not proportional (73.8 vs. 73.2). Conversely, CodeLlama's 's weaker performance narrows when using Code Evol-Instruct (73.2 vs. 70.1), highlight- ing its crucial role. More experiments details are listed in Appendix E Additionally. Figure 3 presents results for different data evolution rounds. All models are fine-tuned with 200 steps. Due to the limited size of the dev set MBPP, we merged the training set and dev set, forming the MBPP-400 dev set. The experiments reveal that the highest pass scores on both the MBPP-400 dev set and the HumanEyal are achieved subsequent to three rounds of evolution.
Table 4: Different evolution execution models.
also
Base Model
Evol Model
Pass
StarCoder-15B
GPT-4
62.2
StarCoder-155 15B
GFF:33
59.8
StarCoder 15B
Cama
55.5
CodeLlamm-334 34B
GPT-4
73.8
CodeLLama
GPT.3.5
Codel Jama-
Codel Codel lama-334
70.1
GGien that this benchmark and its evaluation codes are not designed for the instruction fine- fine-tuned models, we encounter significant challenges in aligning our 34B model with this framework. Moreaver, the Codellama-34B base model does not support code insertion. Thus, we only include 15B model results.
8

---

Published as a conference paper at ICLR 2024
0+1x2e3e4
57.7
0x1x2-9H4
Ox1-213
Evol Round
0+1-2+3
57.2
Del
0-1e2
0:1+2
565
0r1
56.8
55
56
57
58
Pioa HumannEial
(a) Pass@1 performance on MBPP-400 dev set.
(b) Pass@1 1 performance on HumanEval.
Figure The impact number of data evolution rounds.
Complexity and Quantity. While the enhanced performance attributed to our Code Evol-Instruct method has been evident in prior experiments, it remains an open question whether this performance gain is result of an increase in the number of samples includes more samples, and the introduction of more complex instructions inevitably leads to an increase in tokens within the training data. To address this question, we fine-tune the models using only the specific round data separately from scratch with a similar number of samples (upper part) or tokens (lower part) in Table 5
tokens. During the evolution, each round
When each round contains the same number of samples or tokens, models trained with the seed data still lag behind the evolved rounds. Furthermore, combining data from different rounds leads to the best performance. These results suggest that the primary source of the gain is indeed attributable to our Code Evol-Instruct method, rather than merely increase in samples or tokens.
Table 5: Analysis of whether the perfor- mance gain comes from more tokens.
Evol
#Samples
Pass
20.0k
Round
18.8k
56.
Round 2
19.7k
53.0
Round
19.3k
54.3
Complexity and Similarity. Apart from the quantity analysis, we also investigate whether evolution leads the inclusion of data more similar the test set. To address this, we perform an analysis of the HumanEval test set. We employ test samples queries retrieve the top-1 sample from each evolved round's training data, utilizing the SOTA embeddings model, gte-large (Li al.. [023b). Additionally, we employ GPT4, to provide average sim- ilarity scores between the samples. The details are shown in Appendix C
Round 4
19.0k
51.2
Evol
ATokens
Pass
Round 0
2.3M
44.5
Round
2.3M
51.8
Round 2
2.3M
52.4
Round 3
2.3M
50.0
2.3M
and the retrieved top-1
Figure iilustrraes that the evolution process does not yield higher similarity scores. Furthermore, similarity scores across all rounds remain relatively low. These findings indicate that the primary source of performance gain is the introduction of more complex data.
Sibbitt Score
4.5 0.15
CONCLUSION AND FUTURE WORK
Round
Figure 4: Average similarity scores be- tween HumanEval samples and the top- retrieved data, ranging from 1 (com- pletely different) to 10 (identical).
This paper introduces WizardCoder models, the Code Evol- Instruct fine-tuned Code LLMs. The experimental results demonstrate that WizardCoder models achieve SOTA per- formance surpassing all existing open-source Code LLMs on five widely recognized code generation benchmarks: HumanEval, HumanEval+, MBPP, DS-1000 and MultiPL- E. Notably, WizardCoder 15B model surpasses some of the well-known closed LLMs, such as Claude and Bard. Additionally, WizardCoder 34B achieves HumanEval score comparable to GPT3.5 (ChatGPT) and surpasses it on the HumanEval+ bench- mark. Furthermore, analysis underscores the pivotal role of instruction complexity in enhancing performance. For future work, as depicted in Figure 2 our model still falls significantly behind the SOTA LLM, GPT4, Therefore, future work will further augment the performance of our model.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work is partially supported by National Natural Science Foundation of China Young Scientists Fund(No. 62206233) and Hong Kong RGC ECS (No. 22200722).
REFERENCES
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent EI Shafey. Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernández Ábrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christo- pher Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, and al. Palm 2 techmical report. CoRR, 0ttps/2305.000000 2023. doi: 10.48550//rXiv.230000 1005. URL ht.pps://dos .org 485404 2305 10403
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Prakash Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. Ext5: Towards extreme multi-task scaling for transfer learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL http:t/:://// wwwpereeeeeeeeeeeeeeeeeeeee /
net. / forum id=
Jacob Austin, Augustus Odena, Maxwell I. Nyc, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732 2021. URL http:///wwwxxv...o/abbabaaa
2108. 07732]
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang. and Samuel Weinbach. Gpt-neox-20b: An open-source autoregressive language model. CoRR, abs/2204.06745 2022. doi: 10.48550/ar .rg. /0. 48550/arXiii 20w.ex00...22220455555.5
2204.06745. URL htttht:/s://///
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Vorbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu. Clemens Winter, Christopher Hesse. Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewriPS 2020, December 6-12, 2020, virtual, 2020. URL httt:::::///oocettttiiiii 1L5700d8222230111111555222222222
www.ddiiiinnnn.u..s....e.. neurips wttpapp.cccppppee pttpp/pppe///22///aaaaae
html
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. scalable and extensible approach benchmarking nl2code for 18 programming languages. CoRR, abwp/2208.082//2 2022. doi: 10.48550/arXiv.2208.. 08227
08227. URL /gg.thab.ccm/aaaa122211///aaaaae
Sahil Chaudhary. Code alpaca: An instoactionnfollloiiing llama model for code generation. https 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri,
10

---

Published as a conference paper at ICLR 2024
Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.033 2021a. URL htttt::ttaaaa..
org/ aba/ 2107 03374
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, ftb//210.003774 2021b. URL https:/:: //at:i///xx hogg/bs//20
03374
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. https
//vicuna. Imsys.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyzz, Er- ica Moreira, Rewon Child, Öleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language model- ing with pathways. CoRR, abs/2204.02311, 2022. doi: 10.48550/axxi..2204.023 02311
URL
Inttps://doi org/10..
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slay Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction- finetuned language models. CoRR, abs/2210.11416, 2022. doi: 10.485500/rXivv2221..1 1416. URL 48550/ 11416
https:://oi..og wttp:///gg.....////.......
Tim Dettmers, Artidoro Pagmoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. CoRR, abs/2305.14314 2023. doi: 10.48550/axX1..2355.1111 URL https
//doi, /
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling
and synthesis. CoRR, abs/2204.05999 2022. doi: 10448500/rrii..2244.5999 URL https
/ w... 05999

---

Published as a conference paper at ICLR 2024
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. CoRR, abs/2203. 15556, 2022. doi: 10.48550/arXiv22203. 15556. URL https:///bl..rr//100.. 48550
4ttpp//////o....////0044444///// wtt.:iiwii................ 2
Akshita Jha and Chandan K. Reddy. Codeattack: Code-based adversarial attacks for pre-trained programming language models. In AAAI Conference on Artificial Intelligence, 2022. URL
https://appt www.sttetatttttteeeeeeeee
org/ Ctrpps ID : 249240330
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. Unifiedqa: Crossing format boundaries with a single QA system. In Trevor Cohn, Yulan He. and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pp. 1896- 1907. Association for Computational I Linguisties, 2020. doi: 10. 10.18653/1/2020...mo 0018663/110000...ddnsssmmm/ppp
URL https: / doi 1ttp://0...00333//22222222222222 -indingg---::1..1111
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen-tau
Yih, Daniel Fried, Sida Wang, and Tao Yu. DS-1000: A natural and reliable benchmark for data science code generation. CoRR, abs/221 1. 11501 2022. doi: 10.48550/arxiv.2211..1 11501
URL
ht.ps:://of .al.doo.orgg//o1 www.orgg........ 48550/ (3550/aaxxxv/
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06160 2023a.
Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. CoRR, abt/s/03.022222.... 2022. doi: 10445500anXi..200..777 wtt../ooo//...555//222..... www.aaawwvvv...... 07814
Sven Gowal, Alexey Cherepanov, James
URL
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. CoRR, abs/t300.03222222 2023b. doi:
10.48550//rXiw.2338..3322 03288. URL 1B/tttteeer//.......00....44111. wtt.aawwwii........... 03281
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. CORR, abs/2305.01210, 2023. doi: 0.485500/axxi.2355..0122
hiRlihhttss///oi..o
48550/arXiv. wwt.:aiii.......22.
Aman Madaan, Alex Shypula, Uri Alon, Milad Hashemi. Parthasarathy Ranganathan, Yiming Yang, Graham Neubig, and Amir Yazdanbakhsh. Learning performance--mpprovnng code edits.
ArXiv, abs/2302.07867, 2023. http::/:wppp...aaaaiaaaaaaaaaaap
256868633
Mierosoft. Azure openai service models.
wtt.:rooooft.com/euuuuue. 2023.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. CoRR, abs/2308 071244 2023. doi: 10.48550aarxx.22300072222 URL 07124
httpss://ont wrroiioorg/in http://1..441////1
Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. Codegen2: Lessons for training lims programming and natural languages. CoRR, abs/2305.022 023009 2023a. doi: 10.48550/a2xiv.2305.000 URL https:// /ww.dotgoo..oooooooooo//oooeeeet 48550/
2305. 02309
12

---

Published as a conference paper at ICLR 2024
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference Learning Representations, 2023b. URL
OpenAL. GPT-4 technical report. CoRR doi: 10.45550/uX1..2333.0777. 0ttt:s:://oll.rg1110.6...33
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback. In NeurIPS, 2022. URL htRLhttp: mipsr.s..pppppo.......... wttp://pss.ccpppppreeeeeeeeeeeee httpp////////////pp/prprpppoepa. http://ppppprrreeepppaee///////p /ttple////pppee//2222 2ttp:////22/hhhhhh//aae hash/
html
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cap- pelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The re- finedweb dataset for falcon LLM: outperforming curated corpora with web data, and web data only. CoRR, abs/2306.01116, 2023. doi: 10.485500arX1..2300.11116 URL https
//doi.org/10. www.oo/g/......////. 48550/ arrivv wtt.aawwwww......... 2306. 01116
Jack W. Rac, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmman,, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring. Susannah Young. Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Pagamini, Laurent Sifre. Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean- Baptiste Lespiau, Maria T'iipouueell, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals. Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrre Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs/2112 12.1 1446, 2021. URL. httphhttts//axttt. org abs/ / 21 112 11446
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to- text transformer. J. Mach. Learn. Res., 21:140:1-140:67 2020. URL /tmtt::///mmr.rg///
paaers/v21/20-01 074 .tml
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier. Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code. CoRR. abs/2308. 12950, 2023. doi: 10.448550/rxiv.2208..1
URL httLhtts::// doi
10. 10..0485//a21... http:aawwii............ 2ww.lv.......2200
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika. Zaid Alyafeai, Antoine Chaffin, Amaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference Learning Represemaations, ICLR 2022, Virtual Event, April 25- 2022. OpenReview.net, 2022. net. /ttp
URL hhtttps:///ppereeeetttt
13

---

Published as a conference paper at ICLR 2024
Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, Yuenan Guo, and Qianxiang Wang. Pangu-coder2: Boosting large language models for code with ranking feedback. CoRR, abs/2307.14936. 2023. doi:
20,48550/rXxiv.2377..5 149366. URL httpthtts///...o.. wttpr//1...0000//1.....2.2 230..
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto. Stanford alpaca: An cottnactioofoolowinng Ilama model. https://githhii com/ tat st stanford alpaca, 2023.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. CoRR. abs/2205.05131, 2022. doi: 10448500axxi..205.01111.. URL httpp:://oi..oo org//000.orgg00
48550/arXiv 2205 05131
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng. Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang. Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung -hing Chang, Igor Krivokon, Will Rusch, Mare Pickett, Kathleen S. Meier-Helllteni Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Lguera-Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoc Le. Lamda: Language models for dialog applications. CoRR, abs/2201.085 008239, 2022. URL https: aaxiv. eerxiv.rrg.oo abs 2201 08239
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, 2tt 2023a, doi: 10. (0.448550/aXXX123322.339 org 005//0.4888 400.48850/444 arXiv 2302. 13971
BRR.htts:: //doi
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, 0abb/2307.092883 2023b. doi: 10.485500arXiw.2300 200...0.....000....... URL https: petttp://doi.ooo org/ 10. http://tww........oooooooeeeetta
2988888888
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https:
May 2021.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXh::2212. 2022.
Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. Codet5: Identifier- unified pre- trained encoder-decoder models for code understanding and generation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.). Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. (696-8708 Association for Computational htt::/oo..oo/// Linguistics, 2021. doi: 10.18653/v 1ttp3/111122221m///p. maamlp-miin6685. URL httpl/ttt//o.o....o
18653/v1/2021 emn 1p pp n. 685
14

---

Published as a conference paper at ICLR 2024
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. Codet5+: Open code large language models for code understanding and generation. CoRR. abs/2305.07922, 2023. doi: 1048550/aXX..2205007922 URL
48550/arXiv 2335.00922
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. CtRl.hhtps:/// http:///ppeevvvieeeee net/ forum?ide gEZrGCozdRR
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 222::2330..2244,, 2023.
Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, and Zhilin Yang. Zeroprompt: Scaling wrompt-based pretraining to 1. 000 tasks improves zero-shot generalization. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-
11, 2022, pp. 4235 4252. Association for Computational Linguisties, 2022. URL (tRpss/t::/ acl sclanthology.oo wrg./022. findings-emn 312
Zhou Yang, Jicke Shi, Junda He, and David Lo. Natural attack for pre-trained models of code. 2022 IEEF/ACM 44th International Conference on Software Engineering (ICSE), pp. 1482-1493, 2022. URL httt:htts:://// //ppi semanti wws.mmtttscccolaaaa org/ Corpus 24421 0250
Aohan Zeng, Xiao Liu, Zhengxiao Du Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam. Zixuan Ma, Yufei Xue. Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130P an open bilingual pre-trained model. CoRR, abs/2210.02414, 2022. doi: (0448550/arxiw:220002224 URL wtt.h:tpss.... http://doooooooo 48550/arxiv. 2210. 02414
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: open pre-trained transformer language models. CORR, abs/2205.01068 2022. doi: 10.48550/ arXiv.2205.0106 URL //cttps://do...
03550/arXiv.2200 01068
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. CoRR, abs/2303. 175668 2023. doi: 10.48850//rXiv.23333, 17568. URL/http /doii org 10. 48550 (05550/aXXivv 2303.
15

---

Published as a conference paper at ICLR 2024
A
PROMPT FORMATS
In this section, we include the prompt for evaluation on different tasks.
Zero-Shot Prompt for Evaluation on HumanEval and HumanEval+
Below is an instruction that describes a task, paired with an input that provides further context. Write response that appropriately completes the request.
### Instruction:
Create Python script for this problem: {Question]
#### Response:
Three-Shot Prompt for Evaluation on MBPP
Below is an instruction that describes a task, paired with an input that provides further context. Write response that appropriately completes the request.
### Instruction:
Create Python script for this problem:
[uesttion]
Test Example 1)
(Test Example 2}
[Test Example 3}
### Response:
Zero-Shot Prompt for Evaluation DS-1000 (Completion)
Below is an instruction that describes a task, paired with an input that provides further context. Write response that appropriately completes the request.
### Instruction:
{Question}
Complete the Python code "...".
### Response:
In the case of DS-1000 (Insertion), adherence to the benchmark's specifications necessitates the utilization StarCoder's specialized insertion symbol. Consequently, we have found it imperative to align with the same prompt format employed by StarCoder for this particular benchmark.
For the MultiPL-E benchmark, we recognized the need to align with the evaluation codes provided by bigcode-en bigcode-evaluation eeallationn Consequently, opted to adopt the same prompt format utilized by StarCoder.
B BASELINES DETAILS
We include large amount of models as our baselines. For GPT3.5 (ChaiGPT)&&GT4, their results are obtained from GPT4's report and EvalPlus. The results of Coode-Dvvincieeee Codex, PaLM, PaLM 2, LaMDA, AlpahaCode, Incoder. StarCoder, LLaMa, CodeGen, CodeGeeX. CodeT5+. and InstructCode are from StarCoder or CodeT5+'s 's paper. The results of Bard are evaluated with Google's API. The results Claude are evaluated with Anthropic's API. The results
Code-CCshman-O00
wwwde-ooooooooooooooo uuat -ottiinnnheesss
16

---

Published as a conference paper at ICLR 2024
of Instruct- Cooluutt-Cooe--e--- 16B, Guanaco-65B Falcon-40B- Innssrrct,, and Vicuma-13B evaluated with the open-sourced checkpoints. The results of CodeLlama- Series are from CodeLlama' 's paper. The results of OctoCoder are from its paper. The results of PanguCoder2 are also from its paper.
The MBPP score of StarCoder differs from that in its original paper. Through a personal contact, we informed that StarCoder evaluated using cleaned and smaller version of MBPP, comprising only 397 problems, significantly fewer than the original MBPP benchmarks (500). Consequently, we conducted re-evaluation of StarCoder using the original MBPP.
C SIMILARITY CHECKING AND DATA FILTERING
The prompt formats to compute the similarity score follow:
System Prompt for Similarity Checking
Your task is to evaluate the similarity of the two given coding tasks. Please review the two coding tasks carefully, paying close attention the overlap in function names, code structures, topics, and contents. Once you have carefully reviewed both coding tasks, provide a similarity score between these two coding tasks. The score should range from to 10 (I: completely different coding tasks; 10: identical coding tasks). You only need to provide your score. The response format is: Score: "...'
User Input for Similarity Checking
Task1 [task1}
# Task2 [task2)
To thoroughly prevent data leakage from the test datasets to the training dataset, we implemented an additional data filtering step. Utilizing the SOTA embeddings model, gte-large, we treated all test samples queries to extract the top samples from the training data. Following this, GPT-4 was employed to evaluate the similarity between the retrieved samples and the test sample. The task for GPT-4 is simplified to binary decision- either "yes" or "no" indicating match. In case of positive match, the sample excluded from the training data.
D EVOL EXAMPLES
In this section, we present some evolved examples to elucidate the influence exerted by our Code Evol-Instruct.
Example 1:
Round O: Write a MongoDB query to select all documents in collection where the field
'category' is 'clothes'.
Round I: Write a MongoDB query to select all documents in collection where the field 'category" is 'clothes' and the 'brand" field is not equal 'Nike'.
Round 2: Write MongoDB query to select all documents in collection where the field
'category' is 'clothes' and the 'brand' field not equal to 'Nike', and the 'price' field is greater than equal 100 and less than or equal to 500.
Round 3: Write MongoDB query to select all documents in collection where the field 'categor'' is "clothes' and the 'brand' field not equal to 'Nike', and the 'price' field is greater than equal to 100 and less than equal 500, and the 'color' field is either 'red' or 'blue'. Additionally, sort the documents in descending order by the 'date added' field and limit the result to the first 10 documents.
17

---

Published as a conference paper at ICLR 2024
Example 2:
Round O: Using Tensorflow, set model to predict the classification of a handwritten digit.
Using Tensorflow, set up model predict the classification of handwritten digit with the constraint that the model must be trained using only 1000 samples from the MNIST dataset.
Round I:
Round 2: Using TensorFlow, set up model to predict the classification of a handwritten digit with the constraint that the model must be trained using only 500 samples from the MNIST dataset. Additionally, implement transfer learning to incorporate the features learned from a pre-trained model.
Round 3: Using TensorFlow and the MNIST dataset, develop a convolutional neural network (CNN) to accurately classify handwritten digits. However, in this version of the task, you must limit your training to a maximum of 500 images. Furthermore, incorporate transfer learning by using pre-trained model extract features from the data and improve your model's accuracy.
Example 3:
Round O: Create class in Java that can store list of students and the courses they enrolled in.
Round I: Create a Java program that manages a student database with courses, including options to add, remove, and search for students, well courses for each student. The program should also be able to display statistics on the number of students and courses.
Round Develop a Java program that manages student database containing courses and grades. Your program should allow users to add, remove, and search for students, well as add or drop courses for each student. Furthermore, it should display statistics on the number of students and courses, and calculate the average grade for each student. Ensure that your program implements appropriate error-handlig mechanisms.
Round 3: Develop a Java program manage library system where users can check-in, check-out and search for books in the library. The system should allow users add new books and delete existing ones from the library. The program should also provide statistics the number of books checked-in and checked-out, and calculate the average reading time for each book. Ensure that your program implements appropriate error-handling mechanisms and enforces data integrity constraints such not allowing users to check-out books that have already been checked-out.
E USING DIFFERENT EVOLUTION EXECUTION MODELS
We explored using the open-source models (OSS) instructions. However, it demonstrated relatively low coding performance in response generation. To address this, we fine-tuned it using our code-alpaca dataset and utilized this model for response generation.
for generating evolved
INTERACTION EXAMPLES
Table 6 showcases examples of interactions with WizardCoder. The examples demonstrate that our model consistently generates accurate responses accompanied by clear explanations.
18

---

Published as a conference paper at ICLR 2024
Table 6: Examples of interaction with WizardCoder.
Instruction
Response
Here's Python function that tells you what the date is today:
import datetime
def get date
new datetime datetime.. now () return now.strfting ("$A, $B &d, 3Y")
Write a Python function tell me what the date is to- day.
print (get_date ())
This function uses the datetime module get the current date and time. It then uses the strftime() method to format the date as a string in the format "Day, Month Date, Year". Finally, it returns the formatted string.
You can call this function from your Python code to get the current date and time.
Sure, here's 's an example code for integrating a Python function numerically using the trapezoidal rule:
def integrate (f, b, n)
(b a) n
(a) + (bb) 2
for in range (1, n): += f i h) return h*s
This code takes function f, the lower bound a, the upper bound b, and the number of subintervals n. It then calculates the width of each subinterval h, and uses the trapezoidal rule approximate the integral of between and b. The result is returned as float.
do not know how to integrate Python function numerically. Please write a code for me.
To use this function, you would call with the function you want inte- grate, the lower bound, the upper bound, and the number of subintervals you want use. For example:
def (
return x**2
result = integrate (2, 0, 1, 1000) print (result)
This would integrate the function f(x) x**2 between 0 and using 1000 subintervals, and print the result.
19

---

Published as a conference paper at ICLR 2024
Table 7: Examples of interaction with our WizardCoder.
Instruction
Response
Here's the Java code to list the prime numbers between 20 and 30:
public class PrimeNumbers {
public static void main (String [] args)
for (int i 20; i 30; i++)
boolean isPrime true;
for (int j 2; < i; j++) {
if (i $ -- C)
isprime false;
break;
if (isPrime) { System. out, print (1 " ");
List the prime numbers between and 30 with Java.
Output:
23 29
Explanation:
We use a nested for loop to iterate through all the numbers between 20 and 30.
For each number, we use another for loop to check if it is prime or not.
If number is divisible by any number other than and itself, then it is not prime.
If number not divisible by any number other than and itself, then it is prime.
We print the prime numbers to the console.
20

---

Published as a conference paper at ICLR 2024
Table 8: Examples of interaction with our WizardCoder.
Instruction
Response
Here is simple neural network with pytorch:
import torch
import torch. nn as nn
class Net (nn Module)
def
init (self) super (Net, inif)._init_ ()
self.fcl nn nn. Lineai (2, 4)
self.fed no Linear (4, 1)
self.sigmoid
Sigmoid
def forward (self, x):
= fcl
self. sigmoii))
self.
return
net N)t ()
criterion MMSELsss
optimizer torch opt.m.ttimmmmmmmmm. SGD SGD (net .eea.paaeeeeess (), 1r=0, 011)
Write simple neural network with pytorch
# Training
for epoch in range (1000):
inputs torch (10, targets torch, (10, 1)
(
outputs net (inputs)
loss criterion (outputs, targets) loss bacckwarc)) aptimizer.step
A. Testing
inputs torch.randn. (10, outputs net (inputs) print (outputs)
In this example, we define a neural network with two fully connected layers and sigmoid activation function. We use the mean squared error loss function and the stochastic gradient descent optimizer. We train the network for 1000 epochs and then test it with some randomly generated inputs. The output of the network is tensor of predicted values.
21


Reviews:
[{"review_id": "fCVAuyazDV", "reviewer": "Reviewer_6CYQ", "summary": "WizardCoder proposes an instruction fine-tuning method for code - Code Evol-Instruct where a large synthetic dataset is created using a seed set of code tasks (Code Alpaca) which are evolved by GPT-3.5 to increase their difficulty and complexity in multiple rounds. When fine-tuned using this dataset, publicly available models like CodeLlama and StarCoder are shown to outperform all other open sourced alternatives, and are comparable to some of the closed source ones on many different benchmarks on code generation like HumanEval, MBPP and DS-1000.", "questions": "- Figure 3: Why are results from evol round 0 + 1 + 2 + 3 + 4 worse than 0 + 1 + 2 + 3? Do returns from EvolInstruct start to diminish or turn negative after a certain number of rounds? This is not discussed or explored in this paper.\n\n- Analysis - Complexity and Quantity: This section is not clear to me. What are the results shown in Table 4? How does Table 4 make one conclude that the gains are not due to increase in samples or tokens? Are the 4 rows in Table 4 pass@1 metric for model checkpoints when sequentially trained on each round?\n\n- How dependent is the performance of Evol-Instruct on the evolving model? Will the performance substantially improve if we use GPT-4 to generate evolved rounds? Will it worsen if the evolving model is changed to other open sourced alternatives?\n\n- Section 3.1: We optimised the evolutionary instructions by eliminating deepening and complicating inputs, as well as In-Breadth Evolution. What does this mean? Not clear how 1 and 2 are different, can you describe precisely with an example?\n\n- Can you provide details of the compute infrastructure involved and hyperparameters in fine-tuning? (number of GPU hours, batch-size, sequence length) and other details)\n\n- The abstract (and some places in the paper) comment on model sizes of Claude and Bard as 'largest closed-source LLMs', this is not verifiable, so I suggest the authors re-word this without commenting on their model size and avoid other such colloquial writing present in the paper to improve its soundness.\n\nTypos:\n- Intro Para 3: several key adaptions --> adaptations", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Clever instruction finetuning idea on creating datasets synthetically using gpt-3.5 and a small set of seed tasks\n- Exhaustively tested on different programming languages, impressive performance gains using publicly available models (StarCoder and CodeLlama-34B) across several benchmarks. -HumanEval+, MBPP, MultPL-E, and DS-1000", "weaknesses": "- While results on code benchmarks are impressive, novelty of the scientific methodology itself is quite limited as it is an adaptation of Evol-Instruct for Code.\n- Missing human assessment - It is not clear how useful the final fine-tuned model is outside the benchmarks that focus exclusively on functional correctness. Model hasn't been tested on developer productivity tasks like completion, code refinement.\n- Not clear if data leakage has been prevented. Does the evolved data or seed data overlap with HumanEval or other benchmarks' test set?\n- The paper assumes the reader to be familiar with Evol Instruct, and does not provide sufficient context to follow the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rIGktcwlx8", "reviewer": "Reviewer_fiHu", "summary": "The paper proposed a new way, Code Evol-Instruct, to fine-tune an LLM for coding tasks. The method starts from a set of existing code questions, then uses another LLM (GPT 3.5, in this case) to add one more step of complexity or difference to make a new question. Detailed prompts for evol-instruct are provided in the paper. 4 rounds of evol-instruct expanded the original 20k Code Alpaca instruction set to 87k. The WizardCoder model fine-tuned with this set ranked only behind GPT-4 in HumanEval and other popular coding benchmarks, beating other major open and closed-sourced models.\n\nThe paper also analyzes the effectiveness of the evol-instruct set from different rounds of expansion. The results show that generally more data leads to better performance (except with the addition of round 4), the same amount of tokens from more complex instructions works better, and the data expansion didn't create more similar data to the test set.", "questions": "- Where does the new coding solution come after expanding the instruction?  Did we use the base model itself, or GPT-3 to generate them?\n\n- Did we do any deduplication of the expanded instructions?\n\n- Did we verify the quality of new instructions by executing the code generated?\n\n- Figure 1 lower plot. It's better to use pattern instead of color to separate two data classes to make it more friendly for color-blindness.\n\n- Section 1 paragraph 2. Reviewer thinks \"Code\" need not to have capitalized first letter.\n\n- Section 2 Related Work. We should also mention CodeLlama's self-instruct work and compare the differences. https://arxiv.org/abs/2308.12950\n\n- Page 4, Code Evolution Heuristic Methods table. Please explain the 4th instruct's purpose and examples of expansion. \"Provide a piece of erroneous code as a reference to increase misdirection\". It's not clear to the Reviewer what this is doing and why it would be effective.\n\n- Page 4, Code Evolution Heuristic Methods table. Last prompt about increasing time/space complexity: how can we verify the response actually achieved it?\n\n- Section 5 paragraph 1, \"amalgamate the training set\", what about using a simpler word like  \"merge the training set\"? \n\n- Section 5 paragraph 2. The Reviewer assumes more rounds equals more complexity. But it's better to make it explicit and provide some examples to show what does complexity actually mean in this context..\n\n- Section 5 \"Complexity and Similarity\". This paragraph didn't really talk about complexity, unless more rounds = more complexity?\n\n- Table 4. It's better to show the confidence interval on this table, if possible, since 400 examples is not that large a test set.\n\n- Section 6: conclusion. \".. the pivotal role of instruction complexity in enhancing coding performance\". Reviewer doesn't think this claim holds well given the analysis in the paper. For example, in Table 4, later rounds (assuming more complexity) lead to a lower performance (or statistically insignificant changes).", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The method works. It produced a top-performing open source code model that surpasses bigger and closed-source models in multiple open evaluations. This is the biggest strength of the paper and the value to the research community.\n\n- Paper provided extensive comparison with existing models and between different rounds of evol-instruct expansions.", "weaknesses": "- Missing some key details. Where does the new coding solution come after expanding the instruction? Did we use the base model itself, or GPT-3 to generate them? Did we do any deduplication of the expanded instructions? Did we verify the quality of new instructions by executing the code generated?\n\nUpdate: based on the author's feedback, the coding solutions in the training data are from GPT-3.5-turbo. This presents a risk of data leakage, that WizardCoder becomes an implicitly distilled model of GPT-3.5-turbo on a specific capability.\n\n- Weak analysis. Why does round 4 have a negative impact on the quality? The paper mentioned \"some evolution process will fail\" but provided no detail. Other analysis related questions will be in the \"questions\" section.\n\n- Writing quality can be improved. The paper uses more sentences to repeatedly claim WizardCoder's performance (which is indeed impressive) but vague on implementation details and analysis.\n\n- Inconsistency in the data. Figure 2 reported GPT-4 has a HumanEval pass@1 of 88.4, while Table 1 showed 67.0.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "unsL3ugxtv", "reviewer": "Reviewer_QRie", "summary": "The paper extends the idea of instruction fine-tuning to code LLMs. The proposed approach uses GPT-3.5 to evolve existing instruction data to generate complex and diverse set of examples in an iterative manner. Using the synthetic evolving data followed by instructions, the authors fine-tune SOTA coda LLMs. The results show improved performance of the models for a wide range of tasks.", "questions": "(Q1) Table 4, the Pass@1 score decreases as the number of iterations increases. Do you have any insights as why this is happening? \n\n(Q2) How the performance of your model increases as you use less capable models for generating evolving data? In addition, do you think we can use the same model to generate the data for next iteration? \n\n(Q3) Can you clarify how did you come up with the heuristics? Did you eliminate any of them based on the final outcome? Was the process a trial-error approach or there were some insights behind making any of the decisions?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "$\\mathtt{+}$ I think overall exploring ideas around how we can improve the efficacy of LLMs for different application is interesting.\n\n$\\mathtt{+}$ Improving SOTA using the proposed instruction-tuning method is valuable and opens up new direction. The ablation studies further help to expand how and to what extend each technique helps (with some caveats that I will expand in the question section).", "weaknesses": "$\\mathtt{-}$ It is not clear how the authors came up with the list of heuristics for data evolution. This unclarity makes such approaches less applicable to wide range of tasks.\n\n$\\mathtt{-}$ While the ablation studies in the main body provides some insights on the efficacy of the technique (additional clarification in the questions/recommendation section).\n\n$\\mathtt{-}$ While the idea is interesting, but it seems very incremental compared to prior work and the contributions are limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vWIaR1xatK", "reviewer": "Reviewer_FnZW", "summary": "This paper presents a novel approach called \"Code Evol-Instruct\", which adapts the \"Evol-Instruct\" method to the domain of code, enhancing the performance of open-source Code LLMs.The authors use Code Evol-Instruct to evolve basic code instruction data and then fine-tune existing open-source Code LLMs, resulting in the creation of \"WizardCoder\" models to improve the ability of these models to generate high-quality code based on more complex instructions. The WizardCoder outperforms other open-source models and even rivals larger closed-source LLMs in certain coding tasks.", "questions": "1. In section 3.2, TRAINING WizardCoder,\" We iteratively employ the Code Evol-Instruct technique on this dataset consisting of around 20k samples to produce evolved data.\"  What is the number of iterations and the criteria for completion?\n2. In section 3, when iterating the Code Eval-Instruct on the sample dataset, what model is used to generate evolved data? Does the difference in the quality of pre-trained model outputs lead to different results?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Introduction of Code Evol-Instruct, an innovative method for enhancing open-source Code LLMs, which significantly improves their performance in code generation tasks.\n2. Even the smaller 15B version of WizardCoder outperforms larger closed-source LLMs like Claude and Bard on certain benchmarks, while the 34B version achieves performance comparable to GPT-3.5 (ChatGPT) and surpasses it on specific benchmarks.", "weaknesses": "1. In section 4.3, Table 1, Result of pass@1(%) on HumanEval and MBPP, the result of pass@1(%) on MBPP is not as good as the result on HumanEval. HumanEval only tests a narrow distribution and can be overfitted. The benchmark needs to be upgraded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)