Personalized Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation
Authors: Hailin Chen
Amrita Saha
Steven Hoi
Shafiq Joty
Venue: EMNLP 2023 Main (2023)
Keywords: Distillation
code generation
adaptive learning

Abstract: With the rise of powerful closed-sourced LLMs (ChatGPT, GPT-4), there are increasing interests in distilling the capabilies of close-sourced LLMs to smaller open-sourced LLMs. Previous distillation methods usually prompt ChatGPT to generate a set of instructions and answers, for the student model to learn. However, such standard distillation approach neglects the merits and conditions of the student model. Inspired by modern teaching principles, we design a personalised distillation process, in which the student attempts to solve a task first, then the teacher provides an adaptive refinement for the student to improve. Instead of feeding the student with teacher's prior, personalised distillation enables personalised learning for the student model, as it only learns on examples it makes mistakes upon and learns to improve its own solution. On code generation, personalised distillation consistently outperforms standard distillation with only one third of the data. With only 2.5-3K personalised examples that incur a data-collection cost of 4-6\$, we boost CodeGen-mono-16B by 7\% to achieve 36.4\% pass@1 and StarCoder by 12.2\% to achieve 45.8\% pass@1 on HumanEval.\footnote{We will release our models and codes upon acceptance}

Personalised Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation
Hailin Chen
Amrita Saha* Steven HOI Shafiq Joty
Nanyang Technological University, Singapore
Salesforce Research
(hailin001, srjoty) (nt.eeuuuuuuuuu......... shoi] @salesforce com
(amrita.
Abstract
all of which limit their seamless integration into real-world applications. In light of these concerns, a natural question arises: Can we distill the remark- able abilities exhibited by closed-source LLMs into smaller open-source LLMs?
With the rise of powerful closed-sourced LLMs (ChatGPT, GPT-4), there are inereasing inter- ests in distilling the capabilies of close-sourced LLMs to smaller open-sourced L.MSs. Previ- distillation methods usually prompt Chat- GPT to generate set of instructions and an- swers, for the student model learn. How- ever, such standard distillation approach ne- glects the merits and conditions of the student model. Inspired by modern teaching principles, we design personalised distillation process, in which the student attempts to solve a task first, then the teacher provides adaptive re- finement for the student to improve. Instead of feeding the student with teacher's prior, person- alised distillation enables personalised learning for the student model, as it only learns on exam- ples it makes mistakes upon and learns im- prove its own solution. On code generation, per- sonalised distillation consistently outperforms standard distillation with only one third of the data. With only 2.5-3K personalised examples that incur a data-collection cost of 4-6$, we boost CodeGen- montn--ooo--68 by 7% to achieve 36.4% pass@1 and StarCoder by 12.2% to achieve 45.8% pass@ on HumanEval.'
Researchers have explored such distillation idea (Taori et al., 2023; Wang et al., 2022; Xu et al., 2023b), by qucrying ChatGPT to generate task instruction and solution pairs, and using the col- lected data to finetune a student model. However, this standard distillation approach fits different stu- dent models to the same data distribution (teacher's prior), disregarding their unique abilities and ca- pacity. In education domain, personalised learn- ing which provides customized learning experience that adapts to student's learning progress and capac- ity, has proven highly effective and widely adopted TRoberts-Mahone et al., 2016; Shemshack and Spector, 2020). Inspired by such finding, we hy- pothesize that personalised learning is also benefi- cial for model distillation.
In this work, we propose personalised distilla- tion and empirically evaluate its effectiveness in the domain of code generation. Similar to standard distillation, we first employ ChatGPT to generate task instructions accompanied by unit test cases. Then we follow three steps for personalized distil- lation as shown in Figure 1. First, let the student model attempt to solve the task. Then, we evaluate the student's attempt with unit test cases and get execution feedback. If the execution feedback con- tains errors, in the final step we prompt the teacher model (ChatGPT) to refine the student's attempt.
1 Introduction
Recently, powerful close-sourced large langauge models (LLMs) including ChatGPT, GPT-4 have become predominant, accumulating over 170 mil- lion users within 5 month of its launch. Such close- sourced LLMs demonstrate strong performance in a wide range of tasks, from improving writing pro- ficiency to code generation. However, due to their closed-source nature, concerns have been raised regarding factors such as the availability of these services, high associated costs, concerns on ethics and safety, and potential data privacy implications,
Such data collection process makes the learning experience both interactive - as the student partic- ipates to make attempts, and personalised both the input (tasks) and output (refinement data) are customised to the student. Essentially, personalised labeled data help the student to refine its own policy. rather than adopting new prior of the teacher.
"These authors contributed equally to this work OOur codes will he available at https: //gi thub. com/ sa lesferree forceeeeeee... Peer/eersssss still
With the personalized code data as target out-

---

Task: write on remove_ remove_dupiicates diplicates to remove duplicated nems of list
Task:write function remove. ddppicates to remove duplicated items of list
1 standard answer
del remove_ duplicatesfliss:: new. for item list
code generation
X
personalised
Student
refinementt
if item not new list. (pppand(tem) new [is..oon) return Jist
Teacher
def remove duplicates(kett lstt)
def remove, duplicatesllitt lis(sst((s)))
X esecution feedback
TypeEiron: list object
Student
Teacher
Figure 1: Overview of our framework. Left: standard distillation. given problem for the student to learn Right: personalised distillation. solve the task. 2 Executor evaluates generated code with unit test cases.
! Teacher generates standard answer
Student first generates its own attempt to Teacher provides adaptive refinement
given student's attempt and its execution feedback.
put, we construct three variants of finetuning data (i) PERsD data which formats it as a typical text- to-code generation task, (ii) PERsD-refine which treats it as code-refinement task, given task in- struction, incorrect code and execution error feed- back (ii) PERsD-combine which simply combines PERsD and PERsD-refine finetuning data, i.e. code generation and refinement tasks.
Methods
Personalised
Interactive
Code-related
Alpaca
X
X
Vicuna
X
Baize
X
WizardLM
X
WizardCoder
X
Lion
Input
PERsD
Input Output
V
Table I: Related work on distillation from ChatGPT
We collect 10K standard distillation examples and around 2.5-3K personalised examples for pre- training. Through zero-shot evaluation on Hu- manEval (Chen et al., 2021) and MBPP (Austin et al., 2021), we observe that all PERsD variants consistently outperform their counterparts which use standard distillation. This compelling result strongly validates our hypothesis regarding the advantages personalized distillation. Ablation studies further reinforce our hypothesis, uncover- ing intriguing properties such as the benefits of multi-round personalized distillation and the ability of our models to leverage execution feedback for self-correction Notably, personalised distillation boosts the state-of-the-art open-sourced pretrain model StarCoder (Li al., 2023a) significantly by 12.2% to achieve 45.8 in pass @1 and 82.3 in pass @1 100 on HumanEval.
ardLM(XU et al., 2023a) and WizardCoder(Luo et al., 2023) iteratively prompts teacher model to generate more complex instructions. Their ap- proach can be seen as an orthogonal advancement that can potentially be combined with personalised distillation.
Lion (Jiang et al., 2023) proposes to incorpo- rate student model's answer and sample more hard tasks for which the student failed to solve. Thus, Lion can be considered input personalised dis- tillation only the input tasks customised for different student. Our approach differs as we pro- vide customization both on input and output, and we empirically show that personalising labels is critically beneficial.
2.2 Code Generation with Feedback
2 Related Work
Recently, there has been an increasing amount of research on exploring on how to use feed- back for an iterative and improved code genera- tion through code-refinement, Self-refine(Maduun et al., 2023), Self-debug et al., 2023b) and Reflexion (Shinn et al., 2023) are inference-tine methods which use powerful close-sourced L.LMs to generate better code from internal or external feedback. Although they show high performance,
2.1 Distillation from ChatGPT
Previous works have explored distillation from ChatGPT including Alpaca(Taord al., 2023), Vi- cuna(Chiang et al., 2023) and Baize(Xu et al., 2023b). However, these works can all be consid- cred as standard distillation as they do not consider the conditions and capacity of student model. Wiz-

---

Methods
Traning
Indorenes
assumption is that there are no ground truth targets and thus they try improve the LLM based on the assessment (critic) of multiple generated out- puts. However, such RL-st training will be less effective and efficient to supervised finetuning, es- pecially for challenging tasks with sparse rewards e.g. sovling math puzzles or coding tasks. Unlike these methods, approach can acquire "ground truth" outputs from personalised teacher, thus pervised finetuning can be applied which makes the learning effective and efficient, even for chal- lenging tasks like solving coding problems.
Pessoralised
Madel
feedBock
CharGP
Self-refine
No Training
Self debug
No Training
V
Reflexion
No Training
X
/
Self
Standard GT
Self-cormes
eepplrationn
ILF
Human labeled
PERvDereline
ChaiGPT
Table 2: Related work Code Generation w/ feedback
these methods are limited as they require access to close-sourced LLMs.
Self-edit (Zhang al., 2023) trains separate code editor to rectify generated code from base LLM. The training label is from original gold answer, thus not label-personalised. Similarly, Self-correct (Welleck et al., 2022) trains a separate corrector model to rectify the output from a fixed genera- tor model. However, the training label is from self-exploration of the corrector model: sampling multiple refinements and choosing the one leading higher reward. Finally, ILF (Chen al., 2023a) collects human-annotated code refinement data to train a separate refinement model on it. Fhe refine- ment model is used to generate text-to-code data for finetuning the code-geme oode-generation LLM. Unlike ILF, our approach is more scalable as we do not re- quire human annotation and our personalized data proves significantly more effective than ILF as we empirically investigate in $5.
3 Method
3.1 Standard Distillation
Assume dataset of code generation tasks D {tt, u}} where each problem (or task) consists of a task instruction t and a unit test collection u. During training, we have access to a teacher model To and a student model To- The objective is to distill how the teacher solves code generation tasks to the student model, in the context of D. For each task (4, u), we first query the teacher (t) with the task instruction, get direct generated code snippet cg. Then, execute the generated code Co against unit test cases u and get its execution feedback f ExEc(cs, u), where the EXEC function returns passed if the code passes all the unit tests, otherwise it returns an error message from the executor. By filtering out the tasks where Ga do not pass all the unit tests (i.c., f + passed),
2.3 Reinforcement Learning from (Human) Feedback
clean dataset DSTAND = {tt, u. c)}, where each task consists task instruction suite of unit tests and a correct solution code e.
get
After the launch of ChatGPT, aligning LLMs to human preference has drawn tremendous attention to research communities. As one of the most influ- ential approaches in this direction, reinforcement learning from human feedback (RLHF) (Ouyang al., 2022; Li et al., 2023b), adopts an actor-critic framework, where the student model is optimized to generate responses to receive higher reward from the critic model. In InstructGPT (Ouyang et al., 2022), the critic (reward model) is trained from human annotation. Direct Preference Optimiza- tion (DPO) (Rafailov et al., 2023) drops the need of training reward model, by using reference LLM and offline trajectories to estimate the reward. Bhain-of-Hindsigee (Liu et al., 2023) converts hu- man preference annotations into simple natural lan- guage feedback, and thus turns RL optimization to conditional generation. In above methods, the
We then finetune the student model mo on {(u, c)} ~ DSTAND. where the input is the task instruction u and the output is the corresponding code solution We name this approach STAND.
3.2 Personalised Distillation
The STAND approach simply samples training examples (instructions and labels) from the prior distribution of the teacher model and feeds it to the student without considering the conditions of the student model. Inspired by modern education principles which advocates interactive and personalised learning experience, we propose personalised distillation: adapting teaching materials to student's 's current knowledge

---

Algorithm 1 personalised distillation for code gen- eration (PERsD-combined))
PERSD This variant takes the training data Prefine from PERsD-refine and replace the input of each data point from code refinement prompt to original task instruction. It thus trains the student model with personalised labels on code generation.
1: Input: Dataset Dsrwnn, student LLM Te. unit test ex- ccutor EXEC, refinement template Tainee teacher LLM
The
2: Dreine + 11 refinement data for finetuning
3: Dodde 11 i direct generation data
To illustrate the difference between personalised refinement and teacher's direct solution, we show real example in Figure 2. The top shows the per- sonalised refinement for the given task, while the bottom section shows the direct teacher's gener- ation for the same task. Note how the teacher's direct generation significantly different from the student model's attempt, while the teacher's refine- ment follows the student's attempt and improves upon it. We hypothesize that such adaptive refine- ment where the teacher aligns to student's 's genera- tion, helps the student to learn more efficiently and effectively, similar to how humans benefit from personalised learning.
4:
for (L, u. E DSTAND do
5:
69 TT())
I student generates co b exec. feedback for ce
6:
I < ExEC(Ce, u.)
7:
if f passed then If personalised refinement from teacher (f.(t.cg) co:f)
8:
9:
10:
If create refinement task instruction
11:
Irefine Tretne (t, co, f]
12:
if EXEC(Creaae U.) passed then
13:
Dietue insert( (Latnne Cusine))
14:
Deede insert( ct)
15:
end if
16:
end if
end for
18:
mT- FINETUNE( Desine Doste)
and capacity. We propose three variants:
PERsD-combined Algorithm shows detailed steps for PERsD-combined This method takes the standard distillation dataset DSTAND from $3.1 and first lets the student generate solutions for each task. Then filters out the tasks where the student model can already solve correctly. For the remaining tasks, it obtains the teacher's personalised refinement con- ditioned on the student's 's attempt and its exccution error feedback, and only keeps the tasks where the teacher's s refinement is valid (i.e., passes all the unit test cases). Figure 1 visualizes these three steps. For this final task-set, we create two datasets: i) Deode containing task instruction as input and teacher's direct answer as output, and ii) Drefine containing task refinement instruction as input and personalised refinement answer output. The task refinement instruction (line 9 in Algorithm is created by concatenating task instruction f, student's 's attempt cg and its execution feedback with refinement template Trefine (More details in Appendix C). Such refinement instruction turns standard code generation into a code refinement task, teaching the student how to refine its own solution. PERsD-combing then finetunes the student model on Drefie combined with Doxte-
Input
Output
[Task]
[Teacher's Refinement]
Create program to to compare two lisss and identify all the elements present list
def eloments not commonja, bj:
common_ elements =
the Example: 1.2311
for element
(1.4)
iTelement
common_eie wwannonneemmett.oo oppenctelmment for element
def elements, not, commonia, bj:
if elleement not inar:
(Student's Generation]
common, elements aspanctelkment) return cormonnoleemets
di elements, not commonts, common, elements = for etement in if eelemenn not in l:
common alements appeddiipnnd elementsss
[Feedback]
ERROR: AssertionError INPUT: 21,22444
OUTPUT: [3]
EXPECTED:
[Tank]
(Teacher's Direct Generation)
progrem to compane and idontity all the cloments present one lit and not present in the other.
def elements, pommentss_nu not_commonta, bl: sntlal
sct son(b)
Example: assert) (323, m- [1.4]
symmetric_dfeeeeeee differeece
not_commenia,tee bl:
Figure 2: Example: (Top) Personalised refinement from student's attempt and execution feedback; (Bottom) Di- rect solution generated by teacher conditioned on task.
3.3 Iterative Inference
Let Drest = [(t,u)) denote our test set for inference, where each data point (t,, u) consists of a task instruction and suite of hidden unit test cases We also assume that the task instruction contains some simple unit test cases in its doc-string (as often seen in code generation instructions), which we can extract and format using qule-basedd heuristics to obtain a suite of seen unit test cases theen (More details in Appendix A).
PERsD-refine Similar to PERsD-combinee this variant follows line 1-15 of Algorithm collect refinement data Drefne However, it differs from the above model it only uses D Prefin finetune the student model.

---

For single-step inference, we use the standard approach to evaluate pass@k. Specifically, for each task i, we query the model n times with the task instruction: cg 6 (t) for - n. Then, following (Chen et al., 2021), we estimate pass@k from the number of attempts that passed the hidden unit test cases: ExEC(E): u) passed.
answers as well directly solving the task. The difference is that in InpD-combined, the labels
for both code refinement and code generation are taken from teacher's ' direct solution c.
4.2 Pretraining Data Construction
Multi-step inference If the model To has been trained to rectify. following our approach in PERsD-refine or PERsD-combine, and if unit tests are available during inference, can per- form 2-step inference: for each generated at- tempt ca in 1-step, we first get execution feedback feeen < ExEC(C): useen). If freen passed, we reuse the original attempt as the 2-step at- tempt. Otherwise, we create a refinement in- struction + (1efine ch. freen) following the ap- proach in PERsD-refine or FERsD-combing and query the same model with the refinement instruc- tion for 2-step attempt: Cg,2-stop +- (t())). We then compute pass @k over the 2-step generations similar 1-step inference.
To construct pretraining data, adopted the data collection process in code-alpaca(Chaudhary 2023) and used a set of 374 seed tasks from MBPP (task-ids 601-974) as in-context prompt to query ChatGPT for novel code generation tasks. This set increases the likelihood of ChatGPT gen- gen- erating python codes.
Through this process. we obtained a corpus 20K code generation tasks from ChatGPT each comprising a task instruction and the corresponding generated code, which is typically single python function. Next we show each generated instance to ChatGPT again and prompt test-case inputs (i.e. input argument values) for the python function. We then parse and format the generated test-case input and execute the gen- erated code on it obtain an output. Thus, out 20K, for 14880 instances we could successfully generate and parse 5 unit test case inputs and for 10172 instances we were able to successfully exe- cute the generated code and obtain outputs on all 5 inputs. This final corpus of 10K code generation tasks, each comprising task instruction and the corresponding generated code along with 5 unit test input and outputs forms our standard distillation dataset D DSTAND-
generate 5 unique
4 Experimental Setup
4.1
Baselines
The first baseline is STAND, the standard distilla- tion approach mentioned in $.
To measure the effectiveness of personalised labels quantitatively, we also compare with input-personalised distillation baselines as well, where only the input tasks are seleeted a manner customized to the student's s abilities. However, the output labels are not personalised, as they are taken from teacher's direction generation c instead of personalised refinement Grefine- We start with Doode from PERsD-combined and have three variants:
To collect personalised distillation data, we fol- low $3.2 to first ask the student model to generate output code per task, setting sampling temperature to 0.3. We then evaluate the student's attempt and only keep the tasks with the wrong generations (i.c. the ones which failed any of the unit test-case). We use this to query ChatGPT for personalised refine- ments and only retain the valid refinements which passed all unit tests. Our prompt to ChatGPT con- tains the original task instruction and code from DSTaN along with the student model's generated code and execution feedback (compiler errors or unit test failures). Our instruction to ChatGPT is to generate correct solution that rectifies the errors and is closest in semantics to the student's code (More details in Appendix B). Table 3 shows the statistics of personalised data construction process.
INPD We finetune the student model on {(t,c)} Deode. where the input is task instruction and the output is a code solution. This variant is more customized than STAND as filters out the tasks which the student can already solve correctly.
INPD-refine Similar PERsD-refine, InpD-refine trains the student model to rectify its wrong attempt. The difference is in InpD-refine, the refined code is from teacher's direct solution c, instead of personalised refinement Crefine-
INDD-combined Similar to PERSD cEnsD-combined, InpD-combined trains the student on rectifying its

---

Suddet Model
Wrong Atterne
Validaeed Por-
Personalised labeled-data is generally bet- ter than standard data Comparing PERsD- combine to InpD-combine, we find PERsD- combine outperforms InpD-combine in all settings, often with a significant margin (two backbones, two datasets, two inference steps, 4 pass @k met- ric). Similar observation holds true when compar- ing PERsD-refine to InpD-refine (except for 2/32 settings), and PERsD to InpD. Thus, we conclude that PERsD variants are generally significantly bet- ter than their InpD counterparts, providing strong evidence that personalised labels are more effective for the student model to learn than standard labels.
Stodent
sounlised Tasks
ChooCiea mono 6B (Nijkamp et a.., 2023)
6.5k
3.25K
5.55
ChceCies mono 6B (round2)
4K
1.K
4
mopo
2.88
SteCoddr ilietal, 20236)
4,3K
2.58
1435
Table 3: Statistics of Personalised Data Construction
4.3 Model Evaluation
We evaluate our models on two datasets: Hu- maheval(Chen al., 2021), which contains 164 Python problems, and the subset MBPP(Austin et al., 2021) sanitized set that has no overlap with our MBPP seed tasks for pretraining data collec- tion. This corresponds to dext+validation--rrmmp splits of MBPP-sanitized and consists of 306 Python problems. We use nucleus sampling with temperature 0.2 to generate 20 candidates per task for estimating pass@I 1, and with tempera- ture 0.8, 100 candidates per task for estimating pass @ 1/10/0005000000
PERsD outperforms StanD with less than one- third of its data We observe that PERsD out- performs StanD for every pass@k k on both 16B and 6B CodeGen-mono backbone across both Hu- manívaal and MBPP, even though StanD has 10K data and PERsD has only 3.3K and 2.8K exam- ples for CodeGen-mono-6B and 16B. The only exception is in the setting CodeGen-mons mont-eee-mooo--eee 16B, MBPP, pass@1. I, where StanD edges out PERsD by 1.2 points. Given that our pretraining data is constructed from seed tasks taken from MBPP, we hypothesize that StanD might enjoy an unfair ad- vantage due to its having three times more data, making it more susceptible to data leakage. We verify such hypothesis further in $5.2. In summary, with PERsD outperforming StanD in 15 out of 16 settings while having less than third of the data, it's evident that personalized labeled data makes the learning more efficient.
For multi-step inference, we first extract the "seen" unit test-cases from the doc-string of the task instruction (More details in Appendix A). Next, we generate output samples in the usual code-generation style forming the set of 1-step gen- erations for each instance. Each of these candi- date generations are then exccuted on the extracted "seen" unit test cases to obtain a refined code, thus forming the set of 2-step generations.
4.4 Pretraining Setup
For all experiments with CodeGen-mono-6ß back- bone, we use effective batch size of 1024 and pre- train for 20 epochs. For backbone as CodeGen- mono-16B, we use effective batch size of 1024 and pretrain for 3 epochs, as the training converges much faster than CodeGen-mono-6h For PERsD- effective
Multi-step inference consistently improves an- swer quality For PERsD-refine and PERsD- combine models, we find that 2 step inference con- sistently improves performance on HumanEval and MBPP. This shows the models successfully learn how rectify its solution based on execution er- ror feedback. Note that InpD-refine yields worse accuracy with step inference on HvaanEval pass @ 10/20, strengthening the advantage of per- sonalised labeled data over standard labeled data.
combine with StarCoder model, batch size of 1024 and pretrain for 8 epochs, which results in similar training loss as CodeGen-monn monoo- 16B. We implement using HuggingFace transform- ers(Wolf et al., 2020) and DeepSpeed Zero (Ra- jbhandari et al., 2020). All experiments are con- ducted on cluster of 8 A A100-40GB GPUs.
5.2 Train-Test overlap analysis
As observed in Table 4, Persa-variants enjoy higher average improvements over their InpD coun- terparts, on HumanEvan than on MBPP. To delve deeper, conduct a data overlap analysis. For each test task, we extract the most similar training task and use GPT-3.5- -ttrbb to score their seman- tic similarity, with indicating no relation and
Experimental Results 5.1 Main Results
We empirically test the hypothesis that personalised distillation helps student model learn more effec- tively, by comparing PERsD models with baseline distillation methods (InpD, StanD) in Table 4.

---

(a) Backbone as CodeGen-mono-68
Method
Backbone
Sec "leak")
Similarity
HumannEvll
Mithots
wibrr
Pasc
Pa
Pas 10
StanD
6B,
6.1%
0.22
Hamantival
PERsD
3.6%
SuaD
J0K
32.41
44.70
45.67
40.26
byp
31,65
44.35
36.76
PERsD
16B
3.05%
0.22
3.39
29.70
29.70
43.52
41.99
51.29
47.80
55.29
53.51
MBPP
comtined
6.5K
30,15
PERID
3.5K
34.63
49.36
55.34
60.41
StanD
6B,16B
18.24%
0,40
3,3
32.35
49.15
56.87
PERsD
6B
8.47%
0.30
sumbinal
6.5K
33,61
35.53
44.64
49.67
49.96
55.67
55.23
61.21
PERSD
16B
7.49%
0.30
MBPP
StarD
J0K
43.11
15.224
20.07
1,6251
lepli
45.59
Table 5: Train- Overlap Analysis. 6B/16B denotes CodeGien-mono-(6// backbones. %c"leak") denotes the percentage of test data that are semantically leaked in training data. 'Similarity" represents the average similarity score (range: 0 to 1; higher values indicate greater similarity)
5.5K
46.44
47.81
66.25
66.45
67.60
71.44
11.88
15.23
combinod
65k
42.00
47:25
16.00
62.17
66.40
46
PERLD
3.33
45.47
59.90
64.85
69:73
soface
3.3%
48.24
52.66
63.65
65.49
10.00
73.34
73.6
77.02
6.5K
42.77
56.91
62.29
66.89
65.22
(b) Backhone as Code 16B
Mathols
P 1
Pas 10
2tep=2
wop-i
siep2 step-l
wep-2
HamanEval
StanD
10K
33.96
50.56
57.89
63,82
roughly maintains its performance on other set- tings, mixing StanD data to PERsD-refine signifi- cantly deteriorate its performance (except pass @ inf-step=2 on HumanEval). We conjecture that as 67.13
(np
2.5K
15.66
55.05
57.47
refne
285K
30.55
31.28
1:0.40
45.15
55.00
56.52
16.31
contined
5.6K
31,66
26.09
50.05
53.89
56.75
00.07
03.7H
5585
PERSD
2.8%
37.74
56.57
288
35.77
37.99
51.36
54.15
58.07
60.02
65.17
StanD has much larger data volume than PERsD- refine, it overwhelms the student training on stan- dard distillation. However, combining with bal- anced input- personalised data can be beneficial, we observe from the good performance of PERSD- combined in Table 4 on CodeGen--ono- 16B.
StarD
10K
45,90
62221
71.38
(npl)
2.6K
4h.27
55-45
60.61
66645
2.5K
combinee
5.6K
47.39
55.99
59.06
66.15
65.08
70.26
.67.00
70.35
PERLD
285
47.68
71.56
36.22
<cfine
288
91.50
56.21
06.82
71.86
72.05
76.78
76.08
51442
sumbiral
5.6K
$1.44
56.44
66.45
21.31
76.43
96.04
50.20
Table 4: Comparing PERsD models to StanD & InpD
Inf Step
Pacsit
Pes
Homannival
StanD+ InpD reline
30.39
40UU
44.20
54.23
58.51
indicating complete semantic overlap (further de- tails in Appendix D). Table 5 reveals more overlap in MBPP than HumanEval, and more overlap for StanD compared to PERsD. This overlap could be why StanD surpasses PERsD in the 1/16 setting (CodeGen-mono--lee
StanD InpD refine
29.15
39:83
44.07
54.35
59.76
SanD PLXSD reline
32.15
43.82
18.56
59.55
64.02
refine
$6,07
SomD + Inpi-
30,87
42.85
47.90 17.90
56.21
60.8
Sunl Infine
30.12
43:71
43.42
54.60
66:02
2
SunnD Prstafleetine
47.80
33.35
49.35
56.87
70,88
MBPP, pass 1), as StanD has an unfair advantage of having significantly more data leakage. In addition, we test our meth- ods on clean-MBPP where the leaked data points are removed, then PERsD becomes almost on-par with StanD in this specific setting while having larger margin over StanD on the rest 15/16 settings (from 4.8 points average margin to 5.9 points, more details Appendix E). Altogether, this overlap analysis, coupled with results from cleaned MBPP. further underscores the advantages of personalized distillation.
MBPP
StantD tnpi reline
42.60
53.18
56,99
02.11
63:07
StaaD
44.00
54.12
34
StaiD PERaDeefiie
45.63
55.20
56.38
63.02
65.36
PERoD-refle
48.24
63.65
69.00
78.16
81.70
StaiD Iep-scllie
46.32
58.84
62.30
71.23
StaaD tap refise"
58.18
62.00
6895
StarD PERADD
48.44
58.37
02.47
70.64
73.20
PER-D reluee
52.45
68.49
70.34
83.72
85.42
Table 6: Ablation on mixing StanD, with Backbone as CodeGen-mont 6B. InpD-refine* denotes using all 6.5K tasks where the student model made mistakes, which covers around 3K more tasks than InpD-refine.
Similarly, in Table 7 we show another ablation: that mixing InpD data with PERsD roughly main- tains the performance on HumanlEval but degrades on MBPP. This shows personalised labels are of higher quality and mixing non personalised labels for the same task generally hurts performance.
5.3 Effect of mixing StanD and InpD data
Table 6 shows the ablation study on mixing stan- dard distillation data to PERsD-refine and InpD- refine: while mixing standard data to InpD-refine improves its 1-step performance on MBPP and

---

Methods
Pass
5
Pascei
Pussúi 100
forms PERsD-combine. This result aligns well with our expectations that code-rectification needs the execution feedback to guide the refinement.
HumanFEvll
PER&D
34,63
49,34
55.34
65.56
67.93
PERsD Inpl)
34.88
48.35
54.06
64.88
68.90
MВPP
Paee
Pavo6850
Paaede
PERsD
45.47
59.90
64.85
76.05
80.07
Step
HumanExal
PER&D Inpl>
43.84
59,02
63.77
71.09
74.84
PERsD-combine
3381
44.64
49.96
61.75
70.73
PER-Dl-umbine"
33.29
45.47
50.90
62.87
Table 7: Ablation on PERsD mixing InpD with
PER.D.cambinc
68.16
CodeGen-mono 6B as backhone
combine"
MBPP
PEReD.cumbinc
42,77
56.91
61.43
68.84
70.67
Round
Inf Step
@Pss
Ps @
Pass 10
50
Pass @ 100
PERaD-combine combine"
44.76
56.95
60.85
68.57
71.57
HumanEEal
PERdD PER=ID-combine
48.92
62.39
66,89
75.09
77.25
33.81
44,64
49.96
61.75
70.73
PERaD combine"
47.83
61.28
65.54
73.03
75.49
32.74
45.50
51.52
71.95
35.53
49.67
$5.67
68.16
77.44
Table 9: Ablation removing execution feedback with CodeGen-men mono 6B as backbone. PERsD-combine* denotes combined personalised distillation without exe- cution feedback in input prompt.
2
36,75
49.71
56.13
70.24
75.00
MHPP
42.77
56.91
61.43
68.84
70.67
2
45.07
57.75
62,27
70.49
72.55
2
48.92
62.29
66.89
75.09
77.25
5.6
Cross-Model Personalised Distillation
49.59
63.43
68.30
78.10
To investigate whether personalised distillation data of one model can be benefical to another, we conduct an ablation in Table 10 by using PERsD-combined data of CodeGen-mono-6i to train CodeGen-monn--166 The results show that such cross-model persionalised data do not per- form as well as real personalised data: leading to consistent performance drop by large margin. This finding reinforces should be tailored to the specific student model, personalized data suitable for one model may not necessarily benefit others.
Table 8: Ablation on multi-round distillation on PERsD- combined with CodeGen- mono 6B as backbone
5.4 Multi-round Distillation
After finctuning the student model with the per- sonalised distillation data, can we perform an- other round of personalised distillation, on the new model? We show such an ablation study in Ta- ble 8. Encouragingly, we find PERSD-combined round-2 generally outperforms PERsD-combing round-1 by a modest margin. This improvement provides further evidence of the benefits of per- sonalized learning, even when applied to models trained with personalized distillation. These find- ings suggest the intriguing possibility of an online active version of personalized distillation, where data collection and model training occur simulta- neously ensure each batch fully personalized and has higher sample efficiency. However, we will leave such intriguing exploration for future work.
notion that learning data
Passd:10
HunsuEval
dil
48.04
49.96
01.75
70.73
Codetion l68"
47.81
54.58
73.95
Clodetion mone 10U
36.40
53.57
6881
74.64
79885
CodetGnn mono bil
15.35
49.77
35.77
68.16
77.04
CodeGGnn mono- 16B
37.74
$5.80
63.37
77.14
81.10
MRPP
Clodetizn mono-58 5B
42.77
56.91
61.43
70.67
Codetion 168*
00.14
72.31
74.19
Codetion mono 10U
51.44
66.45
71.64
80,62
82,93
ColkGen-mmoo-6
48.92
62.29
66.89
75.00
77.25
5.5 Utilizing feedback for multi-step Inference
CodeGan-mons
65.31
70.02
78.70
Cludetion
56.44
71.31
86.76
To better understand the role of execution feed- back during training and multi-step inference, we show an ablation study in Table 9, where we compare PoRsD-combine with a specific variant (PERsD-combine*)) that excludes feedback dur- ing both training and inference. observed that PERsD-combine performs comparably PERsD- combine on HumanEval and slightly better on MBPP for 1 1-step inference. However, for 2-step inference, PERsD-combine** consistently underper-
Table 10: Ablation on cross-model personalised distil- lation with PERsD-combined. GodeGGe-monne 1 means distillation data from CodeGen GodeGGen-mono-G2 mono-6B.
5.7 Comparison with other Feedback-based Code Generation Models
Comparison with ILF (Chen al., 2023a): In order to compare with ILF, one of our closest re- lated work, we experiment on separate setting:

---

starting with full MBPP dataset (974 tasks) and use Task-Ids 11-111 as test split and remaining 863 as training data. On the training set, our stu- dent model CodeGen-6B (same as ILF) generated wrong attempts on 562 tasks, which were shown to ChatGPT along with the task instruction and execu- tion error feedback to eventually collect 288 valid personalized code rectification labels.
Pissie
Cloasl-sonee models
LAMDA
1378
14.0
PaLM
5408
26.2
76.2
Codes
12R
28.8
46.8
72.3
codesnssmon-O00
33.5
davinci
47.0
GPT335
68.1
phi-1
1.38
50.6
GPT-4
67.01
Open source usodels
ColeCeee
138
22.9
The original MBPP cext-to-code personalized
LLaMA
data and this collected code-refinement data for the 288 tasks respectively form the finetuning data Doade and Dreline on which we train models PERsD and PERsD-refine. We further combine Doode and Dretine to train PERSD-
StrCOoder
150
33.6
CodiCen-mmnu
681
26.1
42.3
65.8
CodeGin mono
168
29.3
49.9
75.0
I
3500
WaardCaderr
158
57.3
MBPY Test Set
CodeGen-mnn (PElRsD-comminn))
68
33.8
50.0
70.7
Method
Coot
Pawce i Passo to
CodeGen mono (PERSD combised)
168
36.4
60.8
79.9
ILY >4KS 36
68
StuCoder (PERoD combiued)
15U
45.8
68.3
82.3
PEXSD
40.8 87.4
Table 12: Results of pass @ ((%) on HumanEval
combined
0688 47.8
6188
Table II: Comparison with ILF
that our model 5K data examples while Wiz- ardCoder uses 78K. As mentioned in $2.1, Wiz- ardCoder an orthogonal approach that can be integrated into personalised distillation.
combined. Our experimental results in Table Il show that all PERsD- variaatss significantly outperform ILF by 11.8% at pass@1 at a cost le-4 times lower than ILF, thus showcasing the lack of scalability of ILI--style models.
6 Conclusion
In this paper, we introduced personalized distilla- tion as a method for collecting customized labeled data that adapts to the capacity of student models, resulting in more effective learning. We demon- strated the advantages of personalized distillation standard distillation in the field of code gen- eration, achieving superior performance on both the HumanEval and MBPP datasets. Through com- prehensive ablation studies, we confirmed that per- sonalized distillation leads to higher data quality, benefits from multi-round distillation, and enables models to leverage execution feedback for self- rectification. We believe personalized distillation represents exciting step towards better distilla- tion of closed-source LLMs open-source models.
Comparison with Self-Edit: Since Self-Edit (Zhang et al., 2023) uses a trainable CodeGen- 350M code editor model and a frozen code- generation model, our experimental setup is not directly comparable with theirs. However, INDD-refine and INDD-combined models actu- ally be considered as very close counterparts version of Self-Edit with shared code-generation and code-refinement model and CodeGen-6B back- bone. The consistent performance improvement of the personalized distillation models over the input- distilled ones across the board, alludes towards the prospect that PERsD-models are indeed more ef- fective than Self-Edit style models.
5.8 Comparison with SOTA Models
Fianlly, we compare PERsD-combine models with open-source and close-sourced state-of-the- art models on HumanEval in Table 12.We find that PERsD-combing methods can significantly im- prove the backbone model, with performance gain of 6.2 points for CodeGen-mond 6B (8.4% er- ror reduction), 5.9 points for CodeGen-mono 16B (8.3% error reduction) and 12.2 points for Star- Coder (18.4% error reduction). Moreover, Star- Coder with PERsD-combined, outperforms other open-sourced models except WizardCoder. Note
Limitations
In this section, we discuss some limitations of this paper and future directions make it more valu- able:
On Data Scale For a fair comparison, we have conducted all experiments based on the same 10K DStAnd data (introduced $4.2) and the correspond- ing personalised data processed from DSTAND are of size 2-3K as shown in Table 3. However, as we have proven personalised distillation supports

---

more effective and efficient learning, is intriguing investigate how well does personalised distilla- tion scale with the data size. For example, we scale personalised distillation data to 50K, how much more performance gain will PERsD meth- ods receive compared to InpD and StanD with the scaling of data size.
Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojciech Zaremba. 2021. Evaluat- ing large language models trained on code. CoRR, abs/2107 107.03374.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou, 2023b. Teaching large language mod- els to self-debug. CoRR, abs/2304. 1.8
Online Personalised Distillation As discussed in $5.4, conducting second round personalised distillation continues to improve a student model that is already trained with PERsD-combine. Such observation suggests the potential of an online ver- sion of personalised distillation, which collects a batch of personalised data on-the-fly with the teacher model, after each optimization step during finetuning. As we have proven that true person- alised data is more beneficial than standard data or cross-model personalised data ($5.6), such on- line personalised distillation will in-principle maxi- mally benefit from personalised distillation, as each batch of training data is fully tailored to the student model.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao 7hang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. 2023. Lion: Adversarial distillation of closed-source large language model. CoRR. abs/2305. 12870.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff. Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang. Olivier Dehaene, Mishig Davaadorj- Joel Lamy- João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze. Ming-Ho Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy V, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Ur- vashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero. Tony Lee, Na- dav Timor, Jennifer Ding, Claire Schlesinger, Hai- ley Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Ander- son, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jer- nite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023a. Starcoder: may the source be with you! CORR, abs/2305.06161.
References
Jacob Austin, Augustus Odena, Maxwell I. Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program synthesis with large language models. CoRR, 0os/2108.00732.
Sahil Chaudhary. 2023.
Code alpaca: An inttructionfoollwwiin Ilama model for code genera- tion. https:/ /ttps://gittiie //ggtttttt..////////////t com/sahil280114/ codealpaca.
Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos. Jun Shern Chan, Samuel R. Bow- man, Kyunghyun Cho, and Ethan Perez. 2023a. Im- proving code generation by training with natural lan- guage feedback.
Zihao Li, Zhuoran Yang. and Mengdi Wang. 2023b. Re- inforcement learning with human feedback: Learn- ing dynamic choices via pessimism. CoRR. abs/2305.184l
Mark Chen, Jerry Tworek. Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sas- try. Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert-Voss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantamu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles
Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. 2023. Chain of hindsight aligns language models with feed- back. CORR, 0to///32.002666
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xi- ubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol- instruct. CoRR, abs/2306.08568.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang.

---

Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. CoRR, abs/2303.1766
Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 38 38-45, Online. Association for Computational Linguistics.
Erik Nijkamp, Bo Pang. Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-tum program synthesis. ICLR.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023a. Wizardlm: Empowering large lan- guage models to follow complex instructions. CoRR, abs/2304.12244,
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In NewrIPS.
Canwen Xu. Daya Guo, Nan Duan, and Julian McAuley. 2023b. Baize: An open-source chat model with parameter-efffcient tuning self-chat data, arXiv preprint arXtv:2304.011990
Rafael Rafailov, Archit Sharma, Eric Mitchell, Ste- fano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. CoRR, abs/2305.18290.
Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023, Self-edit: Fault-aware code editor for code genera- tion. CoRR, abs/2305.04087
A Details in Multi-step Model Evaluation
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: memory optimizations toward training trillion parameter models. In SC, page 20. IEEE/ACM.
As the doestrings are formaaddtedd in HumanEval, we write simple rule-based parsing code snip- pet to extract its seen unit test cases. On average per task, there is 2 seen unit test cases and 4.2 unseen unit test cases, The overlap between seen and unseen tests is 11.33%. For MBPP, since con- ventionally the instruction prompt is constructed by taking the task description and example usages (from the unit test cases) as part of the doc-string, we consider all the unit test cases to be "seen" and use them for multi-step inference.
Heather Roberts-Mahoney Alexander Mcans, and Mark J. Garrison. 2016. Netflixing human capital development: personalized learning technology and the corporatization of k-12 education. Journal of Education Policy, 3 31(4):405-477
Atikah Shemshack and Jonathan Michael Spector, 2020. A systematic literature review of personalized Iearn- ing terms. Smart Learning Environments, 7(1):1-20.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal rein- forcement learning.
B ChatGPT Prompt Template for Personalised Distillation
In Figure 3, we show the prompt template we use to query ChatGPT for personalised refinement. For each task example, with task instruction 4, unit test cases and correct code c, we query ChatGPT API with two turn conversation history.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stan- fnolloition-folloiingg llama model. https:// stanford_alpaca
ford alpaca: ///g/tttubbb//t com/tatsu-lalt
For the first turn, we use the template in Fig- ure 3a and replace <TASK>, <HEADER> with the actual task instruction t and function header ex- tracted. This is added to first turn's user input and the correct code cis included as first turn's 'ss assistant output. For the second turn, we use the template in Figure 3b and replace <CODE>> <ERROR with the student model's attempt and its execution feed- back. This is added to second turn's user input and we query ChatGPT with the constructed conver- staion history to get second turn's 's assistant output as personalised code refinement.
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra. Alisa Liu. Noah Smith, Daniel Khashabi, Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions.
Sean Welleck, Ximing Lu, Peter West, Facze Brah- man, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2022. Generating sequences by learning to self-correct. CoRR, abs/221 1 100053.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz,

---

Complete the python function solve the following task. The input format
Rectify the below code for the given task based the arrors obtained the unit tests.
TASK: << Some Task description >>
TASK:
HEADER: << Header the python function definition. Your output should start with this header
<<TASK>>
<<CCDE>>
ERROR:
Example
<<ERROR>>
TASK: <<TASK>>
<<TASK>>
<cHEADER>>
HEADER:
<<HEADER>>
Figure 4: Prompt template for code refinement finetun- ing.
CORRECT CODE:
(a) TurnI Prompt Template
Now act like a python expert, whose objective to rectify a given incorrect for the task, based the error obtained unit testing.
Firstly, we have performed exact string match and found no data leakage in any test data (Hu- manBval/MBBP).
INCORRECT CODE:
To measure the semantic similarity between training/test tasks, did the following:
<<CODE>>
ERROR: <<ERROR>>
1. For each task in the test (MBPP/HumanEvall we retrieve two closest training tasks (based on cosine similarity of starcoder embedding & tf-idf vectors of task description).
Instructions for Code Rectification:
Based above errors, rectify the lagic of the incorrect code and write complete python function to correctly solve the task.
If the code is significantly wrong then you can completely re-write the code, otherwise try to come up with correct version that closest
given code.
2.
We use gpt-3.5-turbo-1e to identify whether there is a data leak between train and test instance by classifying the pair into ("leak", "somewhat similar", "somewhat not similar", "not related")). We use prompt with instruc- tions and manually created few-shot exam- ples and ask gpt-3.5 to generate the reasoning and categorization. We manually examined several examples per category to ensure the reasoning and judgment is done correctly and consistently.
All output code should be inside code block and start with the given header.
RECTIFIED CODE:
(b) Turn-2 Prompt Template
Figure 3: Prompt templates to query personalised re- finement. Top(a): prompt template for first turn con- versation, Botton(b): prompt template for second turn conversation.
Prompt Template for Code Refinement Finetuning
Map the similarity categories to 0-1 similarity- score ("leak" -> 1, "somewhat similar" >> 0.75, "somewhat not similar" -> 0.25, "not related" -> 0) and show the mean score and % of cases classified as "leak", Note that StanD & PERsD have 10K & 3K training data spectively so their scores are different.
Figure 4 shows the refinement template Trefiee in- troduced in $3.2), which used construct input prompt for code refinement finetuning. we replace <TASK> with task instruction, <CODE> with the initial wrong attempt from student, <ERROR with the execution feedback, and <HEADER> with func- tion header extracted from task instruciton.
E Results in MBPP-Cleaned
In Appendix D, we find 55 data instances that are potentially leaked (with similarity score 1) in MBPP test data. In this section, we construct a new MBPP-Cleaned dataset, where the leaked data points are removed (originally 306 problems 251 problems after filtering). The results on this new MBPP-Cleaned dataset shown in Table 13. From
D Details in Data Overlap Analysis
This section describes the detailed procedures to conduct train-test data overlap analysis. The objec- tive is to assess the extent of data leakage in the test datasets originating from our self-constructed pretraining corpus.

---

the results, we can for setting CodeGen-mono- 16B, pass 1, PERsD becomes almost on-par with StanD (from gap of 1.21 to -0.17). For the rest of 15/16 settings on PERsD comparing with StanD, its average margin is increased from 4.8 points to 3.9 points. Besides, PERsD-refine on MBPP-Clcaned shows more consistent and sizable improvements over InpD-refine, with an average edge of +0.86 for step inference, and +1.91 for two step inference. Overall, with overlapped test data removed, PERsD methods show even larger advantages compared to StanD or InpD methods.
(a) Backbone as CodeGen-mono-6B
Proc
Or
of
s
MBPP Cleused
SuiD
10K
37.51
58.97
bnp
33K
38,80
-3.91
62.73
rofire
3.3K
$7.58
42.95
13565
62.19
55.52
67:00
69.02
71.96
PERUD
333
41.30
55..00
61.88
67.53
6.5K
38.56
43.75
52.78
57.04
57.35
61.78
61.52
66.19
(b) Backhonc as CodeGGen-mono 168
Mithads
Paod's
Pauco
MBPP-Clesaal
SuaD
J0K
43
57.53
50.02
66.12
25K
43.67
10.00
63.16
68.11
19.23
7126
73.66
76.55
PERLD
28K
42.05
02.40
50.90
74.10
74.00
sumbrnal
5.K
45.35
51.87
63.46
68.65
40:40
34.25
34.53
78.83
Table 13: Comparing performance of PERsD models StanD & InpD on MBPP-Cleaned


Reviews:
[{"review_id": "R6DgAMYmB3", "reviewer": "Reviewer_cUGP", "summary": "", "questions": "1. Is the 2-step refinement instruction for Multi-step inference in Section 3.3 generated with the help of ChatGPT? Does this go against the previously mentioned reduction of dependency on closed-source macromodels?\n2. How were the training data for Round 2 of Multi-round Distillation in Section 5.3 obtained? How is it different from the data of Round 1?\n3. I found that the experimental results of training CodeGen-momo-16B with the data of CodeGen-momo-6B in Table 9 are lower than the results of StanD in Table 4(b), does it mean that a separate training data must be developed for each model in order for the results to be better than the results of StanD? Does this suggest that this method does not generalize well?\n4. How was the Cost>4K$ figure for ILF in Table 10 obtained? I didn't find in the original article that it costs that much.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that R1 finds the motivation for “personalized learning” compelling and validated by sufficient experiments. However, we disagree with the arguments that the paper is “minimally challenging and not technically challenging”. We hope to clarify the importance and challenges of our approach with some additional information here:\n\n> “Training data not publicly available”: \n\nAs we mention in our paper, we will release all our model checkpoints, codes and training data upon acceptance. \n\n> “Minimally Challenging”:\n\nFrom ML perspective, standard distillation – both white-box and black-box (Section 2.1) – can be seen as forcing student’s prior to teacher’s prior, in spite of the fact that the two prior distributions could be very different. In our work, we explore a fundamentally different approach: adapting the student’s prior to its improved self, acquired by asking the teacher to correct the attempt such that the resulting distribution is still close to the student's original prior. We motivated such an approach from “personalized learning” in human education and validated it with comprehensive experiments.\n\nTo the best of our knowledge, such a personalized distillation idea has not been studied in the literature and it is a timely topic as the teacher model needs to provide personalized refinements (a non-typical task requiring general instruction following ability), which is only possible with large LLMs nowadays.\n\n> Not technically challenging. Why is personalized distillation orthogonal to WizardCoder?\n\nCode generation on HumanEval is still challenging for open-source code-LLMs (StarCoder only achieving 33.6% pass@1). Our PERsD can boost the performance to 45.8% pass@1 with only ~3k data points. The contemporary work WizardCoder achieves the SOTA among open-source models with 78k data points. \nWizardCoder is an orthogonal approach, as it proposes to generate more complex/diverse task instructions via evolve-instruct, while PERsD can take the data points generated by WizardCoder, and apply personalized distillation upon it. However, since the WizardCoder-78k data is not open-sourced, we couldn’t integrate it with PERsD.\n\n> Why not evaluated on more complex programming problems (e.g., APPS or CodeContest)\n\nHumanEval and MBPP are still quite challenging to smaller open-source LLMs (6B-16B). For example, StarCoder and CodeT5+ – two best models (at the time of submission), can achieve less than 35% pass@1 on HumanEval. Hence we conduct comprehensive evaluation on them across multiple settings, and validate the power of personalized distillation.\n\nWe are aware of the more challenging coding datasets like APPS, CodeContest, XCodeEval. However, preliminary results on challenging datasets like APPS shows that even WizardCoder can only achieve around 10% / 2% / 0.4% pass@1 on introductory / interview / competition level problems. As such, these competitive programming problems might require some additional effort like curriculum learning + self-critique/reasoning along with personalized distillation, which we will leave to future work.\n\n> Overlap of ChatGPT generated training data with HumanEval & MBPP\n\nWe have done a rigorous inspection of the possible overlap, as follows:\n\n\nFirstly, no exact match was found in training and test task instructions.\n\nTo measure the semantic similarity between training/test tasks, we did the following:\n1. For each task in the test (MBPP/HumanEval) we retrieve two closest training tasks (based on cosine similarity of starcoder embedding & tf-idf vectors of task description). \n2. We use gpt-3.5-turbo-16k to identify whether there is a data leak between a train and test instance by classifying the pair into (“leak”, “somewhat similar”, “somewhat not similar”, “not related”). We use a prompt with instructions and manually created few-shot examples and ask gpt-3.5 to generate the reasoning and categorization. We manually examined several examples per category to ensure the reasoning and judgment is done correctly and consistently.\n3. Map the similarity categories to 0-1 similarity-score (“leak” -> 1,  “somewhat similar” -> 0.75, “somewhat not similar” -> 0.25, “not related” -> 0) and show the mean score and % of cases classified as “leak”.  Note that StanD & PERsD have 10K & 3K training data respectively so their scores are different. \n\nResults:\n\n—----------------—----------------—----------------—----------------\n\nTest: HumanEval\n\n| method | backbone         | %(\"leak\") | mean similarty-score |\n|--------|------------------|-----------|----------------------|\n| StanD  | \\                | 6.1%      | 0.22                 |\n| PERsD  | Codegen-mono-6B  | 3.6%      | 0.18                 |\n| PERsD  | Codegen-mono-16B | 3.05%     | 0.22                 |\n\n\n—----------------—----------------—----------------—----------------\n\nTest: MBPP \n\n\n| method | backbone         | %(\"leak\") | mean similarty-score |\n|--------|------------------|-----------|----------------------|\n| StanD  | \\                | 18.24%    | 0.40                 |\n| PERsD  | Codegen-mono-6B  | 8.47%     | 0.30                 |\n| PERsD  | Codegen-mono-16B | 7.49%     | 0.30                 |\n\nFrom the table, we can see \n1. There is more overlap for MBPP than HumanEval, as is expected since a part of the MBPP data was sequestered away as seed for generating training data.\n2. StanD has more overlap than PERsD (or InpD). \n\nTo the best of our knowledge, we are (one of) the first group providing such semantic overlap analysis (unfortunately it’s not seen in Starcoder, CodeT5+, WizardCoder, etc). We suspect WizardCoder might have higher data leak, as they use pass@1 on humanEval as selection criteria of their generated Evol-instruct type finetuning data.\n\nWe will add the above semantic data overlap analysis to our final draft and provide more details in the appendix.\n\n> For MBPP, only the pass@1 result of PERsD-refine is a sizable improvement compared to InpD-refine,Pass@1 results for PERsD seem to be lower than those for StanD\n\nThis is definitely an important observation. We appreciate the Reviewer's attention to details and apologize for not properly addressing it in the paper. For better clarity, we first summarize the average improvements PERsD achieves over StanD and InpD in the different settings below. \n\n—----------------—----------------—----------------—----------------\n\nMBPP: Avg Improvement across P@1,5,10,20\n\n| method                      | Codegen-mono-6B | Codegen-mono-16B |\n|-----------------------------|-----------------|------------------|\n| PERsD vs StanD              | + 5.0           | + 2.93           |\n| PERsD vs InpD               | + 2.52          | + 6.83           |\n| PERsD-refine vs InpD-refine | + 2.02          | + 0.56           |\n| PERsD-comb vs InpD-comb     | + 0.02          | + 7.12           |\n\n—----------------—----------------—----------------—----------------\n\nHumanEval:  Avg Improvement of across P@1,5,10,20\n\n| method                      | Codegen-mono-6B | Codegen-mono-16B |\n|-----------------------------|-----------------|------------------|\n| PERsD vs StanD              | + 9.1           | + 6.63           |\n| PERsD vs InpD               | + 4.01          | + 7.67           |\n| PERsD-refine vs InpD-refine | + 4.41          | + 3.65           |\n| PERsD-comb vs InpD-comb     | + 2.53          | + 3.31           |\n\n\nWe observe the advantage of personalized distillation is generally lesser on MBPP than on HumanEval, though the overall averages indicate it still leads to 2-7% gains in the majority of settings and performs at par in remaining two settings, without actually hurting performance.\n\nOne of the main reasons behind this is the data-overlap of MBPP being higher than HumanEval. More specifically, StanD enjoys a much higher (almost x3) data-overlap in MBPP over HumanEval. Additionally, StanD training data is almost 3x that of PERsD models. This means StanD has an unfair advantage (more data and more related data) which explains why it is possible for it to slightly outperform PERsD in 1/16 settings. However PERsD still outperforms StanD over 15/16 settings, showing the value of our proposal. \n\nWe will summarize and add the above interpretation to our final version.\n\n\n\n> Limited improvement of PERsD over InpD in -refine and -combined settings: \n\nFollowing our data-overlap analysis we want to emphasize that MBPP is more of an in-domain evaluation (as pretraining data constructed from seeds tasks of MBPP) and HumanEval is more of an out-of-domain (more general) evaluation. We find the effect of personalized distillation more pronounced when evaluating in “out-domain” setting (e.g. PERsD-refine has a clear edge over InpD-refine on HumanEval for all settings). In HumanEval the overlap-score of all the models being reasonably low, we see more regular patterns of average improvement across all settings in the above table. Thus our main evaluations and comparison with SoTA has been only on HumanEval.\n\n\n> Is 2-step refinement for Multi-step inference done using ChatGPT?\n\nNo, all inference is done by the local llm. We uphold the value that research and working products should be based on open sourced llms. The iterative inference is just a natural property that the model achieves simply because of the inherent design of the personalized data collection. There being quite limited work on using small open LLM for iterative self-correcting code-generation, PERsD advances this research as well by performing better than them (Sec 5.6 of paper)\n\nHowever our main goal/experiments show that personalized data in-fact helps boost 1-step code-generation, while the code-refinement step helps further. \n\n> How is Round 2 training data in Multi-round Distillation obtained\n\nRound2 training data is generated by treating the model finetuned in Round1 as the student. Round1 uses the original off-the-shelf pretrained model. \n\n> I found that the experimental results of training CodeGen-momo-16B with the data of CodeGen-momo-6B in Table 9 are lower than the results of StanD in Table 4(b), does it mean that a separate training data must be developed for each model in order for the results to be better than the results of StanD? Does this suggest that this method does not generalize well?\n\n “not generalize” is actually an expected property, as it is intuitive that personalized data must be truly personalized. This is infact verified by our experiments: Training 16B model with 6B model’s personalized data should perform poorer than standard distillation. The prior of 6B leads to inferior quality code-generation than what the more powerful pretrained 16B model can generate, which causes it to actually unlearn itself.  \n\nIn human education, personalized teaching also costs more effort, but it’s more effective and more data efficient. In addition, it is understandable that learning the proxy data generated by another less competent student, resulting in worse performance than learning from the teacher.\n\n> Evidence that Cost of ILF (related work) is >4k\n\nFrom ILF paper,  \"The final dataset consists of 195 triples of (incorrect program, human written feedback, human-written refinement). On average, workers are paid 23 dollars per annotated sample and take 27 minutes/sample\" -- Assuming only 1 worker per task, the cost comes to 4.4K dollars", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper presents Personalized Distillation in the field of code generation, which improves the ability of some open source code models to solve basic programming problems with a small amount of training data.\n2.The writing style of this paper is very good, the author can make good use of diagrams and graphs to get the ideas and thoughts across, and the experimental design is very adequate.", "reasons_to_reject": "1. Only basic programming problems (e.g., HumanEval and MBPP) were focused on and not evaluated above complex programming problems (e.g., APPS or CodeContest).\n2. The training data in this paper is not publicly available, which is not conducive to the reproduction of the experimental results.\n3. Section 5.7 of the paper mentions that personalized distillation is orthogonal to WizardCoder, but there is no experimental data to prove this.\n4. Section 5.1 of the paper mentions that \"Comparing PERsD-refine to InpD-refine, we find PERsD-refine outperforms InpD-refine in all settings (two backbones, two406datasets, two inference steps). two406datasets, two inference steps).\" But I find that in the experimental results of Table 4(b) MBPP, only the pass@1 result of PERsD-refine is a sizable improvement compared to InpD-refine, and the other results don't seem to be improved. And the pass@1 results for PERsD seem to be lower than those for StanD, which is not well explained in the paper.\n5. There does not seem to be any experimental data in the paper on whether the different training data generated by ChatGPT overlap with the data from the test benchmarks HumanEval and MBPP, and the experimental results are not very rigorous.", "questions_to_authors": "1. Is the 2-step refinement instruction for Multi-step inference in Section 3.3 generated with the help of ChatGPT? Does this go against the previously mentioned reduction of dependency on closed-source macromodels?\n2. How were the training data for Round 2 of Multi-round Distillation in Section 5.3 obtained? How is it different from the data of Round 1?\n3. I found that the experimental results of training CodeGen-momo-16B with the data of CodeGen-momo-6B in Table 9 are lower than the results of StanD in Table 4(b), does it mean that a separate training data must be developed for each model in order for the results to be better than the results of StanD? Does this suggest that this method does not generalize well?\n4. How was the Cost>4K$ figure for ILF in Table 10 obtained? I didn't find in the original article that it costs that much.", "paper_topic_and_main_contributions": "This paper presents a novel distillation method in the field of code generation for extracting capabilities from large closed-source models into smaller open-source models, improving the ability of open-source models to solve fundamental programming problems.\n1. While I found the paper to be minimally challenging and not technically challenging, the starting point and motivation for \"personalized learning\" is compelling, and the authors have done sufficient experiments to validate this motivation.\n2. There has been relatively little research in the code area on distillation of closed-source macromodels, and this paper extends that direction.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. There seems to be an error in the labeling of the results for PERsD and PERsD + InpD at pass@1 for Table 6, as per the data in the paper, the PERsD + InpD results are better.\n2. There are some spelling errors in the text; see line 152, FHe --> The; line 543, Fianlly --> Finallly ."}}, {"review_id": "HNexA84Wzp", "reviewer": "Reviewer_Y76m", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> How does this paper compare to training with human feedback and reinforcement learning\n\nWe highly appreciate the comments from the reviewer! To clarify the differences and links with feedback-based training of LLMs, we will provide a short survey here:\n\n“The reinforcement learning from human feedback (RLHF) [1,2], adopts an actor-critic framework, where the student model is optimized to generate responses which can receive higher reward from the critic model. In InstructGPT[1], the critic (reward model) is trained from human annotation. Direct Preference Optimization (DPO) [3] drops the need of training a reward model, by using a reference LLM and offline trajectories to estimate the reward. Chain-of-Hindsight [4] converts human preference annotations into simple NL feedback, and turns RL optimization to conditional generation. “\n\nThe methods above can all be broadly categorized as RLHF-style training, as they try to improve a LLM based on its generation and preference feedback. While Learning from Feedback is both in RL-style-training and personalized distillation, the role and objective is different. \n\n**RLHF-style training**: For RLHF-style training, the model needs to be already competent enough such that “generally” one of its generations should be correct/good, in order to learn effectively from scalar rewards. This issue will be exacerbated when the task is hard and the reward is sparse – for example, solving math puzzles or coding tasks. \n\n**Personalized Distillation**: \nCompared to RL-style-training from (human) feedback, the fundamental difference is PERsD gets the “correct” answer from the teacher instead of just a preference feedback signal. This allows effective and efficient knowledge acquisition. In this case, the agent does not need to be competent enough to begin with, and can effectively learn to solve complex tasks like code generation.\n\nCompared to Standard Distillation, in which the student’s prior is adapted to the teacher’s prior despite the fact that they might be very different, PERsD allows the learning to be personalized and adapts the student’s prior to its improved self. In our paper, we have shown such personalized learning enjoys consistent empirical advantages over standard distillation.  \n\nWith the above two comparisons, PERsD actually enjoys the best of two worlds. In RL literature, “On-policy Imitation Learning”[5,6] shares similar spirit as PERsD, where the method is both on-policy (personalized) and imitation learning (distillation). However, the settings of [5,6] are traditional RL settings with multi-step trajectories and a small action space. Our setting differs as code-generation has only one-step trajectory and a rich action space (code output). In this perspective, our paper can be viewed as the first work which realizes “On-policy Imitation Learning” for real-world generation application with modern LLMs.\n\nComparison with ILF (related work): In Sec 5.6 we compare with ILF, a related work which uses human-labeled personalized feedback for finetuning the model. In Table 10, we show PERsD achieves a much better pass@1 at a much lower cost than ILF.\n\nWe hope the above short survey and comparison resolves R2’s concern. We thank R2’s original comments again for helping us clarify with and draw a link to recent methods of training llms with feedback. We will consider turning the above comparison into a new section in related work to our final draft.\n\n\n[1] Long Ouyang, et al. (2022). Training language models to follow instructions with human feedback.\n\n[2] Zihao Li, Zhuoran Yang, & Mengdi Wang. (2023). Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism.\n\n[3] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, & Chelsea Finn. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model.\n\n[4] Hao Liu, Carmelo Sferrazza, & Pieter Abbeel. (2023). Chain of Hindsight Aligns Language Models with Feedback.\n\n[5] Kwangjun Ahn, Zakaria Mhammedi, Horia Mania, Zhang-Wei Hong, Ali Jadbabaie, Model Predictive Control via On-Policy Imitation Learning (2022)\n\n[6] Stephane Ross, J. Andrew Bagnell, Efficient Reductions for Imitation Learning (2010)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and is easy to follow.\n- The experiments are comprehensive and solid.\n", "reasons_to_reject": "- In my opinion, the idea of training with feedback is widely adopted in many research fields and seems not novel. Nowadays, LLMs also widely leverage human feedbacks with reinforcement learning to train a better model. What are the main contributions of this paper compared with previous methods and those human feedback-based training methods in other areas?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a personalized distillation method to distill knowledge from close-source LLMs to smaller open-source LLMs. The student solves the tasks first, then the teacher provides an adaptive refinement to improve the student. The experiments show the proposed method outperforms traditional distillation methods with one third of the data.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WsfuVBByOE", "reviewer": "Reviewer_ksiw", "summary": "", "questions": "It is fair to do multi-step inference in the experiments?\nI feel that is like to have a second chance for author's method to solve the task.\nIf it failed in 1-step and make it in 2-step, does this count pass@1 or pass@2?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The idea seems a general way to do distillation from ChatGPT to public small models on any tasks. But authors only focus on code generation in this paper, so I wonder the generalization ability of this method.\n\nWe sincerely appreciate the comment on “The idea seems a general way to do distillation”, which is exactly how we view our method. In ML perspective, standard distillation can be seen as forcing student’s prior to teacher’s prior, in spite of the fact that the two prior distributions could be very different. Personalized distillation, on the other hand, aims to adapt student’s prior to its improved self. \n\nIn this paper, we propose PERsD idea and verify it comprehensively on the coding domain. The main reason we choose coding domain is that the answers can be objectively and automatically evaluated (whether or not they pass the unit tests), thus we can have a clear understanding of how well PERsD performs compared to StanD/InpD.\n\nAs the first work in this direction, we hope it to elicit future works on verifying personalized distillation theoretically / applying it to more domains\n\n\n> It is fair to do multi-step inference in the experiments? I feel that is like to have a second chance for author's method to solve the task. If it failed in 1-step and make it in 2-step, does this count pass@1 or pass@2?\n\nThanks for bringing up this important question! In all comparisons, we compare 1-step and 2-step separately. When inf-step is not mentioned explicitly (Table 10,11,6), only 1-step results are reported to ensure fairness. We will add a sentence in Section 4 to make this clear in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea is smart and sound to me.\n\nFrom Table 4 (a), (b), the proposed method largely outperforms standard method to do distillation.", "reasons_to_reject": "The idea seems a general way to do distillation from ChatGPT to public small models on any tasks.\nBut authors only focus on code generation in this paper, so I wonder the generalization ability of this method.\n\n", "questions_to_authors": "It is fair to do multi-step inference in the experiments?\nI feel that is like to have a second chance for author's method to solve the task.\nIf it failed in 1-step and make it in 2-step, does this count pass@1 or pass@2?", "paper_topic_and_main_contributions": "This paper proposes a new distillation method: the teacher model will provide refinement results based on student's failed attempt, and then student will learn the teacher's refinement results.\n\nCompared with standard distillation where teacher model directly gave answers to students, the authors claim that their method is more personalized for student models and hence can achieve better distillation performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None", "Justification_for_Ethical_Concerns": "None"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript introduces a personalized distillation method to extract knowledge from closed-source to open-source Large Language Models. Unlike conventional distillation that forces student’s prior closely to follow teacher’s prior, the student LLM tackles given tasks first, then the teacher LLM adaptively offers refinement to improve the student’s own prior. The experimental results show that the personalized distillation outperforms the conventional distillation with only one third of the data. All reviewers concur that the paper proposes a smart idea, and the experimental results are solid for code generation domain. To further improve the clarity, incorporating friendly comparisons against traditional distillation and RLHF elucidates a clearer distinction across various approaches.
Decision: Accept-Main