MiniLLM: Knowledge Distillation of Large Language Models
Authors: Yuxian Gu
Li Dong
Furu Wei
Minlie Huang
Venue: ICLR 2024 poster (2024)
Keywords: Large Lanauge Models
Knowledge Distillation
Primary area: generative models
TL;DR: We propose a knowledge distillation method for Large Language Models with minimizing the reverse KL divergence as the objective.

Abstract: Knowledge Distillation (KD) is a promising technique for reducing the high computational demand of large language models (LLMs). However, previous KD methods are primarily applied to white-box classification models or training small models to imitate black-box model APIs like ChatGPT. How to effectively distill the knowledge of white-box LLMs into small models is still under-explored, which becomes more important with the prosperity of open-source LLMs. In this work, we propose a KD approach that distills LLMs into smaller language models. We first replace the forward Kullback-Leibler divergence (KLD) objective in the standard KD approaches with reverse KLD, which is more suitable for KD on generative language models, to prevent the student model from overestimating the low-probability regions of the teacher distribution. Then, we derive an effective optimization approach to learn this objective. The student models are named MiniLLM. Extensive experiments in the instruction-following setting show that MiniLLM generates more precise responses with higher overall quality, lower exposure bias, better calibration, and higher long-text generation performance than the baselines. Our method is scalable for different model families
with 120M to 13B parameters. Our code, data, and model checkpoints can be found in https://github.com/microsoft/LMOps/tree/main/minillm.

Published as a conference paper at ICLR 2024
MINILLM: KNOWLEDGE DISTILLATION OF LARGE LANGUAGE MODELS
Yuxian Gu!:2; Li Dong", Furu Wei", Minlie Huang
'The CoAl Group, Tsinghua University Miicosoof Research
guyx21emails wtt.iss..ssssssiieeeeeeeeeeeeese tsinghua. en {lidongl, fuwei cmicrrosoft.com Shhuangêtsinghus edu.cn
ABSTRACT
Knowledge Distillation (KD) a promising technique for reducing the high com- putational demand of large language models (LLMs). However, previous KD methods are primarily applied white-box classification models or training small models to imitate black-box model APIs like ChatGPT. How to effectively dis- till the knowledge of white-box LLMs into small models still under-explored, which becomes more important with the prosperity of open-source LLMs. In this work, we propose a KD approach that distills LLMs into smaller language models. We first replace the forward Kullback-Leibe tive in the standard KD approaches with reverse KLD, which is more suitable for KD generative language models, to prevent the student model from overesti- mating the low-probabiligy regions of the teacher distribution. Then, we derive effective optimization approach to learn this objective. The student models are named MINILLM. Extensive experiments in the cittructioo-follovvii setting show that MINILLM generates more precise responses with higher overall qual- ity, lower exposure bias, better calibration, and higher long-text generation perfor- mance than the baselines. Our method scalable for different model families with 120M to 13B parameters. Our code, data, and model checkpoints can be found in https: ///g//g/ htt///gtttuuu/e. thub. com/mi croso ft LLMOps tttees/rreeeee tteeeeeeeeeeeeeeeeeeeeeeeeeeeeee / ma iniii//mmm m/mmnii 11m.
divergence (KLD) objec-
Aversqe Rouqei
25
Teacer: GPT-2-1.58
26
Teacher: GPT-J
Teacher: OPT 138
25
20
24
MiniLLM
MiniLLM
SeqKD
SeaKD
SeqKD
100M
200M
400M
700M
18
1.58
29
2.5B
1.58
3B
6B
student parameters
of student parameters
student parameters
Figure The comparison of MINILLM with the sequence-level KD (SeqKD)² in terms of the average Rouge-L score on the evaluation sets. Left: GPT-2-1.5B as the teacher and GPT-2 125M, 340M, 760M as the students. Middle: GPT-J 6B as the teacher and GPT-2 760M, 1.5B, GPT-Neo 2.7B as the students. Right: OPT 13B as the teacher and OPT 1.3B, 2.7B, 6.7B as the students.
INTRODUCTION
With the rapid development of large language models (LLMs; Brown et al., 2020; Han et al., 2021; Bommasani et al., 2021; Chowdhery et al., 2022; OpenAI, 2023), common technique to reduce their high computational resource demand is knowledge distillation (KD; Hinton et al., 2015), where we train a small student model with supervision from a large teacher model. Two categories KD commonly applied: black-box KD, where only the ceacher-generate texts are accessible, and white-box KD, where the teacher model's output distribution or intermediate hidden states are also available (Jianping al., 2021). Recently, black-box KD has shown promising results in fine-tuning
"Contribution during internship at Microsoft Research.
1 "Correpoondig author.
'SeqKD (Kim & Rush, 2016; Chiang et al., 2023; Taori et al., 2023) directly trains the student model on the texts generated by the teacher model, which is widely used KD method for language models.

---

Published as a conference paper at ICLR 2024
small models on the prompt-response pairs generated by LLM APIs (Taori et al., 2023; Chiang et al., 2023; Wu et al., 2023; Peng et al., 2023). With the emergence of more open-source LLMs (Zhang et al., 2022; Touvron et al., 2023), white-box KD becomes more valuable for both research commu- nities and industry sectors because student models receive better signals from the output distribution and hidden states of teacher models, thereby potentially resulting in higher performance. However, white-box KD approaches are mostly studied for small (< 1B parameters) language understanding models (Sanh et al., 2019; Wang et al., 2020), while white-box KD for LLMs yet be explored.
In this work, we investigate white-box KD of LLMs where the output distribution of the teacher model is available. We argue that the standard KD objectives (Kim & Rush, 2016; Song et al., 2020; Chiang et al., 2023; Taori et al., 2023) are sub-optimat for LLMs that perform tasks in generative manner. Given the teacher distribution ply) (r)) and the student distribution qo (yy(z)) (z)) parameterized by 0, standard KD objectives (including several variants for sequence-level models) essentially min- imize the approximated forward Kallback--eiible divergence (KLD) between the teacher and the student distribution, termed KL[pl1ag], which forces qo to cover all modes of p. For text classi- fication tasks, [L'polgo] works well because the output space usually consists of a finite number of classes such that both (()/() and ()(y)z)) have few modes. However, for open-ended text gener- ation tasks, the output spaces are much more complex, and p(y) xz) can contain many more modes than what go(y) ær) can express due the limited model capacity. Minimizing forward KLDD causes to to assign unreasonably high probabilities to the void regions of (Malinin & Gales, 2019) and produces very unlikely samples under p during free-run generation (Huszár, 2015).
To alleviate this problem, we propose to minimize reverse KLD, KL Ip]. widely used in computer vision (Lee et al., 2023) and reinforcement learn- ing (Czarnecki et al., 2019). Compared to [L(plgg)) minimizing [L[gol|p] causes qo to seek the major modes of p, and assign low probabilities to p's void re- gions (Minka et al., 2005), as illustrated in Table 2 and discussed in Section 2.1. In LLM text generation, this means that the student model avoids learning too many long-tail variants (Holtzman et al., 2020) in the teacher model's 's distribution and focuses on the correctness of the generated cotents, which is critical in practical scenarios that require truthfulness and reliabil- ity (Ji et al., 2023b). To optimize ming [L [p). as shown in Section 2.2, we derive the gradient of the objective with Policy Gradient (Sutton et al., 1999). To further stabilize and accelerate training, we propose (I) single-step decomposition to reduce variance, (2) teacher-mixet sampling to allevi- ate reward hacking, and (3) length normalization to eliminate the length bias. Finally, we introduce the overall KD algorithm in Section 2.3. Our student models are named MINILLM, indicating our method is suitable and works well for compressing large (generative) language models.
Distribution
0.4
Forward KLD
Reverse KLD (ours)
0.2
0.0
2.5
5.0
7.5
10.0
12.5
Figure 2: We fit a Gaussian mixture distri- bution with single Gaussian distribution using forward KLD and reverse KLD.
We apply our method to various generative language models (Radford et al., 2019; Zhang et al., 2022; Touvron et al., 2023) with sizes ranging from 120M to 13B in the instruction-followwi ting (Sanh et al., 2022; Wei et al., 2022a) that covers a large range of NLP tasks. We use datasets with Rouge-I. (Lin, 2004), human judgment, and the GPT-4 feedback for evaluation. Experiments show that MINILLM consistently outperforms standard KD baselines on all the datasets and scales up well from 120M to 13B models (see Figure 1). More analysis shows that MINILLM yields lower exposure bias, better calibration, and higher long response generation performance.
set-
2
METHOD
We consider conditional text generation where the model produces a response y = {1r condi- tioning on a prompt æ sampled from the distribution Pa, which is typically how LLMs perform tasks. We formulate KD as an optimization problem to minimize the difference between a fixed teacher model distribution p(y|æ:) (z) and student model distribution (y æ) parameterized by 0. The stan- dard KD methods approximately minimize the forward KLD; KL [qe] = Ex-pa.vp'' log p(y|z) ge ( where p' can be real data distribution (word-level KD) teacher distribution p (sequence-level KD).
We say "approximately" because for word-level KD, is sampled from the real distribution, not the teacher distribution. For strong enough teacher model, we can consider the two distributions approximately the same.
2

---

Published as a conference paper at ICLR 2024
Teacher
Teacher
Promptx
Student
((0) klpplael
Forward KLD
Prompix
Studert
((e)
R Reverse KLD
KL lpp
VA(4)
2.21
Sequence-Level
MiniLLM (Ours)
Comparison between sequence-level KD (left) and MINILLM (right). Sequence-level KD forces the student to memorize all samples generated by the teacher model, while MINILLM improves its generated texts with the teacher model's ' feedback.
Figure 3:
Though widely used, KL[pl]ge tends to overestimate the void regions of p text generation tasks when go is insufficiently expressive (Ji et al., 2023a). KD for LLMs fits the case because LLMs perform tasks generative manner, such that the low-capacity student models cannot perfectly imitate the complex text generation distribution of the teacher models or humans.
2.1 MINILLM: KNOWLEDGE DISTILLATION WITH Reverse KLD
We consider minimizing the reverse KLD between the student and teacher model distributions as learning objective for MINILLM:
0 arg min L(0) = arg min KT. [To(golp))
p(y x
(I)
arg min
E
log
go (3/ (y)=)
Minimizing reverse KLD has been shown to cause the mode-seeking behavior in generative model- ing (Huszár, 2015; Nowozin et al., 2016; Chen al., 2018; Lee al., 2023), where qo assigns high probabilities to p's large modes and ignore the small ones (illustrated in toy experiment in Figure 2). In this work, we first study this property for KD of LLMs text generation. Minimizing forward KLD causes qo to place large probability mass the zero-probability places of p, corresponding to the low-quality text generation practice, while reverse KLD focuses on p's major modes, which crucial to ensure the correctness and faithfulness of text generation. As illustrated in Figure 3, unlike sequence-level KD, MINILLM that minimizes reverse KLD does not force qe to all y sampled from the teacher distribution p. Instead, it encourages the student generate samples pre- ferred by the teacher within its own capacities, which is more possible to achieve. Interestingly. also find another perspective of understanding MINILLM motivated by Inverse Reinforcement Learning (Ziebart et al., 2008). We present the related derivation in Appendix A.I.
2.2 OPTIMIZATION WITH POLICY GRADIENT
Gradient Derivation We notice that the gradient of the objective function L(0) in Equation can be derived using the Policy Gradient Theorem (Williams, 1992; Haarnoja et al., 2017):
VC(0)
E
(11a) -
where T' ly and R₁ R ΣJ = log pluelMeee aM)
gu(Ber Mecer.a) the accumulation of re log MMreeeere that golde (azer.a)
measures the quality of each step's generation. Intuitively, the generated texts are supposed to have high probabilities under the teacher distribution by increasing plyelyee ar), but simultaneously stay diverse by lowering go (ye yee,, æ). The expectation in Eq. is computed by Monte-Carlo sampling. Full derivation can be found in Appendix A.2. However, policy gradient suffers from high variance and reward hacking (Skalse et al., 2022), despite some subsequent solutions (Schulman al., 2017). Besides, we notice that R, favors short sentences, which causes the student model to output empty responses. Therefore, we propose three strategies to mitigate these problems.
Single-Step Decomposition Czamecki et al. (2019) has found that the single-step generation qual- ity r, is critical to the training variance because the error in the front tokens accumulates along the whole sentence. To pay more attention to rt, re-write (()) to decompose rt from Rd and
3

---

Published as a conference paper at ICLR 2024
directly compute the gradient of Byye (see Appendix A.3 for the full derivation):
VL(()=
E
E vemgel))
[re] [
B
Σ R V gog.
amp y- (o))
and yougg (olee)
= (Vim)e + (Vn)
where (t) go(-1y=c, v). Note that Eyy [r]] can be computed directly by summing over the vocabulary instead of using Monte-Carlo sampling and derivable with respect 0. This decomposition gives a more precise and efficient estimation of the single-step generation quality, which reduces the variance during training and accelerates convergence.
Teacher-Mixed Sampling We observe reward hacking (Skalse al., 2022) when training with Eq. 2 because de sometimes produces degenerated sentences y that receive high scores from the teacher (e.g., repeated phrases) during sampling, especially for small student models. To create better sampling distribution, we mix the teacher and the student distribution at each time step:
(4) where controls the strength of the teacher mix-in. Sampling from suppresses low-quality gen-
P(Ue Vet, x) = a p(vel z) (1-a) (o x),
cration with the teacher's s help and alleviates reward hacking. We re-write (ing
Single and (V)) with importance sampling to get to unbiased estimator of the gradient (Precup et al., 2000):
C) Single
(re]
Vt (tm)g(())
(V (Vnn
Wc Ret1 V log (o (
where w, I go(ue (eees)) is the importance weight. However, we brings high variance in Playlvee-
practice because requires multiplying per-token importance weight over multiple time steps, and thus the variance of each step accumulates. Therefore, we approximately set wy R ga(ti ( to p(de (.)..)))
reduce the variance of the estimator in Eq. (Serban et al., 2017; Levine et al., 2020).
Length Normalization We found that long sequences tend to have small Re+1, which encourages the model to produce short responses. Therefore, we add length normalization to R441 in Eq.
1
log pluelMers (J y)v.
R++1
(6)
In Summary Combining the strategies listed above, we have the final optimization gradient:
ply (e()))) go (y Uee,
Noem Vi 2)).
VC(0)
E
go (ol'l x) log
qo (3t Me x))
V'EV
S)intt part
(V Cara part
where the vocabulary size the language model and (V L)
Norm Long
(V Lang with Ry
2.3 TRAINING ALGORITHM
We start from a student model pre-trained on large long-document corpus Drr. The algorithm to train MINILLM adapts the student model to text generation task with dataset D. We assume that there is teacher model performing well on D, such as LLM fine-tuned on D (Taori et al., 2023; Chiang et al., 2023) or that with good task-generalization (Chung al., 2022; OpenAI, 2023).
In the training algorithm, we first fine-tune the student model on D and pick the checkpoint with the lowest loss as an initialization for the following training. Then, we compute the gradients (VL) and (VL) N based on Eq. and Eq. with a clipping strategy (Schulman et al., 2017) added to further improve stability. Same Ouyang al. (2022), we include language modeling loss CpT
- Ed-Dpr Drg log go (d) to preserve the model performance on canonical NLP benchmarks. The student model is finally updated using a combination of gradients (imgle (V C) Norm + Lpr- The whole training pipeline is similar to Reinforcement Learning from Human Feedback (RLHF; Ouyang et al.,
2022). We present the details of the MINIL.MM training algorithm in Appendix B.
4

---

Published as a conference paper at ICLR 2024
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
We take instruction-following (Ouyang al., 2022) as the conditional text generation task, where models are trained to generate responses according to the instructions. We fine-tune large model on the dataset 'D consisting instruction-respons. pairs as the teacher model. Then, we compare different KD methods on D by evaluating the student model's instruction-folloving performance.
Base Models Our student models come from three model families with various sizes: GPT- 2 (Radford et al., 2019) (120M, 340M, 760M), OPT (Zhang et al., 2022) (1.3B, 2.7B. 6. 6.7B), and LLaMA (Touvron et al., 2023) (7B). For teacher models of each model family, we use GPT-2-1.5B, OPT-13B, and LLaMA-138 respectively. These models are fine-tuned on D in advance. We also present the results using GPT-J (Wang & Komatsuzaki, 2021) as the teacher in Appendix D.2.
Training We construct the training data from human-written instruction-response pairs. We filter out samples that exceed the context length of the models. Then, we randomly split 0.5K and IK samples for validation and testing, respectively, leaving about 12.5K examples for training. For Dpr, use Open WebText (Gokaslan et al., 2019) for the GPT-2 family and the RoBERTa training corpus (Liu et al., 2019) for other models. We set the teacher-mix-in strength
seabricks-dollyy. 15k consisting of 15K
Rouge-L (Lin, 2004) scores on the validation set to search for hyper-parameters because it aligns better with human preference than validation losses (Wang et al., 2022). More details are shown in Appendix C.I.
0.2 throughout the experiments in Eq. 4. We
Evaluation We evaluate the trained models on 5 Instruction-following datasets:
Dolly: the 500-sample test split from the databricks-dollyy
dataset.
SelfInst (Wang et al., 2023): A user-oriented instruction-followwe set with 252 samples.
Vicuna (Chiang et al., 2023): The 80 challenging questions used in the Vicuna evaluation.
(Wang et al., 2022) consisting of 9K samples ranging from 119 tasks. Following Peng et al. (2023), we split the into 3 subsets whose ground truth response lengths lie in [0, 5]. [6, 10]. and [11, coo]. We use the [11, +o0 subset Section 3.2 and conduct an analysis on all subsets Section 3.3.
S-NI: The test set of
NATURAL
(Honovich al., 2023) containing 60K sam- ples. Similar to S-NI, we first conduct the evaluations on the randomly sampled 10K examples in the [11, +20 subset, followed by an analysis of the performance on all subsets in Appendix D.4.
UnNI:
The core set of
We adopt two metrics to evaluate the model-generated responses:
R-L: The Rouge-L, (Lin, 2004) score to measure the precision the model generation. Wang al. (2022) has shown that Rouge-L is suitable for large-scale instruction-following evaluation.
HumanEval: We conduct human evaluations the SelfInst dataset following Peng al. (2023) by asking volunteers to compare two responses produced by different models and annotate "Win", "Tie", or "Loss". More human evaluation details can be found Appendix C.3.
For all test sets, we sample the responses with the temperature 1 and report the average scores of generations for each prompt with different random seeds. We also include a supplemental evaluation using the GPT-4 feedback (Zheng et al., 2023) by asking GPT-4 (OpenAI, 2023) to compare model- generated responses with the ground truth answers and present the results in Appendix D.
Baselines
We consider three baselines in our main experiment:
SFT w/o KD directly fine-tunes the student model on D supervised by the golden responses.
KD (Sanh et al., 2019; Song et al., 2020) fine-tunes the student model on D using the teacher distribution as the supervision at each token step, also known as word-level KD.
SDqKD (Kim & Rush, 6016; Chiang al., 2023; Taori et al., 2023) fine-tunes the student model on the data generated by the teacher model.
hhtpps:// thub. com wtt.:/ooo//aaaaeeaeeeo htt.m//aaaaiiiic abs dol httpb/////bllll//e////ttttoo
5

---

Published as a conference paper at ICLR 2024
Model
#Params
Method
Dolly
Selfinst
Vicuna
S-NI
UnNI
1.5B
Teacher
27.6
14.3
16.3
27.6
31.8
SFT w/o KD
23.3
10.0
14.7
16.3
18.5
KD
22.8
10.8
13.4
19.7
22.0
120M
SeqKD
22.7
10.1
14.3
16.4
18.8
MINILLM
24.6
13.2
16,9*
25.3
26.6
GPT-2
SFT w/o KD
25,5
13.0
16.0
25.1
28.1
KD
25.0
12.0
15.4
23.7
24.6
340M
SeqKI
25.3
12.6
16.9*
22.9
23.3
MINILLMM
25.4
15.6
17.7*
27.4
30.8
SFT w/o KD
25.4
12.4
16.1
21.5
24.0
KD
25.9
13.4
16.9
25.3
28.0
760M
SeqKD
25.6
14.0
15.9
29.1
MINILLM
26.4
15.9*
18,3*
29.3*
34.5*
13B
Teacher
29.2
18.4
17.8
30.4
36.1
SFT
26.0
11.4
28.4
KD
25.4
12.2
14.9
21.9
27.0
1.3B
SeqKD
26.1
12.7
16.6
21.4
28.2
MINILLM
26.7
14.8
17.9*
28.6
33.4
OPT
SFT w/o KD
27.1
13.9
16.6
24.9
32.3
KD
25.9
13.8
16.7
26.3
30.2
2.7B
SeqKD
27.5
13.3
16.5
32.3
MINILLM
27.4
17.2
19.1*
30,7*
35.1
SFT KD
27.6
16.4
17.8
28.6
KD
28.3
17.0
17.5
30,7*
26.7
6.7B
SeqKD
28.5
17.0
17.9*
30.4
28.2
MINILLM
29.0
17.5
18.7*
32.5
36.7
13B
Teacher
29.7
23.4
19.4
35.8
38.5
SET w/o KD
26.3
20.8
17.5
32.4
35.8
LLaMA
KD
27.4
20.2
18.4
33.7
37.9
7B
SeqKD
27.5
20.8
18.1
33.7
37.6
MINILLM
29.0
23.2
20.7*
35.5
40,2
Table 1: Evaluation results. We report the average R-L scores across random seeds. The best scores of each model size are boldfaced, and the scores where the student model outperforms the teacher are marked with
3.2 RESULTS
We present the Rouge-L evaluation results in Table 1, from which we have three observations.
First, by comparing the overall performance MINILLM with the baselines, we observe that the model distilled by our KD method outperforms the baselines in almost all cases, when trained with different base models and tested on various evaluation sets. This verifies the good generalization and high overall performance of KD method. We also find that MINILLM generally works much better on datasets other than Dolly compared with the baselines, indicating its good out-of- distribution generalization.
Second, the Rouge-L. scores show that the MINILLM produces the most precise responses that have high overlaps with the ground-truth responses. some cases, especially Vieuna, S-NI, and UnNI, student models reach even higher Rouge-L scores than the teacher models, which matches the ob- servation in Furlanello al. (2018). We conjecture that the standard teacher-foreing fine-tuning on inference discrepancy to the teacher model, also known exposure bias (Bengio al., 2015). On the contrary, MINILLM is optimized with policy optimization methods, which samples responses from student models during training and thus alleviates exposure bias (Pang & He, 2021). We include further analysis on exposure bias Section 3.3.
D brings training-inference

---

Published as a conference paper at ICLR 2024
40
SST2
BoolQ
ExAccErr
20
ECE
Ace.
ECE
Acc.
KD
Teacher
0.025
93.0
0.356
74.5
SeqKD
MiniLUM
KD
0.191
84.7
0.682
63.5
SeqKD
0.243
66.5
0.681
62.8
0
50 100
150
200
250
MINILLM
0.099
89.7
0.502
67.8
Generation Length
Figure 6: The excess error caused by the training- inference discrepancy (ExAccErr) accumulated with the generation length. Lower ExAccErr means the method introduces less exposure bias.
Table 2: The ECE scores and accuracy scores (Acc.) on SST2 and BoolQ datasets. The best scores among student models are boldfaced.
Third, comparing the results across model sizes and model families, consistent when the base model sizes vary from 120M to 13B across three model families. This tendency is also illus- trated in Figure 1, which demonstrates the excellent scala- bility and generalization of our method in the era of LLMs.
MiniLLM Wins
Tie
MiniLLM Loses
that the improvement of MINILLM is
SFT
w/o KD
KD
SeqKD
Teacher
0%
20%
40%
60%
80%
100%
The human evaluation results on the SelfInst dataset based on the LLaMA family are shown in Figure 4. MINILLM obtains better human preference than all the baselines, per- forming comparably to the teacher model.
Figure Human evaluation results. We use LLaMA-7B the student and LLaMA-133 as the teacher.
3.3 ANALYSIS
Scaling Law Teacher Although is intuitive that we distill better student models from larger teacher mod- els, Mirzadeh et al. (2020) has shown that increasing the
Average Rouge-L
22
teacher models' sizes does not guarantee the improvement of student models, sometimes even harming the distilla- is not clear how MINILLM works when we scale up the teacher models' sizes. Therefore, we compare MINILLM and SeqKD using teacher models with different sizes and fix the size of the student model. We present the results based on the GPT-2 family Figure 5 and that based on the OPT family in Appendix D.3. We can see that MINILLM constantly outperforms SeqKD, and the student model performance is positively correlated with the teacher model sizes. This shows the potential method compress models with massive parameters.
200
MinLUM
ScoKD
18-
tion performance.
Student:
400M
700M
1500M
of teacher parameters
Figure 5:
The scaling law of teacher based on the GPT-2 family models. We compare MINILLM and SeqKD with GPT-2-125M as the student and GPT-2 340M, 760M, and 1.5B as teachers.
Exposure Bias Language generation models trained minimize forward KL.D suffer from posure bias (Bengio et al., 2015) caused by the discrepancy between teacher-forcing training and free-run generation. When training MINILLM, the student model sees samples generated by itself, alleviating the training-infereen
mismatch (Pang & He, 2021). In Figure 6, we use the ExAccErr metric (Arora al., 2022) defined in Appendix C.5 to measure the excess accumulated error due to exposure bias. The experiment is based on GPT-2-125M, with GPT-2-1.5B as the teacher, using Dolly as the test set. For each prompt, we sample responses to reduce the variance. We can see that the ExAceErss of the baselines continuously grow during generation, while MINILLM has much lower ExAccErr, and the eтror stops accumulating in long-text generation (> 150 tokens).
Calibration OpenAl (2023) has shown that models trained with policy optimization are likely to be poorly calibrated. We test the calibration of MINILLM and the KD baselines on two widely- used text classification datasets: SST2 (Socher et al., 2013) and BoolQ (Clark et al., 2019), based on LLaMA-7B. We design zero-shot classification instructions (see Appendix C.2) and take the

---

Published as a conference paper at ICLR 2024
Dolly
SelfInst
3 Sea
Dist-4
Loss
Dist-4
Loss
2 MiniLLM
Teacher
99.3
3.55
99.1
4.44
1
KD
99.4
3.93
98.8
5.36
ScqKD
99.3
3,91
98.8
5.22
[0. 5]
[6, 10]
(11, +
MINILLM
99.0
3.95
98.6
5.33
Ground Truth Length Range
Table 3: The distinet 4-grams (Dist-4) and language modeling loss (Loss) on the test sets based the LLaMA family. MINILLM pre- serves generation diversity.
Figure 7: The Rouge-L scores of the distilled models against SFT models the different sub- sets of S-NI split by the golden responses' length.
probability of the label words to compute the ECE scores (Nixon et al., 2019). From Table 2, we observe that KD and SeqKD models are worse calibrated than the teacher model, which potentially explains their low performance on canonical benchmarks (Gudibande et al., 2023). We suspect that m forward KLD causes the models to push high probabilities to void regions of the target distribution, which leads to significant distribution differences between the student and the teacher (see the example in Figure 2). In contrast, MINILLM focuses on accurately learning the major parts of the target distribution and narrows the ECE scores gap between the student and the teacher.
Performance on Various Response Length We study the models' performance when the golden response lengths belong to different ranges. In Figure 7, we illustrate the Rouge-L scores of different KD models against the SFT models on three S-NI subsets split by the length of the ground truth responses. We can see that all methods achieve low scores prompts that expect short responses (s 5 tokens), probably because most responses our training set are long sentences, introducing distribution shift between training and evaluation (Peng et al., 2023). Furthermore, the output spaces of these prompts are relatively small, allowing the student model to cover most modes of the teacher, and thus reverse KLD and forward KLD have similar performance. For prompts with longer responses (2 6 tokens), the teacher distribution contains more modes than the students due to the complex output spaces, which shows the advantage of MINILLM against standard KD models. Similar results on UnNI are shown in Appendix D.4.
Generation Diversity Caccia et al. (2020) has found that the model optimized by minimizing re- verse KLD is likely to lose modes, which affects the generation diversity. We follow Pang & He (2021) to discuss generation diversity from three aspects: (i) generating multiple distinct responses given a prompt. (ii) generating linguistically complex responses. (iii) the ability to generate contents that have high coverage of the real data distribution. For (i), we argue that for many NLP applica- tions, generating one correct response sufficient, especially for those scenarios demanding high truthfulness and reliability (Ji et al., 2023b). For (ii) and (iii), report the responses' distinct 4- gram proportion and the language modeling loss on the test sets in Table 3, using the hase models from the LLaMA family (see Appendix C.4 for more details). We can see that MINILLM preserves the distinct 4-gram proportion in the generated responses and language modeling loss on the test set.
3.4 ABLATION STUDIES ON OPTIMIZATION STRATEGIES
We evaluate the effectiveness of the three strategies proposed to stabilize and accelerate optimization in Section 2.2 by distilling a GPT-2-1 2-5M22-1MMM model from the GPT-2-1.5B model. More ablation studies can be found in Appendix D.5. In Table 4, we report the best Rouge-L scores on the validation set of each run and the evaluation results of the corresponding checkpoints. We also plot the reverse KLD between the student and the teacher during training in Figure 8, where the curves are smoothed by 32 steps. We can see that Teacher- Mixed Sampling and Length Normalization works for stabilizing training, Although the reverse KLDs also decrease without these strategies, we find that the models quickly learn to generate repeated, short, or meaningless strings that have high probabilities in the teacher distribution (see examples in Appendix E), which is known as reward hacking (Skalse et al., 2022). This also leads to the low generation performance in Table 4. From Figure 8, we also observe that the Single-Step Decomposition effectively reduces the variance of the training process, which also results in higher scores the validation and test sets.
8

---

Published as a conference paper at ICLR 2024
MinILLM wio Length Norm
Valid. R-L
Dolly R-L
MiiLLMM wib Teache-Mived
MiniLLM wite Singlesttpp
MiniLUM
MINILLM
27.4
24.6
Reverse
4
w/o Length Norm.
17.4
14.7
w/o Texeher-Mixed
22.3
20.4
w/o Single-Step
27.0
23.7
1060
2000
3000
4000
5000
Training Steps
Table 4: The performance on the validation and test set when different combinations of MINILLM optimization strategies are applied.
Figure 8: The reverse KLD between the teacher and the students during MINILLM training when different optimization strategies are applicd.
RELATED WORK
Large Language Models Large language models (LLMs; Brown et al., 2020; Thoppilan al.,
2022: Chowdhery et al., 2022: OpenAl, 2023; Anil et al., 2023) have shown superior performance by solving various NLP tasks in a generative manner. Recent works apply instruction tuning (Wei et al., 2022a; Sanh et al., 2022; Chung et al., 2022) or learning from human feedback (Ouyang al.. 2022; Bai et al., 2022) to improve the alignment of LLMs with humans further and create general AI assistants (OpenAI, 2022; Google, 2023). There are also efforts to build open-source LLMs (Zhang et al., 2022; Touvron et al., 2023; Biderman et al., 2023) to facilitate research and industry development. Although appealing, the broad capacities of LLMS usually emerge with large model sizes (Kaplan et al., 2020; Wei et al., 2022b) that require massive computational resources. Therefore, model compression is critical for the practical deployment and further research of LLMs.
Knowledge Distillation Knowledge distillation (KD; Hinton et al., 2015), as a widely used model compression technique, aims training student model with the guidance of teacher model (Rusu et al., 2015; Sanh et al., 2019; Jianping et al., 2021). In the NLP community, many works apply KD text classification tasks by mimicking the teacher model's output distribution (Song et al., 2020; Liang et al., 2021: Zhang et al., 2023). hidden states (Jiao et al., 2020; Sun et al., 2019), or attention scores (Wang et al., 2020: 2021). For text generation, the standard KD method to approximately minimize the forward KLD between the student's 's and the teacher's generation distribution by using the teacher's output each time step as supervision (Sanh et al., 2019) or direct training on the teacher's generated texts (Kim & Rush, 2016; Taori et al., 2023; Chiang et al., 2023; Peng et al., 2023). In this paper, we minimize the reverse KLD, which is more suitable for LLMs when the teacher distribution is available. Concurrent works (Agarwal et al., 2023; Wen et al., 2023) also explore more the distribution discrepancy metrics in KD.
The distribution discrepancy metrics play a significant role in training text generation models. The forward KLD is widely used due to its simplicity when derived the Maximum Likelihood Estimate (MLE) objective (Zhang & Zhao, 2019). However, previous works show that minimizing forward KLD leads to zero-forcing behavior where models try to cover all modes of the target distribution and sacrifice the accuracy of major modes (Huszár, 2015). Some works resort to using other metrics to remedy this problem, such as re- verse KLD (Jiang et al., 2020), Total Variation Distance (Ji et al., 2023a), and Optimal Transport (Li et al., 2020), Our paper tackles this problem under the scenario of knowledge distillation for LLMs.
Distribution Discrepancy Metrics in Text Generation
5 CONCLUSION
In this work, we investigate the problem of distilling the knowledge of LLMs into small language models. We find that the standard distillation methods minimizing the forward KL.D is sub-optimal in language generation scenarios because the teacher's output distribution contains more modes than the student's, s, and forward KLD forces the student distribution to overestimate the low-probability regions of the teacher distribution. Therefore, we propose MINILLM that minimizes the reverse KLD between the teacher and student distribution and design an algorithm to optimize this objec- tive. Extensive experiments show that MINILLM produce more precise responses that have higher overall quality than standard KD models. We also find that MINILLM has lower exposure bias, better calibration, and higher performance in long-text generation with good generation diversity.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by the National Key Research and Development Program of China (No. 2021ZD01 13304), the National Science Foundation for Distinguished Young Scholars (with No. 62125604), and the NSFC projects (Key project with No. 61936010).
REFERENCES
Rishabh Agarwal. Nino Vieillard, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. GKD: Generalized knowledge distillation for auto-regressive sequence models. arXiv preprint arXiv:2306, (tt.:2230..3649, 2023. URL htttps://ariiv
13649.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023. utthttps: /ttp///axvvvv.//////ee//eeeeeeet org/ /atg//ab//t /00bb//0330..33.
Kushal Arora, Layla El Asri, Hareesh Bahuleyan. and Jackie Cheung. Why exposure bias matters: An imitation learning perspective of enror accumulation in language generation. In Findings of ACI, 2022. URL https: /acl antho 20222 findings-ac! a8...88
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022. URL https::t///.mm http:://aavvvvii orwpv/..rgbbbba... /2004. 05862.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sam- pling for sequence prediction with recurrent neural networks. In Proceedings of NeurlPS, 2015. URL https::/proceedii Abstrratttttttt. htm.a..th.mm......m
neurips wwtppcpppp/aeeeeeee cccppaper/211//hase
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling, arXiv preprint arXiv:2304.01373, 2023. URL https: wttp:/vw...rr///////3300 http:/ddf/223//// 01373. pdf.
Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Au- toregressive Language Modeling with org/10. 5281/ zenodo. 5297715.
March 2021. URL https://doi.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the op- portunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. URL org/ abss 2108 07258.
https:// wttg
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al. Language models few-shot learners. Proceedings of NeurlPS, 2020. URL https: / /pppppaapra..a/////// nips. cc/ cttpaper/////////p//p// /aappr/ 96744444444444 @bfb8acl 42f 44 Abstract catsstracttttt html.
Massimo Caccia, Lucas Caccia. William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Char- lin. Language gans falling short. In ICLR, 2020. URL wwwptttpps///peeeeeiieieeee net. net/pdf?
(V-BJJgza6VtPB.
Alex James Chan and Mihaela der Schaar. Scalable bayesian inverse reinforcement learning. In ICLR, 2021. URL https:
Liqun Chen, Shuyang Dai, Yunchen Pu, Erjin Zhou, Chunyuan Li, Qinliang Su, Changyou Chen, and Lawrence Carin. Symmetric variational autoencoder and connections adversarial leam- ing. In Proceedings of AISTATS, 2018. URL
mir. mir.press/v8..
chen1 8b/ chen1 pdf.
10

---

Published as a conference paper at ICLR 2024
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
//
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roherts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:204.02311,, 2022. URL
https: arxiiiii....//.. org/ abss/204 02311.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuedd language models. arXiv preprint arXiv:2210.11411 2022. URL httpttt/////aae org/ abs//210. 11416.
Kamil Ciosek. Imitation learning by reinforcement learning. In ICLR, 2021. URL https:// http://wwweee/tooumiiiiiiii--eee
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty natural yes/no questions. In Proceedings of NAACL-HLT, 2019, URL https: /tap:p:://aaaaaaa aallannoloogyy..oooooooooooooooa org/N13 /N1 1300.
Wojciech M Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszez, and Max Jaderberg. Distilling policy distillation. In Proceedings of AISTATS, 2019. URL http: //proceedings mllr. press /v89/ /ttp///eecci./////aaaaaeee...eee
pdf..
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar.
Born again neural networks. In Proceedings ICML, 2018, URL htltttss//pprcceeinngg
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpuss 2019. URL
http://Skylinnd/ Be..000....gttte www.tb..hh.......g. io/ /pppn/eeeeetttorppss....
Google. Bard, 2023.
URL https: wow.g///ecohooooggg/////////////
Arnay Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms. arXiv preprint
wrXiv:2305.15717, 2023. URL https://axxiv... http:/xxxivv.rrr..
httg/abb/2305.55111
Tuomas Haarooja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with
deep energy-based policies. In Proceedings of ICML, 2017. URL httt::
www.ppres//vl. www.ress//7//haarooo/ http///aaannaaaaaaaaaaa wtwl ?ref=https http:::::///ggtttthhhee com.
.tmll
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, et al. Pre-trained models: Past, present and future. Al Open, 2021. URL https::/tsstt..
$2666651021000231
Yongchang Hao, Yuxin Liu, and Lili Mou. Teacher forcing recovers reward functions for text gen- eration. In Proceeingy of NeurIPS, 2022. URL https: : httptts://ppeeeeiiiiiieeee wwwp//ppweeeew.oo////p/.. wtwpeerreeeeeeee/ddeeeee.e ntt.pdffidddddeddddodd gypPuWUC3.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1503. 2015. URL wwtpt::////xxiiiee httpp//////aawoooo////////////// http://.wwrppp///////00///
02531. .df.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In Proceedings of ICLR, 2020. URL https: wpwprevien.eeeeeeeoo htwpeeww.eeeeeeee nwt/pdffffft FFgG.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. In Proceedings of ACL, 2023. URL https: //aclantho www.oolooyy....ooo ogg. 2223..22...0 acl-lonc 806.

---

Published as a conference paper at ICLR 2024
Ferene Huszár. How (not) to train your generative model: Scheduled sampling, likelihood, adver- sary? arXiv preprint arXiv:1511.5101, 2015. URL http://axxvvvvo/ org/ halg//ab////e abs/1511. 05101.
Haozhe Ji, Pei Ke, Zhipeng Hu, Rongsheng Zhang, and Minlie Huang. Tailoring language gen- cration models under total variation distance. In Proceedings of ICLR, 2023a. URL https:
/ //openreview. net http:/ddddddddLLLIIE.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey hallucination natural language generation. ACM Computing Surveys, 2023b. URL https: / dl. acm. org/ doi /10. 1t/./11111111 1145/3571730.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled reg- ularized optimization. In Proceedings ACL, 2020. URL https: aclanthoooooyy org/ 2020. acl - main. 197 .dd..
Gou Jianping. Yu Baosheng, Stephen Maybank, and Tao Dacheng. Knowledge distillation: A survey. I/CV, 2021. URL https://link.... /tt::/linnk
111263-021-0140
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In Findings of EMNLP, 2020. URL
https://aclanthin
org/ 2020. findings ew...em..pp.p............. emnlp.372/
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020. URL https: /ttpsi///axvvv.a//a org/ abs/2001. 08361.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of EMNLP, 2016. URL https: // //caaaannnooooyyyyy wwgp///-11111 pdf.
Hyoje Lee, Yeachan Park, Hyun Sco, and Myungjoo Kang. Self-knowledge distillation via dropout. Computer Vision and Image Understanding, 2023. URL https: wwww sciencedieeec.c........... com/ htmm//cienee/artiin wtt.e//rttieeeeeeeeeeeeeeeeeeeee abs /ttpe////////////11.111111112330 012.177711422000055
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems, arXiv preprint wxXiv22005.01643 2020. URL
hittps://at //tp://aavvvvv//a///aa/eeaeaaaaa org/ hdtf//df//// 2005. 01643.pdf.
Jianqiao Li, Chunyuan Li, Guoyin Wang, Hao Fu, Yuhchen Lin, Liqun Chen, Yizhe Zhang, Chenyang Tao, Ruiyi Zhang, Wenlin Wang, et al. Improving text generation with student-forcing optimal transport. In Proceedings of EMNLP, 2020. URL https: httsss///llaaahooggo..o org/ 2020. emnlp-main. 735. pdf.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A ddiversity-prooone objec- tive function for neural conversation models. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of NAACL, 2016. URL https: //a//aaaaaaaalloaeeeeeeeaaaaaaaa
org/ Arg/N166 MNNN-101110 1014.
Kevin J Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and Lawrence Carin. Mix (kd): Towards efficient distillation of large-scale language models. In Pro- ceedings of ICLR, 2021. URL https: / wwtp//wwwwreeeeeeeeeee/ee
Chin-Yer Lin. ROUGE: A package for automatic evaluation of summaries. In Proceedings of Text Summarization Branches Out (ACL 2004), 2004. URL http://fdduuuuueeeeeoooooooo d= w04-1011
net / forum?
.org/
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pre- training approach. arXiv preprint arXiv:1907.11692 2019. URL https:/ http///axxvvvrr/// abs/1907 10692.69........ 1692.
12

---

Published as a conference paper at ICLR 2024
Andrey Malinin and Mark Gales. Reverse KL-divergence training of prior networks: Im- proved uncertainty and adversarial robustness. In Proceedings of NeurIPS, 2019. URL
neurips cc cc/paper files/ https//ppp//22//111122 B3822 -eeeeeeeaaaaaaaaaaaaaaeeee Pap-- pdf.
/tttss:/proceeteee
Tom Minka et al. Divergence measures and message passing. Technical report, Cite-
seer, 2005, URL https::/www..t.... mi wwtp:wwww.ooo....cmo com/ eereeennusss/eeee wtt.//nnn//eeeeeeeeeeeeieeeeaeee raa-ua/reesarcchh/pcooooee
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Has- san Ghasemzadeh. Improved knowledge distillation via teacher assistant. Proceedings of AAAI, 2020. URL https://ojs aaai org/index. 5963/5819.
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. CVPR workshops, 2019. URL https:
Measuring calibration deep learning.
cotc://cccccccnttctoc/ceoooo / ///pppccccess....c/o thecvf com cottent _tcett___________ http://www.....ooooooo////////aa
Measuring_Calin
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative ral samplers using variational divergence minimization. In Proceedings of NeurIPS,
2016. URL https://proceediin neurips ccppaper files/paper/200/ hast//eaebbeee722555ooffffo 94 -ww.ettttdbbttttt.......... html.
OpenAl. OpenAl: Introducing ChatGPT, 2022.
URL
chatgpt.
OpenAl. GPT-4 technical report, 2023. URL WwK.httpp///rdt..ppaa...om pdf.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Proceedings of NeurlPS, 2022. URL https://arxi:: stg/abs/22033 02155.
Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. In Proceedings of ICLR, 2021. URL https: h/tp:://///pperviiieee httppparwweeeeeeeeeeee/aaaoooooo heee:ppdP..--Roox-uulll..
Baolin Peng, Chunyuan Li, Pengeheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. arXiv preprint arXiv:2304.03277, 2023. URL https: arxiv. wrt.ii.orgg/bb///444444 03277.
Doina Precup, Richard S Sutton, and Satinder P Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of ICML, 2000. URL https: //dl. acm. org/doi/100...ooooo 5555/ 655522665881 658134.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl Technical report, 2019. URL http:// www. www.persagen. com fine/siiess
www.auuuuggaaaggggggeee.ea pdf.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre. Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy dis- tillation. arXiv preprint 1rXiv:/511.0629) 2015. URI. arxiv. wwtp/////gg///////// 06295. pdf.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistiBBRTT, a distilled ver- sion of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019. URI. https: //taaaavvvv//////////eeee.aaavva www.v.org/pd...... 91.0 01 108.. .df.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, al. Multitask prompted training enables zero-shot task generalization. In Proceedings of ICLR, 2022. forum?
URL httpps://pppeeeeeeeeeeeeee
http//ppeeeevveeeeeeeeeeeeeeeeee net fer/forummmt d
13

---

Published as a conference paper at ICLR 2024
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. URL https:/ /arxiv. org/pdf/ 1707. 06347.pdE.
Iulian V Serban, Chinnadhurai Sankar, Mathieu Germain, Saizheng Zhang, Zhouhan Lin, Sandeep Subramanian, Taesup Kim, Michael Pieper, Sarath Chandar, Nan Rosemary Ke, et al. A deep reinforcement learning chatbot. arXiv preprint arXiv:1709.02349, 2017. URL https:
/ //arxiv. oot.//////aaa... abs attpaaaaa///// 170. 02349.
Joar Max Viktor Skalse, Nikolaus HR Howe, Dmitrii Krasheninnikov, and David Krueger. Defin- ing and characterizing reward gaming. In Proceedings of NeurlPS, 2022. URL https:
//openreview... net/ http:/dddddddx/b33x000333X
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of EMNLP, October 2013. URL https:/ http:///aaaaaanoaaaeeo org/
D13-1170/.
Kaitao Song, Hao Sun, Xu Tan, Tao Qin, Jianfeng Lu, Hongzhi Liu, and Tie-Yan Liu. Light- PAFF: two-stage distillation framework for pre-training and fine-tuning. arXiv preprint arXiv:2004.12817 2020. URL
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation pression. Proceedings EMNLP, 2019. URL https: htttttss///aaaanllllllloooo hwtp://..aaaa...ooooooooooooa/. wwt.gyyyorg/999111100
distilliiio for BERT model com-
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gra- dient methods for reinforcement learning with function approximation. Proceedings of NeurlPS, 1999. URL pttpp::/prrceediine neurips cc scc/pappr/199//1l1l pdf.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An foolluction-oollowiigg LLaMA model. https: // /githbb. cww.ct///////oo. ttttttttsuulaaa www.sssuaabssttafoodoooe stanford_aappat
2023.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos. Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint 0rXiv:2201.082333 2022. URL 1201.08253
org/ abss
Faraz Torabi, Garreut Warnell, and Peter Stone. Behavioral cloning from observation. In Proceedings of ICAI, 2018. URL https: / www. i ijcai....... httpjaaaaaaaaa......eeeeeeeeeeee http://prcceeenngg//111/////1//e ppf.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397. 2023. URL httptttps///aaateee 19971..99p.pd
Ben Wang and Aran Komatsuzaki. GPT-J-6B: 6 Billion Parameter Autoregressive Language Model, 2021. URL https://githbb com/ wwwikkinnoolonnoooo twansstttrsrssorrreeeee.eeo jax.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MiniLM: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. In Proceedings of NeurlPS, 2020. URL https:ttp///ccc.cctm /tRp:ttt/////rcediiin http///ooeedddee/ee ings nwwrips..s.... wwtppppp///2//0// file/
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. MiniLMv2: Multi-head self- attention relation distillation for compressing pretrained transformers. In Findings of ACI,, 2021.
URL htt.thttps//aclll. htt.::sss//allaahhoooogogo wwwpanhooo....///.......o.
ind mndinngaacl..ii 188.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolaasshi, Yeganeh Kordi, Amirreza Mirzaei, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran. Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. Proceed- ings of EMNLP, 2022. URL https:ttp//// httpe:///peeevvveeeeee n forum? d=Qx 1104
14

---

Published as a conference paper at ICLR 2024
Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self gene-rtee instructions. In Proceedings of ACL, 2023. URL https: htt////aaaantttoooooeee wrg//0200 aal-lccllnnn 754.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In Proceed- ings of ICLR, 2022a. URL
Jason Wei, Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. Transactions on Machine Learning Reseurch, 2022b. URL https: /ppnneview.
Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence-level knowledge distillation. In Proceedings ACL, 2023. URL htt.tttss/acaaattthtttte/ org/ 2023. acl- long. 605. pdf.
Ronald J Williams. Simple statistical gradient-following algoritts for connectionist reinforcement learning. Machine learning, 1992. URL htttt:ttt//inme springer com.a/tiic///////..ccc. 1007 9F0099266.
Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Fikri Aji. Lamini-Im: diverse herd of distilled models from large-scale instructions. arXiv preprint arXiv:2304. /4402, 2023. URL https: //t////aaivv/.///aa//eaa//eae//a org. htt//ddf//// 2304/ 14402. pdf.
Huan Zhang and Hai Zhao. Minimum divergence vs. maximum margin: empirical comparison on seq2seq models. In International Conference on Learning Representations, 2019. URL https:
/ //openeeview net / forum? d=H1 xD 9sR5Fm.
Rongzhi Zhang, Jiaming Shen, Tianqi Liu, Jialu Liu, Michael Bendersky, Marc Najork, and Chao Zhang. Do not blindly imitate the teacher: Using perturbed loss for knowledge distillation. arXiv preprint ar arXiv:2305.0500 2023. URL https: aataiiiiiiivi//ii wttg/////////e////////eeee///ee/ ptdf//305500 05010.pcl
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer lan- guage models. arXiv preprint arXiv:2205.01068 2022. URL 2205 01068.00......000 off.
/
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Erie Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of NeurlPS, 2023. URL https: http//eurppppppeeeeeeee cc/virtual/ 3023/poster/7344
Brian D Ziebart, Andrew I. Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropp inverse reinforcement learning. In Proceedings of AAAI, 2008. URL https: //cdn//ddd.../ aaa org/ AAAT/2008/ 0A000//AAAAA88220 22f..df.
15

---

Published as a conference paper at ICLR 2024
A DERIVATIONS
A.1 A PERSPECTIVE OF MINILLM FROM INVERSE REINFORCEMENT LEARNING
In Section 2.1, formulate KD as optimization problem of minimizing the discrepancy between the teacher distribution and the student distribution and finally reach the objective of minimizing re- verse KLD. Alternativlly, model's guidance, which resembles an agent learning from the feedback from an environment. Fol- lowing Pang & He (2021), we treat token generation step i, the student model chooses an action (token) from the action space (vocabulary) V condi- tioning on the state (prefix) (y<r, x) based on the policy (generation probability) go (vel
also regard KD as training the student model with the teacher
Markov Decision Process. At each time
From this perspective, standard KD corresponds to behavior cloning (BC; Torabi et al., 2018) in imitation learning (Ciosek, 2021). However, BC is known under-perform Inverse Reinforcement Learning (IRL; Ziebart et al., 2008), another imitation learning method that first recovers a reward model from the enviromment demonstrations and then trains the policy with the reward using policy optimization algorithms (Sutton et al., 1999; Schulman et al., 2017). Therefore, in the KD scenario, we seek to first induce a reward ryye, (y<r,, +)) from the environment (the teacher model) and then train the student model to maximize the reward as the objective. We take the maximum-entre enttrop inverse reinforcement learning framework (Ziebart et al., 2008; Chan & van der Schaar, 2021) and thus the Q-function (v,, (3 x)) in the environment satisfies the soft Bellman Equation:
Q1t,, (y<s, (yee (z) y log Σ explQ(v' (Ugr, #))]
(8)
y'EV
We follow Hao et al. (2022) to parameterize the Q-function as Q (Jer, x)) = f(,, (yer, x)) and assume y = 1, where f(ye, (H<t,z)) is the output logits of the teacher model". Then, the reward is given by: r(vt, (v<r, x)) f(Vt, (Ver, x)) log
explf(v/ (Vs),
(9)
To maximize the reward, we apply maximum-entrooy reinforcement learning Haarojaa et al. (2017), whose learning objective is
max J(0) = max
E
Σ
[r(ve, m)) + H [go (=((l)e,,)))]
(10)
0
where H (-1ye m = Eye ( m) log go (-1)<,, is the entropy of the student model distribution at the time step
Equivalence Between Objectives We prove an approximate equivalence between Eq. 10 and Eq.
1. We first rewrite the summation of the reward Σ r(y,, (y<t, x)) by the associative law:
Ml
ryye, (z)
Σ
f(v., (v<r, log Σ expif(y' (ys., x))]
(11)
v'ev
M Σ
f(y1, (ve1, +
f(ve, (V)c, - log Σ
explf(v Ex
(12)
v'eV
- log Σ
(13)
yev
f(ue, (Ver, - log Σ
explf(y'
(14)
Veev
exp(f(ye, exp(f(u)((<c)x Ever exp(f(e' (3)()(((()))))))))) x)))
log
(15)
log logr(veluee..))
(16)
"The teacher model's distribution satisifies ((rllee..
espl flee (1)c))))))
= Evev exp(fty (V
16

---

Published as a conference paper at ICLR 2024
Then, [(0) can be approximately rewritten as:
l
[(0)
E
1) (o0
(17)
B
logp(veleet, log x1)
(18)
a P-
y-uu (e)
= (L((o)lp)
(19)
L(0).
(20)
Therefore, maximizing J(0) is approximately equivalent to minimizing L()).
A.2 DERIVATION OF EQUATION 2
We compute the gradient of C()) KL[ge||p] with respect to 0 using the Policy Gradient Theo- rem (Sutton al., 1999):
log p(y( x))
V (0) =
E wmPe
(21)
go (1/ (z))
gr (2)
(1 (m) log p( (e)
dyda
(22)
{ ax)
/ 2o a)) log
p(y [z)
/ log p(u [e)
dyrda
(23)
2o ( m))
= f go x)) x)(
/ go (v)l))) log p(ylæ) log go (v)l))dddd
log
-
()(y)z))
(24)
(log p(yla) 1)V log (311()))
E Rh youqa 2))
(25)
(o
log P(u Vey, .x)
E
1) log go
(26)
(yte ar)
(2))
T
log p(yer w))
E
V log ())
(27)
(o (V
where Eq. 27 based on the fact that log (ueleee obtain Eq. 2.
yer, x) can only affect tokens at 2 positions in y.
By setting R. Σt' log plur ())).)) golve )))
A.3 DERIVATION OF EQUATION 3
To derive Eq. 3, we first denote:
(V Single
E
rr
Be (c)
(28)
T
(VL)
E
Σ
Res V (x)
y- (1)))
1=1
17

---

Published as a conference paper at ICLR 2024
Then, we-te (() as:
T
VL(0)
Σ
E(Re
RR 1)V log (oglo(
(29)
yougol (ae) t=1
T
Re+1 log (w
(30)
pa go (a) tm1
p tyel x))
1) log go (velleee
log
(31)
qo (uol
yrugel (ae) t=1
log p(u Wer, ()) xt) qo (evee m) -1)
=(V =VVLooone
E
log go v)e,
(32)
Pon yoiiga (ae)
1 i (.)))
log 90 Wece, x)
= -VV Lons
E somppp yrugga (a))
E
(33)
Vi (-)
=( Long
E ae 1r- (-12)
Σ
V
E (t)()
[rd]]
(34)
=VV + (V Singll,,
(35)
where Eq. 33 uses the product rule of the gradient and -log
plu. (ecam) gol V)).)))
B ALGORITHM DETAILS
Algorithm MINILLM:
Knowledge Distillation of LLMs
Input: Conditional generation dataset D consisting of prompts and ground -ruth responses Pre-training corpus Dyr consisting of long-document plain texts A teacher model with output distribution
An initial student model pre-trained Learning rate y,; Batch size M; Clipping Threshold
Dyr, with the output distribution goo
Output: A student model with the output distribution qo
Fine-tune the student model from Og on D supervised by the ground truth responses and choose O with the lowest validation loss.
repeat
Sample a mini- batch of prompts from D and collect responses from ğ to get S [(xx'' y y
Sample mini-batch D' PT (d { m m from Drr
Compute (V Single
yes Ei We V Enee 20 x) log p(ve vc).m)
Eq. 5
avtv (e)c.
Compute (VL) Norm JM En.ves Dn R+t1 min (())),, clip(ar(0),
111+0 e)),
where (0) P(ae (2.))
Sa(ar v=ccmm)
Eq. 5, Eq.
Compute the gradient the language modeling loss: VLT EdeDh V Flog
Update model parameters: 0 - 0 [(V (V hicem + [ until converge and return go
C EXPERIMENTAL DETAILS
C.I TRAINING DETAILS
Baselines Our baselines include SFT w/o KD, KD, and SeqKD. For models with less than 1.3B parameters, we search for the learning rates in [5e-4, le-4, 5e-5], the batch sizes in [32, 64], and train these models for 20 epochs. For other models, we search for the learning rate in [5e-5, Ie-5, 5e-6), the batch sizes in [32, 64), and train these models for 10 epochs. For KD, we follow Song et al. (2020) to mix the distillation loss with the language modeling loss on the ground truth responses by mixture rate of 0.5. The checkpoints of each baseline selected by the Rouge-L (Lin, 2004) scores on the validation set because, as stated in previous works (Wang et al., 2022; Ouyang et al., 2022), also find that Rouge-L is better correlated with human judgments.
18

---

Published as a conference paper at ICLR 2024
Below is an instruction that describes task.
Write a response that appropriately completes the request.
### Instruction:
(instruction)
### Input: [input)
### Response:
Figure The prompt wrapper for training and evaluation.
Below is an instruction that describes task. Write response that appropriately completes the request.
### Instruction:
Determine the sentiment of the input sentence. Please respond as positive or negative.
### Input:
[sentence] }
### Response:
Figure 10: Zero-shot text classification prompt for SST2.
MINILLM As stated Section 2.3, training of MINILLM has two phases which is similar to Reinforcement Learning from Human Feedback (RLHF:Ouyang al., 2022).
We fine-tune the student model on the instruction-responn training set to get starting point for the subsequent MINILLM training. We fine-tune the model for 3 epochs using the best learning rate and batch size of the corresponding SFT w/o KD baselines. Note that different from the SFT w/o KD baseline, we select the checkpoint with the lowest validation loss, not the Rouge-L score in this phase.
Phase 1:
Phase We continuously train the model from Phase as described in Algorithm B using learning rate 5e-6, mini-batch size 64 in all cases. The training and validation set are same as in Phase Similar to Ouyang et (2022), we collect 256 sentences at once and adopt 4 inner epochs when doing the policy optimization. The clipping rate is set to 0.2, and the max length of model 512. We use temperature = when sampling from go. train the model for 5000 steps and select the final checkpoint using the Rouge-L score the validation set. Our experiments are based on the NVIDIA V100 32G GPUs. Distilling LLaMA-7B from LLaMA-13B takes less than 10 ours 16 GPUs.
C.2 AUTOMATIC EVALUATION DETAILS
During the evaluation, we sample the responses from each model using temperature 1, a max- length limit of 512, and random seeds [10, 20, 30, 40, 50]. Similar Taori al. (2023), adopt prompt wrapper shown in Figure 9 to convert each instruction-respons pair to sentence. For the classification tasks in the "Calibration" paragraph of Section 3.3, we prompt the model to do zero-shot text classification with the prompt in Figure 10 and 11.
C.3 HUMAN EVALUATION DETAILS
Following Peng et al. (2023), we use SelfInst (Wang et al., 2023) to do the human evaluation. We ran- domly sampled 50 prompts because found that more prompts do not affect the results much. We
19

---

Published as a conference paper at ICLR 2024
Below is an instruction that describes task.
Write a response that appropriately completes the request.
### Instruction:
Read the input passage and answer the question: {question)? Your answer should be "Yes" or "No".
### Input:
[passage]
### Response:
Figure 11: Zero-shot text classification prompt for BoolQ.
Below is an instruction that describes task, paired with input that provides further context. Write response that appropriately completes the request.
### Instruction:
Desk jobs require writing lot of emails, it isn't surprising get tired of repeating ourselves. Come up with several synonyms for the given word.
### Input:
Sincerely
### Response:
#*###
Answer #1 #*#*#
Fondly, affectionately, lovingly, tenderly, honestly, truly, faithfully, devotedly, passionately
#ННН# Answer #2 #*### Faithfully, Gullibly, Humbly, Piously, Strangely, Weirdly, Yours truly
1: Answer #1 is better
2: Answer #2 is better
3: Tie
Your choice:
Figure 12: The prompt wrapper for training and evaluation.
ask the annotators to compare the responses generated by the baseline models with the MINILLM and decide which response is preferred or neither of them significantly better. Note that which model the responses come from is invisible to the annotators. The interface presented to annotators shown in Figure 12.
C.4 DETAILS ABOUT GENERATION DIVERSITY METRICS
In Table 3, we report the distinct 4- 4-grams (Dist-4) and the language modeling loss (Loss) on the test sets. More details about these two metrics are follows:
"Dist-4" is a fraction: N/C, where N the number of the distinct 4-grams the generated responses and C is the total number of 4-grams. is widely used metric to measure the gener- ation diversity of a language model (Li et al., 2016). The (N/C)s on the Dolly test set across random seeds shown in Table 5. Table 3 reports the average values across the 5 seeds.
20

---

Published as a conference paper at ICLR 2024
Seed
10
20
30
40
50
Model
Teacher
23562 23696
23653 / 23834
24306
24488
24207 /2481
23803 / 23967
KD
25889 26064
24024 24197
25663
25843
25611 25763
26178 26339
SeqKD
25358 25519
25631 25822
26190 / 26370
25574 / 25748
26295 26522
MINILLM
24187 24458
25011 / 25272
25100 / 25436
24067 / 24312
25205 25519
Table 5: The (N/C)s, where .N is the number of the distinct 4-grams in the generated responses and C' is the total number of 4-grams. We report the numbers computed on the Dolly test set when evaluated with random seeds [10, 20, 30, 40, 50].
We would like to request your feedback on the performance of two assistants in response to the user instruction and input displayed above. Please rate the helpfulness, relevance, accuracy, and level of detail their responses. Each assistant receives overall score scale of 10, where higher score indicates better overall performance.
Please first output single line containing only two values indicating the scores for Assistant and 2, respectively. The two scores are separated by space.
In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were pre- sented does not affect your judgment.
Figure 13: GPT-4 evaluation prompt.
"Loss" is the negative log-likelihood loss the test Dresi Ex log to (z))..) measures the mode coverage of the real data distribution because it is essentially the forward KLD between the real data distribution and the model output distribution. This relates to diversity in the ability to generate different generations given one context with different random seeds.
C.5 EXPOSURE BIAS ANALYSIS
Following Arora et al. (2022), we compute the ExAccErr with the following formula:
ExAccEmrl)) = R(I) - le(1) x 100%,
(36)
le(l)
where R(() is the accumulated regret of imitating the teacher distribution p at the time step during the free-run generation:
T
plyeyyee yere x) go (Vi (e
R(1)
Σ
E
log
(37)
==1
vi-p(-llees
and e(() is the average per-step error between go and using the oracle context sampled from p as the prefix:
log Plue x) (o y<t,
<())
(38)
T
t=
yerupl p (e
Intuitively, the regret qo during generation is made of two parts: the error to estimate given prefix. The former is
the oracle context and the error caused by the low-quality calculated by le()), and the latter reflects the exposure bias. Therefore, ExAccErr measures the relative error caused only by exposure bias.
21

---

Published as a conference paper at ICLR 2024
Model
#Params
Method
Dolly
SelfInst
Vicuna
13B
Teacher
79.0
75.5
65.1
SFT w/o KD
73.0
69.2
61.6
LLaMA
KD
73.7
70.5
62.7
7B
SeqKD
73.6
71.5
62.6
MINILLM
76.4
73.1
64.1
Table 6: Evaluation results by GPT-4 feedback based LLaMA family. We report the average
GPT-4 feedback scores across random seeds. The best scores of each model size are boldfaced.
Model
Method
Dolly
SelfInst
Vieuna
S-NI
UnNI
GPT-J-6B
Teacher
27.3
17.3
17.4
28.0
33.6
SFT w/o KD
25.4
12.4
16.1
21.5
27.1
KD
26.7
13.4
16.4
25.9
33.2
GPT-2-760M
ScqKD
26.0
14.0
15.3
25.5
32.5
MINILLM
25.8
16.3
19.1#
27.1
35,5*
SET w/o KD
27.6*
14.3
16.3
27.6
34.6*
KD
26.6
14.5
16.5
27.6
34.9*
GPT-2-1.55
SeqKD
27.0
13.6
16.9
28.0
34.2*
MINILLM
25.9
16.6
19.4*
28.5
35,9*
SET w/o KD
26.8
15.8
17.0
26.5
31.6
KD
26.7
16.0
16.9
27.2
32.7
GPT-Neo-2.7B
SeqKD
25.6
16.2
16.9
26.1
32.9
MINILLM 28.5
17.1
18.6*
29.8
35,4
Evaluation results when GPT-J is the teacher model. We report the average R-L scores across 5 random seeds. The best scores of each model size are boldfaced, and the scores where the student model outperforms the teacher marked with *
Table 7:
D ADDITIONAL RESULTS
D.1 GPT-4 FEEDBACK SCORES
We include the GPT-4 feedback scores (Zheng et al.. 2023) on the LLaMA family a supplementary evaluation metric, by asking GPT-4 to compare modeel-generate responses with the ground truth answers and raise 1-10 scores for both responses. We apply the prompt in Figure 13 and set the temperature 0.7 to call the GPT-4 APIS. We report the ratio of the total score of model responses and ground truth answers. This metric is only applied to Dolly, SelfInst, and Vieuna. The results in Table 6 indicate that MINILLM has the best overall performance.
D.2 GPT-J AS THE TEACHER MODEL.
We present the evaluation results when using GPT-J as the teacher and GPT-2-760M, GPT-2-1.5B, and GPT-Neo-2.7B (Black et al., 2021) as the student in Table 7. MINILLM outperforms the base- lines in most cases.
D.3 SCALING LAW OF TEACHER BASED ON THE OPT FAMILY
We present the performance of MINILLM and SegKD when scale up the sizes of the teacher models in Figure 14. Similar to the observations in Section 3.3, MINILLM constantly performs better and distills better student models from larger teacher models.
'We use the ChatGPT's generation (OpenAl, 2022) for Vicuna's ground truth responses.
We use the APT version of 2023-3-15.
22

---

Published as a conference paper at ICLR 2024
Average Rouge-1
24
WiiLLM
MLLLM
22
SoaKD
Studort 138, Wo KD
20
30
68
138
[0, 5]
[6, 10]
[11. +
of teacher parameters
Ground Truth Length Range
Figure 14: The scaling law of teacher based on the OPT family models. We compare MINILLM and SeqKD with OPT-1.3M as the student and OPT 2.7B, 6.7B, and 13B as teachers.
Figure 15: The Rouge-L scores of the dis- tilled models against the SFT models on the different evaluation subsets of UnNI split by the golden responses" length.
35
Validation Rouge-L
CLS
Inst.
MINILLM
70.2
52.8
1.3B
w/o Ler
65.7
53.2
25
GPT-2-15MM
MINILLM
78.8
71.2
OPT1338
LLaMA-78
w/o LPT
74.3
71.1
0.2
0.4
0.6
Value of
Table 8: The effect of adding the pre-training
Figure 16: Effect of the a value in the teacher mix-in exploration on the validation Rouge-L score. Larger models more robust a.
loss. "CLS" is the average accuracy scores on SST2 and BoolQ. "Inst." is the average Rouge- L score on Dolly, SelfInst, and Vicuna.
D.4 PERFORMANCE OF RESPONSE LENGTH ON U-NI
The performance on different U-NI subsets split by the length of the ground truth response is shown in Figure 15. We have the same observation as in Section 3.3 that on short responses, all KD methods perform similarly, and on long responses, MINILLM outperforms other methods.
D.5 MORE ABLATION STUDIES
Effect of Teacher-Mix-in Strength a In Figure 16, we plot the best Rouge-L. scores on the vali- dation set of GPTM,-125MM OPT-1.3B, and LLaMA-7B using GPT-2-1.5B, OPT-13B, and LLAMA- 13B as the teachers, with different teacher-mix-in strength a in MINILLM. & 0.0 means we only sample from the student distribution, and when a 1.0, sample entirely from the teacher distribution. We find that a = 0.2 is generally suitable across different model families and sizes, and larger models are more robust the choice of
Effeet of Adding Pre- Pre-Training Loss In Table 8, study the effect of adding the pre-training loss Algorithm B by comparing MINILLM with variant where the language modeling loss on the pre-training corpus is removed (wlo Ler). We have similar observation as Ouyang et al. (2022) that adding the pre-training loss helps to preserve the abilities on canonical NLP tasks while keeping the performance cestruction-folloviiii tasks nearly unchanged.
E CASES
We provide some cases generated by the models distilled by different methods based on the LLaMA model family in Table 9. The prompts are sampled from the SelfInst dataset. We find that MINILLM generates more detailed and accurate responses compared with the baselines.
23

---

Published as a conference paper at ICLR 2024
Case #1
Instruction
During writing, we added an asterisk for the word that did not come to mind. You will need provide several examples to demonstrate all the words that can be used in the sentence instead of the asterisk.
Input
we to know
SFT w/o KD
We can use "know" in the sentence instead of "we * to know"
KD
we went to knowl, in we will go to know in we can go know' in we may to know',n in we should go to know', i we must go to know
SeqKD
can do to know' n we should do to know would we do know do to know in we ought to do to know
MINILLM
We can use the following words in place of the asterisk:),n want), have n used' n ought', desire n need',n must
MINILLM w/o teacher-mixed
We to know
Case #2
Instruction
Take the title of the product and extract its attributes. The attributes in this case refer to the characteristics of the products, such as their brand and color. Your answer must list following the format of "attribute: value".
Input
SAMSUNG 980 PRO SSD 2TB PCTe NVMe Gen Gaming M.2 Internal Solid State Hard Drive Memory Card, Maximum Speed, Thermal Control, MZ-V8P2TOB
SFT w/o KD
Attribute: Maximum Speed', n Value: 3500 MB/s
KD
attribute: brand, attribute: color, attribute: memory card, attribute: maxi-
mum speed, attribute: thermal control
SeqKD
brand: Samsung, color: black, size: 2TB, interface: PCIe NVME Gen 4,
design: M.2 Internal
MINILLM
brand: SAMSUNG', color: 980 PRO'(n capacity: 2TB | interface:
PCIe NVME Gen 4 speed: Maximum Speed),n threshold: Thermal
Controll (n form factor: M.2 Internal
MINILLM
SAMSUNGC 980 Pro SSD 2TB PCIe NVME Gen Gaming M.II Inter- nal Solid State Hard Drive memory Card, Max Speed, Thermal Control, AP1500K
w/o teacher-mixed
Table 9: Eestruction-folloviie cases from the SelfInst dataset. MINILLM better follows the in- structions and provides more detailed and accurate responses. Without the teacher-mixed sampling strategy in Section 2.2 the distilled model outputs short responses (Case #1) or simply repeats the input (Cases #2).
24


Reviews:
[{"review_id": "U9xCYMADYu", "reviewer": "Reviewer_v2PG", "summary": "The paper proposes a method called MiniLLM for knowledge distillation of large language models (LLMs). The method focuses on distilling smaller language models from generative larger language models. It replaces the forward Kullback-Leibler divergence (KLD) objective in standard knowledge distillation approaches with reverse KLD, which is more suitable for generative language models.", "questions": "See Weaknessed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces an application for knowledge distillation of generative language models.\n- The proposed method is supported by well-structured experiments and evaluation on various datasets, demonstrating its effectiveness in generating more precise responses with higher overall quality, lower exposure bias, better calibration, and higher long-text generation performance.", "weaknesses": "(major) The novelty of this paper is limited. It is just a simple application of reverse KL Divergence to knowledge distillation. However, distill models with reverse KLD have been researched before. The claim in the abstract \"how to effectively distill the knowledge … is still under-explored\" is not convincing. For example [1]. More importantly, this paper is not cited by the authors.\n\n(minor) In Table 1, the student model even outperforms the teacher model which lacks intuition. Although the authors attributed such results to the exposure bias issue of teacher-forcing, I doubt there is an overfitting problem with the experiments. Could the authors provide the hyper-parameters and the variance of each experiment?\n\n[1] Self-Knowledge Distillation via Dropout", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4dpPpr1zQI", "reviewer": "Reviewer_HmVQ", "summary": "This work primarily concerns distillation of large language models (LLMs) into smaller, more portable versions. Compared to standard distillation approaches, the authors advocate the substitution of reverse KL instead of the more typical forward KL divergence objective. This incentivizes the model to pursue mode-seeking behavior, rather than coverage-seeking behavior, leading to more precise answers, with a lower probability of generating data outside of the teacher distribution. A policy gradient objective function is modified with single-step decomposition, teacher-mixed sampling, and length normalization to further improve optimization. Experiments with three families of LLMs (GPT-2, OPT, LLaMA) on a variety of benchmarks show that the proposed method (MiniLLM) indeed outperforms the baseline distillation approaches, across a range of student sizes.", "questions": "Q1. Given that forward and reverse KL each clearly have their own advantages and disadvantages, why not use both, e.g. as in [a]? See also [c] for a more general treatment of f-divergences for generative modeling.\n\nQ2. The student models outperforming the teacher model in Table 1 is somewhat surprising. Why do you think this is the case? Does it have to do with the specific choice of metrics? I’m somewhat doubtful for example that a 120M parameter GPT-2 model is truly outperforming the 1.5 B parameter teacher model.\n\n[a] Chen, Liqun, et al. \"Symmetric variational autoencoder and connections to adversarial learning.\" AISTATS, 2018.\\\n[c] Nowozin, Sebastian, Botond Cseke, and Ryota Tomioka. \"f-gan: Training generative neural samplers using variational divergence minimization.\" NeurIPS 2016.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "## S1. Relevant topic\nGiven the recent advances and widespread usage of LLMs, ways of compressing models for faster and cheaper deployment can have significant real-world impacts in promoting wider spread usage of such models. The proposed approach shows promise being able to reduce model size without suffering as much performance decay as the baseline approach (Fig 1). This can have the effect of allowing more people to run LLMs, with the hardware they have available.\n\n## S2. Experiments\n1) Models: The experiments are on 3 model families (GPT-2, OPT, LLaMA), across a range of student model sizes (3 each for GPT-2 and OPT). This is a pretty good spread and helps give a sense of how well the proposed method generalizes, and the relation between student model size and performance.\n2) Evaluation: Models are evaluated on DollyEval, SelfInst, VicunaEval, S-NI, and UnNI. Rouge-L, GPT4, and human evaluation are used as metrics. Results are reported on 5 generations from separate random seeds, which gives a better sense of the evaluation’s reliability. \n3) Results: The main results suggest that MiniLLM indeed outperform the baselines, and occasionally even the teacher in certain cases. Taken at face value, this is pretty impressive, as some of the student models are considerably smaller (>10x for GPT=2, 10x for OPT). However, I have some concerns about the metrics and whether they’re capturing the full picture here (see W1.1).\n\n## S3. Writing\nI’ve listed a few miscellaneous corrections below, but overall, the writing is fairly clear and well-written.", "weaknesses": "## W1. Reverse KL vs Forward KL\nOne of the primary contributions of the paper is the substitution of reverse KL Divergence for forward KL divergence. This leads the student model to pursue “mode-seeking behavior”, as opposed to “coverage-seeking behavior”. While this does cut down on unrealistic generation samples, the trade-off is that such an approach will cause much of the long tail to be lost as well. This leads to a couple concerns:\n1) The loss of sample diversity is not captured by the paper’s metrics, which primarily focus on realism/precision, so the baseline forward KL divergence is at a distinct disadvantage. Specifically, the metrics specifically measure where forward KL is weakest (correctness of samples), while ignoring where it is strongest (sample coverage). As such, the evaluation is somewhat unfair. Some sort of metric that captures sample diversity may yield a different story.\n2) The long-tail knowledge of LLMs is arguably one of their most impressive and valuable properties, so sacrificing this in the name of realism is somewhat disappointing. In particular, this also raises potential ethical or fairness issues, as loss of diversity could lead to loss of minority representation or amplification of stereotypes.\n3) Why do we have to make this tradeoff in the first place? Why not use both forward and reverse KL (see Q1), as in [a]?\n\n## W2. Novelty/Contributions\nFrom the abstract and introduction, it would seem that that the primary contributions are a) the focus on white-box KD for generative LLMs and b) the substitution of reverse KL instead of the more typical forward KL. Additionally, an amalgam of modifications (single-step decomposition, teacher-mixed sampling, length normalization) improves the policy gradient objective function to the final form (Equation 7) used in this paper. Some concerns/questions:\n1) This is not a reason for rejection in and of itself, but individually, neither of these are particularly new concepts. White-box KL has been explored in the past, and the merits of forward vs reverse KL (as well as other divergences) for generative modeling has also been well explored. \n2) It’s not clear how related whitebox KD for generative LLMs and a reverse KL optimization are. In fact, they seem almost entirely orthogonal from each other.\n3) It’s not clear from the Methods section how whitebox KL was used in the method. Where does having the teacher model’s parameters play a role?\n4) The methods in Section 2.2 seems to be a series of cobbled together heuristics, and it appears that they aren’t necessarily novel either, as there are clear connections with (cited) prior work. I’m not necessarily saying that that’s a bad thing to have as part of the method, but it doesn’t appear to be something that should be counted as a contribution of this work.\n\n\n## Miscellaneous:\n- The name “MiniLLM” doesn’t fully capture the method. Yes, the model is a smaller LLM, but the same can be said of a distilled LLM learned by forward KL divergence as well, so the name fails to distinguish one of the primary points of the paper.\n- pg 2: “finite-number classes” => “a finite number of classes”\n- Appendix entries out of order\n- pg 5: “Ouyang et al. (2022)” citation at top of the page should be parenthetical?\n- pg 9: Sec 4 – Knowledge Distillation: [b] may be a relevant related work\n- pg 9: Fig 8: The y-axis says “Forward KLD”, while the caption says “reverse KLD”. Also, doesn’t this graph imply that w/o teacher-mixed sampling is better, if trained long enough?\n\n[a] Chen, Liqun, et al. \"Symmetric variational autoencoder and connections to adversarial learning.\" AISTATS, 2018.\\\n[b] Liang, K., et al. \"Mixkd: Towards efficient distillation of large-scale language models.\" ICLR 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GuIPtFb6kD", "reviewer": "Reviewer_BHa2", "summary": "This paper proposes to use Reverse KL Divergence for distilling large-language models. The paper starts from the Reverse KLD objective in section 2, describes the difference with forward KLD and its advantage that it is mode-seaking that is preferred when student has low capacity. Then in section 2.2 they review the challenges of optimizing for Reverse KLD and revisit ideas from prior work to improve it. To resolve challenges of reverse KLD, they propose 3 strategies to mitigate:\n1) Decompose gradient into the gradient for single-step prediction and long sentence prediction\n2) Prevent reward hacking by mixing the teacher/student distributions for sampling next token\n3) Normalize the reward to prefer longer sequences.\nThey refer to Reverse KLD together with their strategies as MiniLLM.\nIn section 3, they evaluate the effectiveness of MiniLLM on instruction-following generation tasks. They use various teacher/student architectures including GPT-2, OPT, and LLaMA. They compare to baselines with and without knowledge distillation. Section 3.2 provides positive improvements using MiniLLM and section 3.3 provides analysis that shows the method scales well, gives well-calibrated models, and generates diverse outputs. Section 3.4 provides ablations on the three elements of MiniLLM.", "questions": "- Page 2, introduction, line 11: How can one force q, the teacher, to do something? The teacher is not learnable. I’m assuming this is a typo and p/q_theta should be exchanged.\n- Why is this work “white-box” KD? How is the method using the parameters of the teacher and not just the predictions of the teacher, p(y|x)?\n- All experiments seem to be on instruction-following generation tasks. How would this distillation method perform for pretraining only? Can it help speed-up the pre-training of small models?\n- Table 1: Can you provide examples of cases where the student is better than the teacher and provide a qualitative analysis of why? 8% improvement should show consistent improved behavior.\n- Figure 5: Does MiniLM benefit more from scaling the teacher compared with SeqKD? Can you report the relative improvements of SeqKD and MiniLM as the teacher is scaled compared with a base teacher? If yes, it is useful for future scaling endeavors.\n- Figure 8: y-axis says “Forward KLD” but the caption says “reverse KLD”, which value is plotted?\n- Does the method have any hyperparameters specific to MiniLLM? For example, is there any thresholding of the ratio of q/q or p/q in Eq. 7 or epsilon in the denominator? If so, can you provide ablations?\n\nSuggestions\n- MiniLLM is a self-contradictory name as it expands to Mini Large Language Model. Please consider changing it, for example, to MiniLM.\n- Figure 1: Please consider adding more description of sequence-level KD in the caption or the intro close to the reference to Figure 1. A reader not familiar with the literature does not learn about SeqKD until page 5.\n- Eq 1 and A.1: For consistency it would help to use the KL with negative sign throughout (Eq. 1 is without negative sign but Eq. 8 is with negative sign). It would also help to highlight the difference in Eq 13 and 14 by color. It’s hard to notice the difference.\n- Eq 7: It might help to use single/long gradients to simplify this equation and other predefined terms. This equation is not easily digestible as a summary equation.\n\nTypos:\n- Page 2: “... approximately minimizes the forward KLD” -> “minimize”\n- Page 3: “... the quality of the each …” -> “the quality of each”", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Significant gains and improvements compared with various baselines. The results show that improvements increase as the teacher gets bigger and all students at all parameter counts improve. So consistently large improvements.\n- Figure 1: MiniLM is 5% better than SeqKD on GPT4 score.\n- Table 1: MiniLM is up to 10% better than SFT w/o KD while KD is up to 5% better and SeqKD is up to 1% better than KD.\n- Table 1: MiniLM is up to 8% better than the teacher while no other baseline surpasses the teacher.\n- Comprehensive evaluations and ablations.", "weaknesses": "- The results seem to point that Reverse KLD might be harder to tune and requires all the strategies in MiniLM for performing better than baselines. This can be a challenge for reproduction and further research. I am specifically pointing to Table 4 and comparing to baseline numbers in Table 1: Why is MiniLM without either length-normalization (DollyEval GPT-4 22.4) or teacher-student distribution mixing (36.1) is significantly worse than comparable SeqKD (41.2) or KD (40.3) or SFT w/o KD (38.6) in Table 1? Does that mean Reverse KLD is generally harder to train without these strategies?\n- Wall-clock time analysis of the method compared with baselines should be discussed. What is the training efficiency? How slow is the training with the MiniLM loss compared with SFT w/o KD, KD, and SeqKD? If the method is slower per iteration, what if baselines are trained for more iterations to match the wall-clock time of the method? Would they match the performance gains?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QbaMsCbvCV", "reviewer": "Reviewer_oknc", "summary": "For lowering computation cost required by large deep learning models, knowledge distillation (KD) is a popular approach. This study presents a knowledge distillation method applied to open-sourced large language models (LLMs). While the standard KD method uses a linear combination of cross entropy loss and Kullback-Leibler (KL) divergence referred to as forward KL divergence in this study, this study also examines a reverse KL divergence, which swaps teacher and student distributions and is used in computer vision and reinforcement learning literature. With the modified loss function, LMs trained on instruction-following datasets with teachers by their proposed approach (called MiniLLM) achieved higher average GPT-4 feedback scores than those trained with the same teacher model by a sequence-level KD (SeqKD) baseline.", "questions": "1. What is the definition of \"white-box\" KD/model in this paper? White-box in this paper sounds misleading. \n2. Why is the specific $w_t$ between Eqs. (5) and (6) expected to reduce the variance of the estimator in Eq. (5)?\n3. What is the definition of \"Exposure Bias\" ?(conceptual definition, not mathematical definition)\n4. What is \"the responses' distinct 4-gram proportion\"?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The reviewer wants to value originality of this study as this study is focused on open-sourced LLMs as targets for knowledge distillation, and black-box APIs can change their internal behavior without notice or proper versioning.\n- This paper seems to well describe the proposed method and cites prior studies that inspired the authors to introduce key concepts in their method such as reverse KL divergence.\n- It is empirically shown that MiniLLMs achieved better performance than KD baselines considered in this paper with multiple evaluation settings and instruction-following datasets. It is also notable that the overall trend in Table 1 seem consistent over different student models in a variety of model sizes.\n- The ablation study attempts to test multiple hypotheses made when designing the proposed loss function.", "weaknesses": "## Presentation\nThis paper needs to improve the presentation and writing.\ne.g.,\n\n- MiniLLM model must be tautology (Mini large language model model) and should be referred to as just MiniLLM instead\n- \"distill <student model> from <teacher model>\" sounds weird to the reviewer, and the reviewer suggests \"distill (knowledge of) <teacher model> into <student model>\"\n- Itemized lists in this paper look very packed. Did the authors change the format and reduce space between items?\n- \"generative LLMs\" and \"generative language models\" also sound strange as language models themselves are generative models. The reviewer suggests just skipping \"generative\".\n- \"the vocab size\" should be \"the vocabulary size\"\n- \"similar to Learning from Human Feedback (RLFH; Ouyang et al., 2022).\" misses \"Reinforcement\"\n- Use [] for the second equation in Eq. (5) as well\n\n## GPT-4-based evaluation\n\nOverall experimental designs in this study look good, but the reviewer has a big concern about evaluations involving GPT-4. Specifically, it is very questionable how scientifically meaningful the evaluations are when leaving all the evaluations to GPT-4, and the reviewer did not find any reasonable justifications of using GPT-4 as part of the evaluation process. Rouge-L should be sufficient for Tables 1 and 4, and the reviewer strongly recommends use of Rouge-L instead of GPT-4 feedback (score) for Figures 1 and 5. The reviewer will improve rating if GPT-4-based evaluations are removed and replaced with Rouge-L.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)