Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression
Authors: Jiduan Liu
Jiahao Liu
Qifan Wang
Jingang Wang
Xunliang Cai
Dongyan Zhao
Ran Lucien Wang
Rui Yan
Venue: EMNLP 2023 Findings (2023)
Keywords: Model Compression
LLM

Abstract: Large-scale pre-trained language models (LLMs) have demonstrated exceptional performance in various natural language processing (NLP) tasks. However, the massive size of these models poses huge challenges for their deployment in real-world applications. While numerous model compression techniques have been proposed, most of them are not well-suited for achieving extreme model compression when there is a significant gap in model scale. In this paper, we introduce a novel compression paradigm called Retrieval-based Knowledge Transfer (RetriKT), which effectively transfers the knowledge of LLMs to extremely small-scale models (e.g., 1\%). In particular, our approach extracts knowledge from LLMs to construct a knowledge store, from which the small-scale model can retrieve relevant information and leverage it for effective inference. To improve the quality of the model, soft prompt tuning and Proximal Policy Optimization (PPO) reinforcement learning techniques are employed. Extensive experiments are conducted on low-resource tasks from SuperGLUE and GLUE benchmarks. The results demonstrate that the proposed approach significantly enhances the performance of small-scale models by leveraging the knowledge from LLMs.

Pre-trained language models (PLMs), such as BERT/RoBERTa (Devlin et al., 2019; Liu al., 2019), have demonstrated exceptional performance across various natural language processing (NLP) applications. However, these models typically com- prise hundreds of millions of parameters, present- ing a substantial challenge for researchers due to Retrieval-based Knowledge Transfer: An Effective Approach for Extreme
Large Language Model Compression
Jiduan Liu , Jiahao Liu", Qifan Wang" ,Jingang Wang', Xunliang Cai²
Dongyan 2255,, Ran Lucien Wang', Rui Yan
Wangxuan Institute of Computer Technology, Peking University
Center for Data Science, AAIS, Peking University; 'Meituan; *Meta AI
SNational Key Laboratory General Artificial Intelligence; BIGAI, Beijing, China
"Gaoling School of Artificial Intelligence, Renmin University of China (liujiduan, zhaody jêpku. edu. cn, ruiyanêruc. edu. cn, wqfcr@fb. com (liujiahaol 2, wangj waaggingangg2,,.m cwwx i @me tuan. com, ran. wang . ma maathhegall... com
Abstract
their massive scale. As result, the full potential of large-scale pre-trained language models (PLMs) remains untapped. To tackle this challenge, multitude of model compression techniques have been proposed, encompassing knowledge distilla- tion (Sanh et al., 2019; Jíao et al., 2020; Passban et al., 2021), network pruning (Liang et al., 202 la; Gordon et al., 2020), quantization (Zhang et al.. 2020; Tao et al., 2022) and weight sharing (Lan et al., 2020).
Large-scale pre-trained language models (LLMs) have demonstrated exceptional perfor- mance in various natural language processing (NIP) tasks. However, the massive size of these models poses huge challenges for their deployment real-world applications. While numerous model compression techniques have been proposed, most of them are not well- suited for achieving extreme model compres- sion when there is significant gap in model scale. In this paper, we introduce novel compression paradigm called Retrieval-bass based Knowledge Transfer (RetriKT), which effec- tively transfers the knowledge of LLMs to ex- tremely small-scale models (e.g.. 1%). In par- ticular, our approach extracts knowledge from LLMs to construct a knowledge store, from which the small-scale model can retrieve rele- vant information and leverage it for effective inference. To improve the quality of the model, soft prompt tuning and Proximal Policy Opti- mization (PPO) reinforcement learning tech- niques are employed. Extensive experiments are conducted on low-resource tasks from Su- perGL.UE and GLUE benchmarks. The results demonstrate that the proposed approach sig- nificantly enhances the performance of small- scale models by leveraging knowledge from LLMs.
However, these model compression methods are not directly applicable to scenarios requiring high compression ratios, such as knowledge distillation. In such cases, the introduction of assistant mod- els (Mirzadeh et al., 2020; Son et al., 2021) often leads to decreased and unstable performance. Re- cently, there has been a growing interest in leverag- ing large language models (LLMs) (Touvron al., 2023; Zeng al., 2022: Ouyang al., 2022; Scao et al., 2022) that possess extensive language knowl- edge and can be effectively employed in various downstream tasks. Consequently, it is essential to explore methods for transferring this knowledge to small-scale models. Nonetheless, existing ap- proaches are inadequate for compressing LLMs due their exceptionally high compression ratios. Some prior research (Wang et al., 2022; Dai et al., 2023; Ubani al., 2023) has suggested utilizing LLMs for data augmentation and knowledge trans- fer to small-scale models, which allows the lat- ter to demonstrate improved performance on low- resource datasets. However, when tackling more challenging tasks like the SuperGL.UF benchmark (Wang et al., 2019a), the limited parameter size of small-scale models becomes a hindrance, prevent- ing them from effectively retaining the knowledge transferred by LLMss. Consequently, the perfor- mance enhancement achieved for scall-scale mod- els remains constrained.
1 Introduction
'Equal contribution.
Zhao (zhaody (cu.cucc) and Rui Yan (ruiyan @rw.edu..nnc..)
Corresponding authors:
Dongyan

---

To effectively transfer the knowledge of Large Language Models (LLMs) to small-scale mod- els, enabling them to efficienlly and accurately complete tasks, we propose a novel compression paradigm called Retrieval-based Knowledge Trans- fer (RetriKT). Our approach involves two main steps: knowledge extraction from the LLM to con- struct knowledge store, and subsequent retrieval of relevant information from the knowledge store by the small-scale model to accomplish the task. More specifically, we employ the technique of soft prompt tuning to fine-tune an LLM, ensuring that it generates in-domain samples. Additionally, we introduce the Proximal Policy Optimization (PPO) (Schulman et al., 2017) reinforcement learning al- gorithm to enhance the generation quality. As final step, the small-scale model learns to retrieve pertinent information from the knowledge store. We perform an extensive set of experiments on truly low-resource and challenging tasks sourced from SuperGLUE (Wang et al., 2019a) and GLUE (Wang et al., 2019h) benchmarks. The experimen- tal results demonstrate that RetriKT significantly enhances the performance of small-scale models and outperforms previous SOTA knowledge dis- tillation methods by leveraging the knowledge of LLMs. This indicates the effectiveness and prac- ticality of the retrieval-baid based knowledge transfer paradigm for extreme model compression.
2 Related Work
This paper involves three areas of work: knowl- edge distillation, reinforcement learning, and data augmentation.
2.1 Knowledge Distillation
We first introduce related work of knowledge distillation (KD), which can be categorized response-based feature-based, and relation-based KD. Response- based KD was initially introduced by Hinton et al. (2015), which transfers label knowledge by minimizing the KL-divergence be- tween predicted distributions of the teacher and the student. Building upon this concept, Sanh et al. (2019); Liang et al. (2021b) applied response- based KD to tasks such as masked language modeling and text classification,, resulting in smaller models with slight performance drops. Feature-based ap- proaches which are initially introduced by Romero et al. (2015), entail aligning the feature activations between the teacher and the student models. Build- ing upon this concept, more recent techniques have expanded the scope by incorporating hidden rep- resentations of the [CLS] token indicators (Sun et al., 2019), matching hidden representations of all tokens (Jiao et al., 2020), and introducing cus- tomized brasdd distillation (Sun al., 2020). On the other hand, relation-bases methods (Park et al., 2021; Liu et al., 2022a; Jiao et al., 2020; Wang et al., 2020, 2021a; Wu et al., 2023) aim to emphasize the importance of capturing and utiliz- ing relations among multi-granularity representa- tions in both horizontal and vertical directions.
Our contributions can be summarized as follows:
We propose new compression paradigm called Retrieval-based Knowledge Transfer, which aims to transfer knowledge from LLMs to extremely small-scale models. This paradigm addresses the challenge of achiev- ing extreme model compression when there is significant gap in model scale.
2.2 Reinforcement Learning
Recently, reinforcement learning (RL) has gained significant attention in the field of language mod- eling. This approach has been successfully ap- plied various language tasks, including summa- rization (Paulus et al., 2018; Ziegler et al., 2019; Stiennon et al., 2020; Wu et al., 2021), dialogue systems (Zhou et al., 2017; Jaques et al., 2020; Hancock et al., 2019), machine translation (Bah- danau et al., 2017; Kreutzer et al., 2018; Kiegeland and Kreutzer, 2021), 1), semantic parsing (Lawrence and Riezler, 2018), story generation (Zhou and Xu, 2020), and question generation (Pang and He, 2021). Concurrently, there has been a growing in- terest in using RI to align language models with human preferences across a wide range of language tasks. For example, Ouyang et al. (2022) employed
We introduce the reinforcement learning algo- rithm PPO to enhance the generation quality, and carefully design the reward function. This technique contributes to improving the diver- sity and accuracy of the knowledge extracted from LLMs used for knowledge transfer.
We conduct extensive experiments on low- resource tasks from the SuperGLUE and GLUE benchmarks. The results demonstrate that RetriKT significantly enhances the perfor- mance of small-scale models and outperforms previous SOTA knowledge distillation meth- ods by leveraging the knowledge from LLMs.

---

[CLS]
Soft Prompis
Soft Logits
R-(Rxoom 1RR boss BB
LLM
Reward Model
Reward
I. with soft P-tuning
b. Reinforcement learning
[CLS)
Key Knowledge
Similarity Distribution
Key ((CLS)
Value ([
Trnee False])
Small Scale Model
Open-book know ledge-store
sentenice emb
0.91,09
sentence emb
0.37
KI. Divergence
K Most Siniilar Samples
Similarity
Nearest Fimbedding
Value
sentence
[0.8,0.2]
Label
Value
Aggregation
emb
1.37,033
True
[0.1,0.]
False 0.33,03
Similarity Distribution
Retrieval-bused Knowlodge Transfer
The framework of RetriKT which consists of three steps: (1) fine-tune additional soft prompts for the LLM by supervised learning; (2) further fine-tune the soft prompts by reinforcement learning to enhance the generation quality: (3) create the knowledge store based on the knowledge extracted from the LLM, and teach the small-scale model how to retrieve relevant knowledge from (Retrieval Reereeeval-based Knowledge Transfer)
Figure I:
RL techniques, specifically Proximal Policy Op- timization (PPO) (Schulman et al., 2017), fine- tune large language model and align it with mod- cls of human preference.
prompts using the corresponding dataset (section 3.1). To further enhance the generation quality, introduce reinforcement learning and carefully design the reward function fine-tune the soft prompts (section 3.2). Subsequently, we extract the in-domain knowledge from the LLM and create a knowledge store (section 3.3). Finally, we enable the small-scale model to retrieve and effectively utilize the knowledge generated by the LLM to successfully perform specific tasks (section 3.4).
2.3 Data Augmentation
Wu et al. (2019) and Kumar al. (2020) have introduced method to create synthetic data by randomly masking words in the original training instances. Another works (Ding et al., 2020; Yang et al.. 2020; Anaby-Tavor et al.. 2020) involved utilizing Language Models (LMs) and Pre-trained Language Models (PLMs) to directly generate syn- thetic data for NLU tasks. Wang al. (20215 lb, 2022) proposed the use of hard prompts and soft prompts generate synthetic data. In related study, Liu et al. (2021b); Zan et al. (2022) have uncovered the complementary nature of PLMs and classical approaches.
3.1
Soft Prompt Tuning
To enhance the extraction of in-domain knowledge from the LLM, it is crucial to introduce appropriate prompts that precede the inputs, effectively guid- ing the text generation process (e.g. "Generate rte examples according the following keywords:"). However, relying solely on manual design often leads to limited diversity and effectiveness the generated knowledge. To overcome such limita- tions, we employ trainable parameters known as soft prompts, thereby replacing manual templates. We only need to update these soft prompts and keep all other parameters of the LLM fixed during training for each specific task.
3 Methodology
Our approach revolves around extracting knowl- edge from the parameters of Large Language Model (LLM) and leveraging it for the benefit of an extreme small-scale model through retrieval. We introduce the technique to extract in-domain knowledge from the LLM for each task by freez- ing its parameters and fine-tuning additional soft
More specifically, we utilize P-tuning v2 (Liu et al., 2022b), which involves incorporating a soft prompt into each Transformer (Vaswani et al.,

---

2017) layer. The soft prompt is represented as sequence of trainable vectors, denoted p (r).... (p'.....,P)) p/)) where j the j-th layer and k is a hy- perparameter which represents the prompt length. The i-th hidden states at the j-th layer, denoted as tel, can be expressed as follows:
accuracy of the knowledge store. Additionally, it ensures that the generated samples exhibit diver- sity, thus fulfilling the diversity requirement of the knowledge store.
More specifically, we utilize the original dataset to train classification model, which serves the reward model denoted as RM(). For each gener- ated sample, comprising both imput text and label, we assess the confidence score of the text's label using the reward model, thereby obtaining the ac- curacy reward Racuraacy Additionally, we evaluate the diversity of the generated samples by employ- ing the Belf-Bleu metric (Zhu et al., 2018 which calculates the Bleu score (Papineni al., 2002) considering other samples as references. Subse- quently, we calculate the diversity reward Raiversity as 1 b3. where b3 represents the Bleu-3 metric. To avoid the generation of overly simplistic pat- terns by the L.M,, we apply a length penalty. If the length of a generated sample falls below a cer- tain threshold min. the reward is multiplied by a discount factor:
P Trans H
i < k i> k fAj=00 (I)
Else
where Trans() represents the Transformer layer, H' represents all hidden states at j-th layer and wy is the word embedding of the input text.
In order to generate diverse samples, we adopt the approach presented in PromDA (Wang et al., 2022), which involves generating samples from both the Input View and the Output View. The Input View is conditioned on the keywords present in the input texts, which are extracted by the unsupervised keyword extraction algorithm Rake (Rose et al., 2010). On the other hand, the Output View is con- ditioned on the labels. Additionally, we train two sets of soft prompts, namely Pimput and Poutput- which are specifically designed to generate samples from the Input View and the Output View respec- tively. Each sample S = (Y, generated by the LLM consists of two parts, namely the label Y = 1 and the input text X = {= The supervised learning object for soft prompts Pimput and Pou output can be formalized as:
2 4min < mmin
BP = exp(1 Lmgin
(4)
where | represents the length of the generated sam- ple. The final reward for each generated sample is:
R- (Raceurayy + a Rddvveritt BP, (5)
Cinput
log(p(s, Paoput. K)),
where a is a hyperparameter to balance the weight of accuracy reward and diversity reward.
Following previous studies (Ouyang et al., 2022: Bai al., 2022), we employ Proximal Policy Op- timization (PPO) (Schulman et al., 2017) the reinforcement learning algorithm fine-tume the soft prompts Pinputt and Poutput- To ensure reliable performance in generating in-domain samples, we incorporate supervised gradients into the PPO gra- dients. Our objective is to minimize the following loss function:
Coutput
log(p(s:|8<c:: Pouspua, Y)).
(3)
where K and Y represent the keywords of the input text and the label, respectively.
3.2 Reinforcement Learning
Constructing a knowledge store that combines ac- curacy and diversity is essential for enabling the small-scale model to effectively retrieve relevant information. However, traditional supervised learn ing methods are unable to optimize these leamn ing objectives due to the lack of per-token differ- entiability. Therefore, we employ reinforcement learning improve the generation quality. This approach involves guarantceing that the generated texts align with the labels, thereby ensuring the
LLLM Eop + Ly + 3 Lates
(6)
where C, and L,, actor loss and critic loss of the PPO algorithm respectively, Latt represents Limput and foutput for P Pinput and Poutput respectively, and 3 is the hyperparameter which controls the strength supervised gradients.

---

3.3 Knowledge Store
Dataset
ITrainl
IDevl
Test
Metrics
We employ the fine-tuned LLM to generate sam- ples and construct knowledge store for each task. Specifically, for each sample the original dataset D, we utilize the keywords extracted by the Rake algorithm and labels as inputs for the model. Each sample is used m times, and in each iteration, n samples are generated. To ensure distinctiveness among the generated samples, we adopt p-sampling (Holtzman et al., 2020), which constrains tokens to the smallest possible set whose cumulative proba- bility exceeds the probability parameter p. The set of samples generated from keywords denoted as Dr. while the set generated from labels is denoted as Do- The quantity of both Dj and Do is mn D), where DD represents the quantity of the original dataset D.
BoolQ
9427
3270
3245
acc.
CB
250
56
250
acc.
COPA
400
100
500
acc.
RTE
2490
277
3000
acc.
WiC
5428
638
1400
acc.
Cola
8551
1043
1063
mcc.
Table I: The data statistics and metrics of all tasks.
where N repre- sents the batch size and & denotes the cosine simi- larity. Similarly, we can obtain the similarity score list S from the small-scale model. These similar- ity scores are then normalized a listwise way to obtain the relevance distributions:
(a)) (c)))
To further enhance the diversity of generated samples, we utilize the label of each sample in Dj as the model input to generate a sample each time, resulting in the sample set Dio- Similarly, we use the keyword of each example in Do as the model input to generate sample each time, result- ing in the sample set Do1. The final knowledge store is composed of these sample sets, namely DUD UD U Dio U Doi, which theoretically yields total of (1 + 4m samples. However, there a possibility of duplicates the generated samples, so we remove them.
ex (2,a))/
3"( 2z) Σ MEELLANAARE (mnma)/m
(7)
e (Mait))/
8 (I,, = ExE[I,N]ARES (1um))/T22
(8)
where 71 and 72 are temperature hyperparameters to smooth the distributions.
Our objective is to enable the small-scale model to learn how to retrieve relevant information from the knowledge store. To achieve this, we mini- mize the KL-divergence between the two relevance distributions $ = {5" (#;, ())] and S {55 (a;, (e))) as the learning ob- ject for the small-scale model:
As illustrated in Figure I, the key for each sam- ple in the knowledge store is the sentence embed- ding produced by the small-scale model. These embeddings enable the model retrieve relevant information from the knowledge store. The value of each sample is the probability distribution pro- duced by the reward model (M()) for each task.
N
SS log
Lsmall Σ
1o
S8
(9)
ST
During the inference of the small-scale model, we retrieve the most relevant samples from the knowledge store for each test sample x, denoted as {a; [= To determine the weight of each re- trieved sample z4, we calculate the similarity score between the test sample and the retrieved sample 2g. The final prediction logit score for each class of the small-scale model is obtained follows:
3.4 Retrieval-based Knowledge Transfer
Due to the limited number of parameters in the small-scale model, is not feasible directly memorize all the knowledge extracted from the LLM. Therefore, we do not train the small-scale model directly with the generated samples. Instead, we leverage the knowledge in retrieval manner. Specifically, we utilize the reward model RM()) to provide the sentence representation f (xx) for each generated sample I;, which then used to calculate cosine similarity scores with other gen- erated samples in the mini-batch. The similar- ity score list obtained from the reward model can be denoted as S. = {s" (1),,,))))) jE(LLN
0(I Σ 1 n..)
Pe
(10)
where e specific class and p is the confidence score of the label e predicted by the reward model RM (). The final prediction of the scall-scalc model is the class with the largest logit score.

---

Model
#Params
COPA
RTE
BoolQ
CoLA
Avg.
5a (Teacher)
3B
75.71
98.20
92.00
92.06
88.99
71.63
86.43
BERT, (Student)
7.2M
59.72
78.57
55,00
61.37
68.81
23.35
57.80
+ Vanilla KD (Hinton et al., 2015)
7.2M
60.66
78.57
56.00
62.09
68.72
27.14
58.86
+ Vanilla KD (w. TA) (Hinton al., 2015)
7.2M
61.13
82.14
58.00
61.37
69.33
26.12
59.68
+ MSGKD (w. TA) (Liu et al., 2022a)
7.2M
59.72
83.93
62.45
68.13
29.39
+ AD-KD (w. TA) (Wu et al., 2023)
7.2M
62.85
83.93
61.00
62.82
69.60
31.89
62.02
+ RetriT-KKD
7.2M
63.79
89.29
63.00
64,26
71.10
40.94
65,40
Retrik--Rctrieall
7.2M
66.61
94.64
66.00
66.43
74.62
47.87
69.36
BERT, (Student)
13.5M
62.85
85.71
60.00
65.34
70.86
38.69
63.91
+ Vanilla KD (Hinton et al., 2015)
13.5M
62.70
85.71
62.00
66.43
70.73
39.16
64.46
+ Vanilla KD (w. TA) (Hinton et al., 2015)
13.5M
63.79
87.50
60.00
66.06
70.98
40.81
64,86
MSGKD (w. TA) (Liu al., 2022a)
13.5M
62.38
87.50
63.00
65.34
71.13
40.78
65.02
+ AD-KD (w. TAA (Wu et al., 2023)
13.5M
64.89
87.50
64.00
66.43
72.81
41.77
66.23
+ -RetriKT-KD RetrikT-RRttieval
13.5M
68.65
92.86
72.00
74.01
77.22
58.16
73.82
13.5M
69.44
94.64
74.00
75.45
78.75
60.77
75.51
Table Main experimental results (%) on six low-resource tasks. We also report the quantity parameters of each PLM (without embeddings). We re-implement all baseline models based the released code provided by AD-KD (Wu et al., 2023), and incorporate BERT hase as the TA model (w. TA). There are two versions of RetriKT,, one of which is trained by Vanilla KD by generated samples (RetriKT-KD), and the other is trained by retrieval learning object ReetiiKTRRetriewa The best results of each backbone are shown in bold. Results are statistically significant with respect all baselines on each student model (all p-value 0.005).
4 Experimental Setup
for datasets COPA and CB, and batch size c [16, 32, 64) for other datasets. The training epoch is set 40 for CB and COPA, 20 for RTE and 10 for other datasets. We present the results on the validation set obtained from the best checkpoint during training.
4.1 Datasets
We assess the performance of our method on mul- tiple datasets from the SuperGLUE beachmark (Wang al., 2019a) and GLUE benchmark (Wang al., 2019b), including BoolQ (Clark al., 2019), CB (De Marneffe al., 2019), COPA (Roemmele al., 2011) 1),1 RTE, WiC (Pilehvar and Camacho- Collados, 2019), CoLA (Warstadt et al., 2019). These datasets pose a challenge for small-scale models as they are considered difficult and low- resource, with training samples numbering fewer than 10K (details in Table 1).
4.3 Implementation
We implement all experiments with the deep learn- ing framework PyTorch based on PromDA (Wang et al., 2022) and trl library (von Werra et al., 2020) on up to eight NVIDIA Tesla A100 GPUs (80GB memory). We build the LLM based on T5g1 with 3B parameters (Raffel et al., 2020), and translate each dataset into single-sentence format accord- ing to the template provided by T5. The pre- processed example of each dataset is shown in Ap- pendix We utilize two small-scale BERT models released by Ture et al. (2019), one with 4 Trans- former layers, 512 hidden neurons and attention heads, and the other with 2 Transformer layers, 512 hidden neurons and 8 attention heads.
4.2 Baselines
We compare our methods with Vanilla KD (Hinton et al., 2015) and recent strong KD methods includ- ing MSGKD (Liu et al., 2022a) and AD-KD (Wu et al., 2023), which are re-implemented based on the released code provided by AD-KD. We fine- tune EncT5x1 (Liu al., 2021a) the teacher model, and train BERT base using the Vanilla KD method as the assistant model (TA) for all base- lines. We carry out gric-search learning rate E { (le-5, 2e-5, 3e-5, 4e-5), batch size E {4, 8, 16)
First, we tune the soft prompts Pinput and P Poutput by supervised learning as the reference and initial models for PPO algorithm. The prompt length is set to for datasets. The learning

---

rate and batch size is set le-3 and 64, respec- tively. The training step is and CB, 20K for RTE, and 40K for other datasets. We fine-tune xl x as the reward model for each dataset, which is also the teacher model for all base- line models for fair comparison. Then we train the soft prompts through combination of super- vised learning and reinforcement learning by the loss function defined in (q.( We set and 0.2 and respectively. Generate number n and probability for top-p sampling are always set to 5 and 0.9 respectively, while sample time m set to 64 for COPA and CB, 16 for RTE, and for other datasets. Finally, we train the small-scale BERT models with a grid search of learning rate
PLM
Model
CB
COPA
CoLA
10K for COPA
RetriKT-K
89.29
63.00
40.94
w/o RL
87.50
61.00
36.38
w/o Raceuracy
87.50
60.00
40.53
w/o Rdiversity
85.71
62.00
38.22
w/o BP
87.50
60.00
39.13
BERT
RetriKT-RRtrieval
94.64
66.00
47.87
w/o RL
89.29
62.00
41.57
w/o Raceuracy
91.07
62.00
45.32
w/o Rdiversity
87.50
61.00
43.24
BP
89.29
62.00
44.98
RetriKT-KD
92.86
72.00
58.16
w/o RL
89.29
67.00
53.18
w/o Raceuracy
91.07
65.00
57.78
w/o Raiversity
89.29
69.00
57.01
w/o BP
91.07
66.00
57.27
BERT,
RetriKT-Retrieee
94.64
74.00
60.77
с (2e-5, 3e-5), temperate combinations (11, T2) E ((0.2, 0.1), (0.1, 0.05)). The batch size is set to 128. There are two versions of RetriKT, one of which is trained by Vanilla KD by generated sam- ples (RetriKT-KD), and the other is trained by re- More
w/o RL
92.86
69.00
56,45
w/o Raceuracy
92.86
69.00
59.94
w/o Rdivorsity
91.07
70.00
58.25
w/o BP
92.86
69.00
57.82
trieval learning object (RetriKT-Retrieval training details for PPO algorithm can be found in Appendix A
Table 3: Ablation studies of different components uti- lizing BERT, and BERT, as the small-scale models on datasets CB, COPA and CoLA.
5 Experimental Results and Analysis
5.1 Main Results
As presented Table 2, it is evident that both RetriKT-KD and RetrikT-Retrievll outperform pre- vious KD methods across all backbones, which demonstrates the effectiveness of our approach. For instance, compared to the previous state-of-the-arn method AD-KI KD, RetriKT- significant improvements: 7.34% on BERT, and 9.28% on BERT, 4. Notably, RetriKT-Retrieval con- sistently outperforms RetriKT-KD, especially on BERT,, indicating that retrieving relevant informa- tion from the knowledge store is more suitable for small-scale models rather than attempting to mem- orize all the knowledge. Furthermore, we observe comparable performance between Vanilla KD and Vanilla KD (W, suggesting limitations in using the TA model for knowledge transfer. more ef- ficient approach for transferring knowledge from the LLM to the small-scale model is to prompt the LLM to generate in-domain knowledge, which can be retrieved by the small-scale model as relevant information for effective inference.
duet series of ablation studies. We remove re- inforcement training (w/o RL), accuracy reward Raccuracy, diversity reward Rdiversity. and length penalty BP, and evaluate the results CB, COPA, and CoLA, shown in Table 3. Several key obser- vations can be drawn from the experimental results.
demonstrates
Firstly, is evident that the model performance both RetriKT-KD and RetriKT-Retriew declines without RL or any component of RL. This high- lights the effectiveness of our reward function de- sign and RL, which enhance the generation quality of the LLM. Secondly, when any component of RL is removed, the model performance on CoLA shows relatively smaller decreases compared CB and COPA. This suggests that the generation quality of the LLM has more pronounced im- pact on smaller datasets, while larger datasets ex- hibit certain level of robustness. However, RL consistently improves the generation quality of the LLM. thereby enhancing overall model perfor- mance. Finally, is worth mentioning that RetriKT- Retrieval consistently outperforms RetriKT-KD across almost all settings, further validating the effectiveness of retrieval-based knowledge transfer for small-scale models.
5.2 Ablation Study
To investigate the impact of different components reinforcement learning in our approach, we con-

---

Metric
Model
WiC
CB
COPA
RTE
BoolQ
CoLA
RetriKT
0.912
0.948
0.970
0.923
0.884
0.959
Self-Bleu
RetriKT w/o RL
0.945
0.968
0.983
0.937
0.939
0.973
RetriKT
0.734
0.781
0.786
0.781
0.760
0.579
Cross Entropy
RetriKT w/o RI.
0.758
0.896
0.796
0.794
0.779
0.618
Table 4: Diversity and accuracy analysis for the generated knowledge. Self-Bleu reflects the diversity of knowledge, while cross entropy reflects the accuracy of knowledge. In both metrics, smaller values indicate better performance.
proach. Furthermore, as continues to increase, observe a corresponding increase in the perfor- mance of both RetriKT-KD and RetriKT-Retrreval, reaching a certain threshold. This indicates that increasing the number of generated samples can enhance the diversity of knowledge. However, due to the limited number of keywords extracted from the original dataset, the diversity of knowledge plateaus once the generation threshold is reached. Consequently, the model performance does not show further improvement beyond this point.
RenKiTKD
Re
13
58
60
39
$2
Sample Time
Sumple Time
(a) BERT,
(h) BERT,
Figure 2: Effect of the sample times m. Results are the model performance on the COPA dataset using BERT, and BERT, The Finetune method trained by original dataset so it is independent of
Effect of the Model Size We conduct an in- vestigation into the impact of model size on per- formance using two datasets, COPA and CoLA. Our experimental results shown in Figure 3 reveal that when employing a smaller model, RetriKT- Retrieval consistently outperforms RetriKT-KD, showcasing the superiority of retrieval-based knowledge transfer for small-scale models. Ad- ditionally, we observe that both RetriKT-Retrieval and RetriKT-KD consistently outperform the train- ing of the small-scale model solely fine-tuning by the original dataset. This observation further em- phasizes the effectiveness of knowledge extraction from the LLM and its application in enhancing model performance.
67
BeuiKT-D
Retio
Stcnas
39
20
Pastace
10
2-18 3:93
2513
1576
Model Size
(a) COPA
(b) CoLA
Figure 3: Effect of the model size. x-axis is the number of layers and hidden size. Results are the model perfor- mance on the COPA and CoLA datasets.
Accuracy and Diversity In this section, we ex- amine whether the LLM trained through reinforce- ment learning can generate more accurate and di- verse knowledge. To evaluate the diversity of knowledge, we employ the Self-Bleu metric on the generated samples, while the accuracy of knowl- edge is measured using the cross-entropy between the probability distribution predicted by the reward model and the label generated by the LLM. In both metrics, smaller values indicate better performance. As shown in Table 4, the results demonstrate that training the LLM through reinforcement learning leads to the generation more accurate and diverse knowledge across all datasets. This outcome high- lights the effectiveness our reward function de-
5.3 Detailed Analysis
Effect of the Sample Times We conduct experi- ments on the COPA dataset investigate the im- pact of the sample time m on model performance. As shown in Figure 2, with a small value of m, the performance of both RetriKT-KD and Re Retrieval is comparable. However, as m increases, we observe a significant improvement in the perfor- mance of RetriKT Retrieval compared to RetriKT- KD. This finding suggests that small-scale models, constrained by their limited parameters, are un- able to effectively memorize such a large amount of knowledge. Therefore, retrieval-based knowl- edge transfer proves to be a more favorable ap-

---

sign. Additionally, we observe that larger datasets tend to yield samples with smaller Self-Bleu met- rics. We conjecture that this phenomenon is re- sult of the larger dataset's ability to extract a wider range of diverse keywords, thereby enabling the generation of more varied knowledge.
References
Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama 7werdling. 2020. Do not have enough data? deep learning to the rescue! In The Thirty-Fourie Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Secund Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7383- 7390. AAAI Press.
6 Conclusion
Our study tackles the task of compressing LLMs and introduces a pioneering compression paradigm called Retrieval-based Knowledge Transfer. This approach efficiently transfers the knowledge of LLMs to small-scale models by creating a knowl- edge store, enabling the small model to retrieve per- tinent information during inference. Through ex- tensive experiments conducted on commonly used benchmarks, we demonstrate that our framework substantially improves the performance of small- scale models by leveraging the knowledge con- tained within L.Mss. In future research, we plan to investigate the application and performance of our proposed method on even larger language models, such T5-11B.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2017. An actor-critic algorithm for sequence prediction. In 5th Inter- national Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Con- ference Track Proceedings. OpenReview.net.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage. Zac Hatfield-Dodds Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. 2022. Train- ing a helpful and harmless assistant with rein- forcement learning from human feedback. CoRR, 0hs/2204.05886
Limitations
In this section, we discuss the limitations of our work as follows. Firstly, due to limited computa- tional resources, we did not attempt experiments with larger models such as 15-11B. Furthermore, resource constraints constrained our grid search for hyperparameters within limited range, which potentially leaves room for enhancing the metrics showcased in this paper. Secondly, our proposed method requires the establishment of knowledge store. Compared to other model compression meth- ods, our approach introduces additional storage space and incurs slightly additional time for re- trieval
Christopher Clark, Kenton Lee, Ming-Was Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACI-III. 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2924 2936. Associa- tion for Computational Linguistics.
Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Zihao Wu. Lin Zhao, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, Hongmin Cai, Quanzheng Li, Dinggang Shen, Tianming Liu, and Xiang Li. 2023. Chataug: Leveraging chatgpt for text data augmentation. CORR, abs/2302.1300)
Acknowledgements
This work is supported by National Key R&D Pro- gram of China (No. 2021 YFC3340303) and Na- tional Natural Science Foundation of China (NSFC Grant No. 62 62122089). Jingang Wang is funded by Beijing Nova Program (Grant NO. 20220484098). We sincerely thank all reviewers for their valuable comments and suggestions, which are crucial for improving our work. We would also like to ac- knowledge Angela Li for her contributions in cre- ating the figures used in this work.
Carre-Catherine De Marneffe. Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmentbank: Inves- tigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107- 124.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for

---

Computational Linguisties: Human Language Tech- nologies, NAACL-HLI 2019. Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. 2018. Can neural machine translation be improved with user feedback? In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HILT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Vol- ume (Industry Papers), pages 92-105. Association for Computational Linguistics.
Bosheng Ding, Linlin Liu, Lidong Bing, Canasai Kru- engkrai, Thien Hai Nguyen, Shafiq R. Jory, Luo Si, and Chunyan Miao. 2020. DAGA: data augmentation with a generation approach for low-resource tagging tasks. CoRR, abs/201 1.01599.
Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. Data augmentation using pre-trained trans- former models. CoRR, abs/2003.02245
Mitchell A. Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing BERT: studying the effects of weight pruning on transfer learning. In Proceedings of the 5th Workshop on Representation Learning for NLP,, RepLANL.P@ ACI. 2020, Online, July 9, 2020, pages 143-155, Association for Computational Lin- guistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Braden Hancock, Antoine Bordes, Pierre- Emmanuel Mazaré, and Jason Weston. 2019. Learning from dialogue after deployment: Feed yourself, chatbot! In Proceedings of the 57th Conference of the Associ- ation for Computational Linguistics, ACI. 2019, Flo- rence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pages 3667-36668 Association for Computa- tional Linguisties.
Carolin Lawrence and Stefan Riezler. 2018. Improving neural semantic parser by counterfactual learning from human bandit feedback. In Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 1820 1830. Association for Computational Linguis- tics.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs//503.02555 12533..
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengeheng He, Tuo Zhao, and Weizhu Chen. 202 Super tickets in pre-trained language models: From model compression to im- proving generalization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing, ACL/UCNLP 2021, ( (Volume 1: Long Papers), Virtual Event, Au- gust 1-6, 2021, pages 6524-6538. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net.
Natasha Jaques, Judy Hanwen Shen, Asma Ghande- harioun. Craig Ferguson, Ágata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard. 2020. Human-centric dialog training via offline reinforce- ment learning. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 3985-4003. Association for Computa- tional Linguistics.
Kevin J. Liang. Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and Lawrence Carin. 2021b. Mixkd: Towards efficient distillation of large- scale language models. In 9th International Confer- ence on Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021. OpenReview.net.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.
Tinybert: Distilling BERT for natural language un- derstanding. Findings of the Association for Com- putational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Find- ings of ACL, pages 4163-4174, Association for Com- putational Linguistics.
Chang Liu, Chongyang Tao, Jiazhan Feng, and Dongyan Zhao. 2022a. Multi Multi- granulartyyl structural knowl- edge distillation for language model compression. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages 1001-1011. I. Association for Com- putational Linguistics.
Samuel Kiegeland and Julia Kreutzer. 2021. Revisiting the weaknesses of reinforcement learning for neu- ral machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 1673-1681. Association for Computational Linguistics.
Frederick Liu, Siamak Shakeri, Hongkun Yu, and Jing Li. 2021a. Enct5; Fine-tuning T5 encoder for non- autoregressive tasks. CoRR, 1bs/2110.08426.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022b. P-tuning:

---

Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 61-68. As- sociation for Computational Linguistics.
projection for knowledge distillation. In Thirty-Fifhh AAAI Conference on Artificial Intelligence AAAI 2021, Chirty-Third Conference on Innovative Ap- plications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Ar- tificial Intelligence, EAAI 2021, Virtual Event, Febru- ary 2-9, 2021, pages 13657-13665. AAAT Press.
Xuebo Liu, Longyue Wang, Derek F. Wong, Liang Ding, Lidia S. Chao, Shuming Shi, and Zhaopeng Tu. 2021b. On the complementarity between pre-training and tack-trrnslation for neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2900-2907, Punta Cana, Dominican Republic. Association for Compu- tational L. Linguisties.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive sum- marization. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net.
Mohammad Taher Pilehvar and José Camacho-Collador 2019. Wic: the word-in-contex cn-context dataset for evaluat- ing context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HILT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 1267- 1273. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: A robustly optimized BERT pretraining approach. CoRR, 100/1907.1692.
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. 2020. Improved knowledge distilla- tion via teacher assistant. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innvaatve Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial In- telligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 5 5191-5198. AAAI Press.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res,, 21: 11:1110:1140:0
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa. Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2022. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimiza- tion. CoRR, 10.0221
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models follow instruc- tions with human feedback. In NeurIPS.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S. Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In Logical Formalizations of Commonsense Reasoning, Papers from the 2011 AAAI Spring Sym- posium, Technical Report SS- 11-6, Stanford, Cali- fornia, USA, March 21 21-23, 2011. AAAI.
Richard Yuanzhe Pang and He He. 2021. Text genera- tion by learning from demonstrations. In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, July 6-12, 2002, Philadelphia. PA, USA, pages 311-318. ACL.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Ka- hou, Antoine Chassang. Carlo Gatta, and Yoshua Bengio. 2015. Fitnets: Hints for thin deep nets. In 3rd International Conference on Learning Represen- tations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Geondo Park, Gyeongman Kim, and Eunho Yang. 2021. Distilling linguistic context for language model com- pression. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Domini- can Republic. 7-11 November. 2021, pages 364-378. Association for Computational Linguistics.
Stuart Rose, Dave Engel, Nick Cramer, and Wendy Cowley. 2010. Automatic keyword extraction from individual documents. Text mining: applications and theory, 1:1-20,
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01 108.
Peyman Passban, Yimeng Wu, Mehdi Rezagholizadeh, and Qun Liu, 2021. ALP-KD: attention based layer

---

Teven Le Scao, Angela Fan. Christopher Akiki, EI- lie Pavlick, Suzana Iiic, Daniel Hesslow, Roman Castagné, Alexandra Sasha I Luccioni, François Yvon, Matthias Gallé, Jonatham Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Am- manamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier. Samson Tan, Pedro Ortiz Suarez, Vic- tor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji. Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong. Daniel van Strien, David Ifeoluwa Adelani, and et al. 2022. BLOOM: A
Hugo Touvron, Thibaut Lavril. Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971.
Tulia Ture, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv preprial (:XXx:1908.099)
Solomon Ubani, Suleyman Olcay Polat, and Rod- ney Nielsen, 2023. Zeroshotdataaug: Generating and augmenting training data with chatgpt. CoRR, abs/2304.14334,
open-acces multilingual language model. CoRR, abs/2211 1. 1.05100.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. CoRR, abs/1 .633777
Wonchul Son, Jaemin Na, Junyong Choi, and Wonjun Hwang. 2021. Densely guided knowledge distillation using multiple teacher assistants. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, and Nathan Lam- bert. 2020. Trl: Transformer reinforcement learning. https: //gi www.ttt........e.eeeee.et 1ttr
IEEE
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stick- ier benchmark for general-purpose language under- standing systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neu- ral Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3261-3275
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe. Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. Learn- ing to summarize from human feedback. CoRR, 0os/2009.01325..
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model com- pression. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Nat- 2019,
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In 7th In- ternational Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
ural Language Processing, EMNLP- Hong Kong, China, November 3-7, 2019, pages 4322- 4331. Association for Computational Linguistics.
Siqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, and Jingjing Liu. 2020. Contrastive distil- lation on intermediate representations for language model compression. In Proceedings of the 2020 Con- ference Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 49844448-900 508. Association for Computational Linguistics.
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2021a. Minilmv2: Multi-head self- attention relation distillation for compressing pre- trained transformers. In Findingy of the Associa- tion for Computational Linguistics: ACL/JCNLP 2021. Online Event, August 1-6, 2021, volume ACL/ICNLP 2021 of Findings of ACL, pages 2140- 2151, Association for Computational Linguistics.
Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang. Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2022. Compression of generative pre- traitraine language mod- els via quantization. nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4821-4836 Association for Computational Linguistics.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang. and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task aask-agnostic compression of pre-trained transformers. In Advances in Newral Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, viriual.
Proceedings of the 60th An-

---

Yufei Wang, Can Xu, Qingfeng Sun, Huang Hu, Chongyang Tao, Xiubo Geng, and Daxin Jiang. 2022. Promda: Prompt-based data augmentation for low- resouree NLU tasks, In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4242-4255. Association for Computational Linguistics.
Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 509-521. Association for Computational Linguistics.
Li Zhou, Kevin Small, Oleg Rokhlenko, and Charles Elkan. 2017. End-to-end offiine goal-oriented di- alog policy learning via policy gradient. CoRR, abs/ 0as/1772.02833.
Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. 2021b. Towards zero-label language learning. CoRR. a0s/210..09193.
Wangchunshu Zhou and Ke Xu. 2020. Learning com- pare for better training and evaluation of open domain natural language generation models. In The Thirry- Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Shirty-Second Innovative Applica- tions of Artificial Intelligence Conference, IAAT 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9717-9724. AAAI Press.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Trans. Assoc. Compul. Linguistics, 7:625 641.
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stien- non, Ryan Lowe, Jan Leike, and Paul F. Christiano. 2021. Recursively summarizing books with human feedback. CoRR, abs/2109.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo. Weinan Zhang, Jun Wang. and Yong Yu. 2018 Texygen: A benchmarking platform for text generation models. In The st International ACM SIGIR Conference Reseurch & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pages 1097-1100, ACM.
Siyue Wu, Hongzhan Chen, Xiaojun Quan, Qifan Wang. and Rui Wang. 2023. AD-KD: attribution-driven knowledge distillation for language model compres- sion. CoRR, 0bss/2335.0010
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. Conditional BERT contex- tual augmentation. In Computational Science ICCS 2019- 19th International Conference, Faro, Portu- gal. June 12-14, 2019, Proceedings, Part IV, volume 1 11539 of Lecture Notes in Computer Science, pages 84-95. Springer.
Daniel M. Ziegler. Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Chris- tiano, and Geoffrey Irving. 2019. Fine-tuning lan- guage models from human preferences. CoRR. abs/ 1et.199999995
Yiben Yang. Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang. Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020. G-daug: Generative data augmentation for commonsense reasoning. In Findings of the As- sociation for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL. pages 1008-1025 Association for Computational Linguistics.
Changtong 7an, Liang Ding. Li Shen, Yu Cao, Weifeng Liu, and Dacheng Tao. 2022. On the complementar- for
ity between pre- pre-training and caadoomiiitializaatie resource-rich machine translation. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 5029-5033 9034- Inter- national Committee Computational Linguistics.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong. and Jie Tang. 2022. GLM- 130B: an open bilingual pre-trained model. CoRR, abs/2210.02414.
Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. 2020. Tern:rybert: Distillation-awar ultra-low bit BERT. Proceed- ings of the 2020 Conference on Empirical Methods in

---

A Training Details for PPO
Processed imput: hypothesis: Valence was helping premise: Valence the void-brain, Valence the virtuous valet. Why couldn't t the figger choose his own portion of titanic anatomy to shaft? Did he think he was helping?
In this section, present the training details of PPO in Table. The meaning of these hyperparame- ters detailed in tri library (von Werra et al., 2020) and NLPO (Ramamurthy al., 2022).
B Preprocessed Examples
Label: (entailment: 0; contradiction: 1; neutral: 23
In this section, we provide examples of our prepro- cessing for each of the data sets we consider.
B.3 CoLA
B.1 boolq
Original input:
Original input:
Sentence: John made Bill master of himself.
Question: Can you have a skunk as a pet in canada
Processed input: sentence: John made Bill master of himself.
Passage: Skunks as pets Canadian pet skunks must be purchased from USDA-certified breeder in the United States. An import permit is required from the Canadian Food Inspection Agency to bring the skunk into the country. The skunk must be spayed or neutered, and receive a microchip implant or tattoo. A vet check fee must also be paid. It is illegal to keep striped skunks as pets in Canada.
Label: (unacceptable: 0; acceptable: B.4 CОРA
Original input:
Question: effect
Premise: Political violence broke out in the nation.
Choice 1: Many citizens relocated to the capitol.
Processed input: question: Can you have a skunk as a pet in canada passage: Skunks as pets Canadian pet skunks must be purchased from a USDA-certified breeder in the United States. An import permit is required from the Canadian Food Inspection Agency to bring the skunk into the country. The skunk must be spayed or neutered, and receive microchip implant or tattoo. vet check fee must also be paid. It is illegal to keep striped skunks as pets in Canada.
Choice 2: Many citizens took refuge in other territories.
Processed input: premise: violence brake out in the nation. choice1: Many citizens relocated to the capitol. choice2: Many citizens took refuge in other territories. question: effect
Political
Label: (choice1: 0; choice2: 1}
Label: (False: True: 1}
B.5 RTE
Original input:
B.2 CB
Sentence 1: A smaller proportion of Yugoslavia's Italians were settled in Slovenia (at the 1991 national census, some 3000 inhabi of Slovenia declared themselves as ethnic Italians).
Original input:
Hypothesis: Valence was helping
Premise: Valence the void-brain, Valence the virtuous valet. Why couldn't
the figger choose his own portion of titanic anatomy to shaft? Did he think he was helping?
Sentence 2: Slovenia has inhabi
3,000

---

Hyperparameters
WiC
CB
COPA
RTE
BoolQ
CoLA
learning rate
2c-3
2e-3
2e-3
2c-3
2e-3
2c-3
batch size
128
64
64
128
128
128
min batch size
32
16
16
32
32
32
epoch
20
100
100
50
10
20
ppo epoch
4
4
4
4
4
sample number
4
4
4
4
initial kl coeff
0.001
0.001
0.001
0.001
0.001
0.001
target kl
6
6
6
6
6
value function coeff
0.5
0.5
0.5
0.5
0.5
0.5
clip ratio 0.2
0.2
0.2
0.2
0.2
0.2
discount factor
0.99
0.99
0.99
0.99
0.99
0.99
gae lambda
0.95
0.95
0.95
0.95
0.95
0.95
Table 5: Hyperparameters for PPO
Processed input: sentence1:
smaller proportion of Yugoslavia's Italians were settled Slovenia (at the 1991 national census, some 3000 inhabitants of Slovenia declared themselves as ethnic Italians). sentence2: Slovenia has 3, 000 inhabitants.
WIC
COPA
CoLA
4.61
4.36
4.49
Table 6: The faithfulness score of the generated texts for datasets WiC, COPA and CoLA.
final faithfulness score. The results the faithful- ness evaluation are shown in Table 6, showing that the faithfulness of generated texts for each dataset is quite high.
Label: (entailment: 0; not_entailment: 1}
B.6 WiC Original input:
POS: N
Sentence 1: was the deliberation of his act that was insulting
Sentence 2: The deliberations of the jury
Word: deliberation
Processed input: sentence1: It was
the *deliberation* of his act that was insul ting. sentence2: The *deliberations* of the jury. word: deliberation
Label: (False: 0; True: 1}
Faithfulness of the Generated Knowledge
To further validate the faithfulness of the generated knowledge, gathered five volunteers to assess the faithfulness of the generated texts for datasets WiC, COPA, and CoLA. Each text will be given rating from to We randomly sampled 100 texts from each dataset and calculate the average as the


Reviews:
[{"review_id": "jvH79EBdPA", "reviewer": "Reviewer_MYsE", "summary": "", "questions": "- Missing ethics section in the paper\n- \"the massive size of these models poses huge challenges for their deployment in real-world applications.\" - because this line has been used to motivate the work, I am adding this comment here:\n\nthe ability to generate more examples for a low-resource end-task is useful, however the selection of low-resource tasks should also be done keeping in mind the usefulness of training examples from that task for other tasks (aka usefulness of the task for OOD generalization)\n\nIf examples from a low-resource task do not contribute to OOD generalization, having such an elaborate knowledge transfer setup for a couple of hundred dev/test set examples seems excessive. If an end-task has just a couple of hundred examples, it's also less likely that the end-user queries will be a good distribution match with the examples from that end-task and highly specialized small LMs on such tasks will be redundant.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your recognition of our work and the valuable feedback. We will provide a detailed explanation based on your concerns.\n\n#### **Q1: Regarding the complexity of the method.**\n\n**A1:** Thank you for the great question. It's true that our approach involves complex computations during the training phase to ensure effective knowledge transfer. However, once our model is adequately trained, the inference stage becomes quite straightforward, involving only sentence encoding and retrieval. Particularly for tasks that pose challenges for smaller models, like those found in the SuperGLUE benchmark, we consider our approach to be highly valuable. We fully concur with the reviewer's observation that simplifying the training process of our approach represents a key avenue for future research. We intend to expand on this aspect with more in-depth discussion in the revision.\n\n#### **Q2: Regarding missing ethics section in the paper.**\n\n**A2:** The ethics section is not mandatory, and we think there are no ethical issues in our paper. Please let us know if there are any ethics we should mention in this work, and we are happy to discuss them in the ethics section.\n\n#### **Q3: Regarding the usefulness of the task for OOD generalization.**\n\n**A3:** Thanks for the excellent comment. We completely agree with the reviewer's perspective that the choice of low-resource tasks should be approached with careful consideration, taking into account the potential utility of training examples from such tasks for other tasks. Our motivation hinges on the extensive pre-training of LLMs on massive datasets, which equips them with a wealth of knowledge. When combined with our PPO training methodology, this synergy leads to the generation of diverse text outputs from the LLM. Consequently, our approach has the potential to enhance out-of-distribution generalization, making it particularly effective for addressing low-resource tasks.\n\nThanks again for your review! Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- strong performance against recent distillation baselines from Liu et al. (MSG-KD) and Wu et al. (AD-KD)\n\n- usefulness of retrieval-based learning objective against vanilla distillation on generated examples (this effect is more pronounced for smaller models) (Table 1 results: RetriKT-KD vs RetriKT-retrieval)\n\n- ablations justify the inclusion of both the PPO setup and the retrieval component of the model", "reasons_to_reject": "- the method seems to contain way too many intermediate steps and has a lot of working parts to setup the entire retrieval mechanism with a small LM (complexity of the method is a downside against ease of adoption)\n- but I agree this is not a super strong reason in itself and should be used more as a motivation for future work", "questions_to_authors": "- Missing ethics section in the paper\n- \"the massive size of these models poses huge challenges for their deployment in real-world applications.\" - because this line has been used to motivate the work, I am adding this comment here:\n\nthe ability to generate more examples for a low-resource end-task is useful, however the selection of low-resource tasks should also be done keeping in mind the usefulness of training examples from that task for other tasks (aka usefulness of the task for OOD generalization)\n\nIf examples from a low-resource task do not contribute to OOD generalization, having such an elaborate knowledge transfer setup for a couple of hundred dev/test set examples seems excessive. If an end-task has just a couple of hundred examples, it's also less likely that the end-user queries will be a good distribution match with the examples from that end-task and highly specialized small LMs on such tasks will be redundant.", "paper_topic_and_main_contributions": "The paper introduce a framework called Retrieval Based Knowledge Transfer for compressing LLMs for low-resource end-tasks. The framework has 4 parts:\n\n- it trains soft-prompts for individual tasks in a supervised setting, using the method from P-tuning v2 paper, and for diversity in text generation, combines it with the idea from PromDA which results in training soft-prompts for generation from two views of end-task examples (keywords from the input text and output labels of the example)\n\n- then it adds the PPO algorithm for tuning soft-prompts for generation diversity and label accuracy. Generations are evaluated using a reward model trained on task-specific data, rewarded for diverse generations measured via self-bleu metric and penalized for generation of short examples\n\n- the datastore is constructed from 5 types of examples:\n  - from the original dataset\n  - examples/labels generated by conditioning prompts on keywords from examples of the original dataset\n  - examples/labels generated by conditioning prompts on labels of the original dataset\n  - examples/labels generated from soft-prompts tuned on labels of original examples and labels of generated examples in step 2\n  - examples/labels generated from soft-prompts tuned on keyword of original examples and keywords of generated examples in step 3\n  - the datastore is created with keys from the sentence representations of generated examples `and values from the output of the reward model for those examples\n\n- the small LM is now trained by generating similarity scores for pairs of retrieved examples from the datastore and aligning them with scores generated for the same set of examples by the reward model", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9Rc6NAhesS", "reviewer": "Reviewer_CJV8", "summary": "", "questions": "1.\tI don’t quite understand the paragraph from line 48 to line 50. You mention knowledge distillation as a compression method, then claim that such “compression methods are not suitable for scenarios requiring high compression ratios, such as knowledge distillation…”. Besides, KD does not necessarily require high compression ratio, you can in theory use any ratio you want. \n2.\tI don’t quite understand the paragraph from line 259 to line 262: “unable to optimize these learning objectives due to lack of per-token differentiability”.\n3.\tHow do we assess the impact of length penalty on the training? Penalizing length only does not mean that the model will generate sensical sentences.\n4.\tParagraph 303->305: What s the LM you are talking about? The classifier or the large frozen model? Is there any equation representing the global loss function (including everything)?\n5.\tHow is the whole training performed? Jointly? Sequentially?\n6.\tLine 407: are you using T5 as a teacher and Bert as student? These are two different architectures. Maybe I am not aware of it, but just out of curiosity, has this been done before in the literature?\n7.\tLine 446: you mean eq. 4?\n8.\tIf I understand well, the small scale is only used to generate a representation that is close to those previously calculated by the reward model.  The decision only relies on the labels previously generated by the RM. Say the test sample and the retrieved sentences are by no means related to each other. That means the final answer is likely going to be based on answers to different questions. Correct?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your careful review of our paper and the detailed feedback, which are crucial for improving our work. We discuss your raised points as follows:\n\n#### **Q1: Regarding how the generator is trained, PPO’s agent and the initialization of small models.**\n\n**A1:** Thank you for the questions. We would like to clarify these questions here.\n\n- For the training of the generator, we initialize it with the T5-3B model and augment it with randomly initialized soft prompts, as detailed in section 4.3. During the entire training process, we exclusively update the soft prompts while maintaining the fixed parameters of the T5-3B model. Our training procedure unfolds in two main phases. Initially, we employ supervised learning using appropriate datasets to train the generator, as elaborated in section 3.1. Subsequently, we fine-tune the generator using the reinforcement learning algorithm PPO, as outlined in section 3.2.\n- The PPO's agent is our generator. We train the generator by the reinforcement learning algorithm PPO, enabling it to generate more accurate and diverse knowledge.\n- Our small models are initialized from the BERT models released by Turc et al. (as mentioned in lines 429-433).\n\nWe hope these additional information answers your questions. We will supplement these details in our revision.\n\n#### **Q2: Regarding the complexity and engineering of our methods.**\n\n**A2:** Thank you for the insightful question. Our approach indeed involves multiple steps during the training phase to facilitate effective knowledge transfer. However, once our model is trained, the inference stage becomes relatively straightforward, involving only sentence encoding followed by retrieval. Particularly for practical applications that pose challenges for smaller models, we consider our approach to be highly valuable and necessary.\n\n#### **Q3: Regarding the comparison with baselines, such as simple finetuning.**\n\n**A3:** Thanks for the suggestion. In fact, we have compared our approach with simple finetuning baselines in Table 2. The rows corresponding to BERT$_2$ (Student) and BERT$_4$ (Student) are the results of simple finetuning. We’ll make this more clear during revision.\n\n#### **Q4: Regarding more details of PPO algorithm.**\n\n**A4:** Since we directly apply the PPO algorithm without any refinement, we do not include its technical details in our paper. Thank you for the great suggestion. We will supplement the technical details of the PPO algorithm in the Appendix.\n\n#### **Q5: Regarding the architecture difference with PromDA.**\n\n**A5:** Thanks for the question. We would like to provide some clarification here. Our approach exhibits significant distinctions from PromDA in several key aspects:\n\n- Motivation: Our approach explores a novel compression paradigm, focusing on efficiently transferring knowledge from Large Language Models (LLMs) to smaller models. In contrast, PromDA's primary aim is to enhance few-shot tasks through data augmentation.\n- Model Architecture: We introduce the PPO algorithm to enhance the accuracy and diversity of generated knowledge. Additionally, we facilitate knowledge transfer to the smaller model through retrieval. The main commonality with PromDA is the utilization of the p-tuning v2 method (Liu et al.) for fine-tuning the generator.\n\nWe hope this clarifies the distinctions between our approach and PromDA.\n\n#### **Q6: Regarding limitation to text classification and models with sentence representations.**\n\n**A6:** Thanks for pointing it out. Currently, our method is only applicable to text classification tasks, just like the baseline methods we compared with. In the future, we will expand our method to more types of tasks (such as token classification and generation). We will provide these discussions in the Limitations section as suggested.\n\n#### **Q7: Regarding the claim that \"compression methods are not suitable for scenarios requiring high compression ratios, such as knowledge distillation**\n\n**A7:** Sorry for the confusion. We understand and agree that KD does not necessarily require a high compression ratio. The intention behind this paragraph is to emphasize that prevailing model compression techniques, including knowledge distillation, pruning, and quantization, might not be apt for situations requiring substantial compression ratios. Consequently, our approach is introduced as a solution to address this specific challenge.\n\n#### **Q8: Regrading unable to optimize these learning objectives due to lack of per-token differentiability**\n\n**A8:** The evaluation metrics of accuracy and diversity operate at the sentence level, which means they lack per-token differentiability. In this context, we'd like to elucidate why reinforcement learning is employed instead of supervised learning. We will provide additional clarification on this point in our revised work.\n\n#### **Q9: Regarding assessing the impact of length penalty on the training.**\n\n**A9:** Thank you for your question. As mentioned in section 3.2, we employ the length penalty as a mechanism to discourage the generation of excessively simple patterns by the generator. The ablation study results presented in Table 3 demonstrate that the inclusion of a length penalty indeed enhances the model's performance. It's important to note that while the length penalty doesn't guarantee that the model generates sensical sentences, its absence may lead to the generation of overly simplistic patterns. This underscores the importance and necessity of incorporating the length penalty into our approach.\n\n#### **Q10: Regarding the meaning of \"LM\" in the paragraph from line 303 to line 305.**\n\n**A10:** Sorry for the inconvenience. The LM loss represents the learning object in section 3.1 to train the Language Model (the generator). We will make this more clear during vision. We’ll also supplement a global loss function as suggested.\n\n#### **Q11: Regarding the whole training process.**\n\n**A11:** We train our models sequentially. **First**, we train the generator by supervised learning so that the generator can generate in-domain texts. **Subsequently**, we employ the reinforcement learning algorithm PPO to further tune the generator to make the generated texts more accurate and diverse. **Afterward**, a knowledge store will be constructed using the generator, and **finally**, the small model will be trained.\n\n#### **Q12: Regarding using T5 as a teacher and BERT as student models.**\n\n**A12:** Previous studies, including MINIDISC (Zhang et al.) and AutoDisc (Zhang et al.), have also employed a configuration where T5 serves as the teacher model and BERT as the student model. This configuration is quite common in the field. In our work, we adopt T5 as the teacher model for several reasons. T5 not only functions as the teacher for knowledge distillation, enabling comparisons with baseline methods, but also serves as a generative model for knowledge generation. The choice of BERT as the student model, as opposed to T5, is motivated by the availability of pre-trained BERT models of various sizes, readily accessible from the work of Turc et al. This allows for easy initialization of the student model with the desired architecture.\n\n#### **Q13: Regarding Line 446**\n\n**A13:** Yes, we meant Eq. 4 (instead of Eq. 3). Thank you for pointing it out, and we will revise accordingly.\n\n#### **Q14: Regarding the decision only relying on the labels previously predicted by the RM.**\n\n**A14:** Your understanding is indeed accurate. However, we’d like to reiterate on three key points:\n- **Diversity of the Knowledge Store:** We ensure diversity within the knowledge store through reinforcement learning and sampled generation. This approach helps ensure that the knowledge store covers a wide range of information, making it, at the very least, relevant, if not entirely specific, to the test examples.\n- **Limited Generalization of the Small Model:** Small models often exhibit limited generalization ability, especially when tasked with inferring samples that extend beyond the training set. This challenge is particularly pronounced for complex tasks.\n- **Enhancing the Small Model through Retrieval:** Enhancing the small model through retrieval serves to mitigate the training difficulty inherent to smaller models. This approach empowers the small model to effectively leverage the knowledge present within the Large Language Model (LLM). Consequently, retrieval-based knowledge transfer consistently outperforms direct knowledge distillation.\"\n\nThese three points collectively underscore the advantages of our approach over direct knowledge distillation.\n\nThank you again for your detailed review and constructive feedback. Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tAuthors present a novel method to train small-scale models on downstream tasks.\n2.\tThough the method is a combination of existing other methods, I haven’t seen it before in the literature.\n3.\tThe reward function is well designed.\n4.\tMethod seems to outperform some KD methods.", "reasons_to_reject": "1.\tPaper’s aspects hard to understand and writing needs to be improved. See questions for authors #1.\n2.\tSome aspects of authors idea are not clearly explained:\n\ta.\tHow is the generator trained? Pretrained only? Or finetuned as well?\n\tb.\tPPO’s agent is not clearly described. It is quickly mentioned in lines 435-436.\n\tc.\tHow do you initialize small models?\n3.\tMethod looks more over-engineered than theoretically sound; besides it is heavy and requires so many steps for a mitigated result:\n\ta.\tPre-training LLM\n\tb.\tFinetuning the reward model.\n\tc.\tTraining the ensemble within the framework imagined by authors.\n\td.\tGenerate a database (knowledge store).\n\te.\tUse data retrieval to enhance the smaller model performance.\nThe above steps should be conducted for each downstream task, which questions the usefulness of this method in practice. \n4.\tMissing some comparison baselines, such as simple finetuning. How does finetuning a randomly initialized/truncated BERT model perform compared to your method?\n5.\tRL section seems a bit handwavy: some maths/equations/details on the PPO agent would be more than welcome!\n6.\tPaper’s architecture is nearly the same as for PromDA’s, except for the processing of the samples, where the former uses PPO algorithm, while the latter uses consistency filtering.\n7.\tMethod limited to text classification (for tasks) and models with sentence representations (like BERT with CLS token). This is not mentioned in the limitations.", "questions_to_authors": "1.\tI don’t quite understand the paragraph from line 48 to line 50. You mention knowledge distillation as a compression method, then claim that such “compression methods are not suitable for scenarios requiring high compression ratios, such as knowledge distillation…”. Besides, KD does not necessarily require high compression ratio, you can in theory use any ratio you want. \n2.\tI don’t quite understand the paragraph from line 259 to line 262: “unable to optimize these learning objectives due to lack of per-token differentiability”.\n3.\tHow do we assess the impact of length penalty on the training? Penalizing length only does not mean that the model will generate sensical sentences.\n4.\tParagraph 303->305: What s the LM you are talking about? The classifier or the large frozen model? Is there any equation representing the global loss function (including everything)?\n5.\tHow is the whole training performed? Jointly? Sequentially?\n6.\tLine 407: are you using T5 as a teacher and Bert as student? These are two different architectures. Maybe I am not aware of it, but just out of curiosity, has this been done before in the literature?\n7.\tLine 446: you mean eq. 4?\n8.\tIf I understand well, the small scale is only used to generate a representation that is close to those previously calculated by the reward model.  The decision only relies on the labels previously generated by the RM. Say the test sample and the retrieved sentences are by no means related to each other. That means the final answer is likely going to be based on answers to different questions. Correct?", "paper_topic_and_main_contributions": "This paper presents a method to enhance the performance of compressed models using a framework. The framework comprises soft-prompt-tuning an LLM that generates samples. These samples are refined through a reinforcement learning algorithm (PPO) using the soft-prompts model as an agent, and a reward function considering the diversity of samples and the accuracy of an auxiliary classifier. The latest is used to train the smaller model to generate closer embeddings, but also to generate the labels and store them together with embeddings in a database. During inference, the smaller model generates an embedding of the test sample, then queries the database. The top closest samples are then selected, and the answer is computed with a formula combining similarity scores along with the confidence of the classifier with respect to each label (like how softmax works). The label with the largest logit is then selected. This method seems to achieve good performance on some superglue/GLUE tasks, considering the size of the final model. However, it looks like it is computationally heavy and not very practical.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IjPatkQ9Yl", "reviewer": "Reviewer_6ByT", "summary": "", "questions": "Please refer to Reasons to Reject, thank you!", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your recognition of our work and the valuable feedback. We will provide a detailed explanation based on your concerns.\n\n#### **Q1: Regarding the T5-3B as the teacher model.**\n\n**A1:** Thank you for pointing it out. We would like to provide some clarification here. Firstly, due to constraints in computational resources, our experiments were exclusively conducted on T5-3B, as indicated in the 'Limitations' section of our paper. Secondly, when we refer to 'extremely large,' we are actually alluding to the significant compression applied to the teacher model. In essence, it means an exceptionally high compression ratio. For instance, transitioning from 3B to 7.2M parameters represents an extreme compression, with a compression rate of 0.24\\%. We will provide additional elaboration on this point in our revision.\n\n#### **Q2: Regarding very small BERT models as the student models.**\n\n**A2:** Thank you for the insightful question. We'd like to provide more explanations as follows.\n\n- We follow established methods for model compression, such as AD-KD (Liu et al.) and MSGKD (Wu et al.), which employ the BERT model as the student model.\n- Our proposed method is primarily designed for scenarios requiring significant compression ratios. Therefore, we conducted experiments using very small student models. These experimental outcomes serve to demonstrate the effectiveness of our approach in high compression scenarios.\n- Given that BERT offers pre-trained models of varying sizes (as released by Turc et al.), we selected BERT as the student model instead of T5.\n\nWe appreciate your suggestions, and will supplement experiments on more student models.\n\n#### **Q3: Regarding the faithfulness of the generated knowledge.**\n\n**A3:** Thank you for the great suggestion. In Table 4, we demonstrate that our generated knowledge becomes more accurate and diverse after PPO training. To further validate the faithfulness of the generated knowledge, we gathered five volunteers to assess the faithfulness of the generated texts for datasets WiC, COPA, and CoLA. Each text will be given a rating from 1 to 5. We randomly sampled 100 texts from each dataset and calculate the average as the final faithfulness score. The results of the faithfulness evaluation are as follows, showing that the faithfulness of generated texts for each dataset is quite high.\n\n| WiC | COPA | CoLA | \n| :----:  | :----: | :----:   |\n| 4.61 | 4.36   | 4.49    |\n\n\n#### **Q4: Regarding extending the experiments from NLU to other sorts of datasets requiring much retrievability.**\n\n**A4:** In this work, we conduct experiments on NLU tasks to get a direct comparison with previous SOTA works. We appreciate your great suggestion to extend the experiments to other types of datasets requiring much retrievability. We plan to evaluate our approach on open-domain QA datasets.\n\nThanks again for your constructive feedback. Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper presents an interesting work. The idea to disentangle and then combine the knowledge base building and embedding learning is well-motivated.\n\n2. The writing is very clear and easy to follow, with good figures and tables presented.\n\n3. The proposed method is evaluated by a diverse set of experiments (GLUE, SuperGLUE, Generation, and Ablation).", "reasons_to_reject": "1. the teacher model used in the paper is a T5-3B model, which is not an “extreme large” language model as said in the title. This is an overclaim.\n\n2. The student model is only tested with very small BERT models (around 1/10 of BERT-base). It raises the concern of the real-world applicability or extendibility of the proposed method. In (Gu et. al., 2023), 100M-700M parameter models are selected as students for a GPT-2-1.5B teacher. Their experiments also include 1.5-6B student models distilled from OPT-13B, which can be claimed as a large model.\nOn the other hand, what about T5 models that belongs to the family of the teacher model? It seems that there is no justification for why small-sized BERT-alike models are selected as the student.\n\n3. Since the teacher model is not large, it is questionable that how well the generated knowledge base is in terms of faithfulness. Some auxiliary evaluation seems necessary to validate this step of the idea.\n\n4. Since the retrievability is claimed to be improved, it seems natural to extend the experiments from NLU (e.g., COPA, CoLA, etc.) to other sorts of datasets requiring much retrievability, such as open-domain question answering (e.g., web questions, natural questions, etc.).", "questions_to_authors": "Please refer to Reasons to Reject, thank you!", "paper_topic_and_main_contributions": "This paper presents an interesting work on model compression. The authors propose to transfer knowledge from larger language models to smaller ones with a multi-step approach: tuning in-domain soft prompts, enhancing model stability with reinforcement learning, creating knowledge stores, and training the small-sized models. The experiments are conducted on various GLUE and SuperGLUE datasets with T5-3B as the teacher and show good performance improvement on very small-sized BERT-7.2M and BERT-13.5M.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Knowledge Distillation of Large Language Models. Yuxian GU, Li Dong, Furu Wei, Minlie Huang. Arxiv 2023.", "Typos_Grammar_Style_and_Presentation_Improvements": "None as far as I know"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new model compression procedure that uses, at inference time, information prepared with a larger model. The proposed method also incorporates further components such as soft prompts and PPO, and its main application scenario lies in low-resource settings. The reviewers highlight the good empirical performance of the proposed method at high compression rates, and the useful ablations. They note that the work is novel in its way of combining existing methods. One of the central criticisms reviewers raise is the complexity of the training/compression procedure, which requires several steps to train soft prompts, PPO, and construction of the database. While this may motivate future work (on simplifying said procedure) it could also limit practicality. Reviewers also mention that the proposed procedure has been evaluated on only NLU/classification tasks, which might be a limiting factor. The information provided via author response on the remaining points of criticism, such as comparison to other baselines, and faithfulness of the database are useful additional datapoints that should be incorporated in the next iteration of the paper.
Decision: Accept-Findings