MEND: Meta Demonstration Distillation for Efficient and Effective In-Context Learning
Authors: Yichuan Li
Xiyao Ma
Sixing Lu
Kyumin Lee
Xiaohu Liu
Chenlei Guo
Venue: ICLR 2024 poster (2024)
Keywords: in-context learning
language modeling
data distillation
knowledge distillation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: efficient and effect demonstration distillation for large language model in in-context learning

Abstract: Large Language models (LLMs) have demonstrated impressive in-context learning (ICL) capabilities, 
where a LLM makes predictions for a given test input together with a few input-output pairs (demonstrations).
Nevertheless, the inclusion of demonstrations poses a challenge, leading to a quadratic increase in the computational overhead of the self-attention mechanism.
Existing solutions attempt to condense lengthy demonstrations into compact vectors. 
However, they often require task-specific retraining or compromise LLM's in-context learning performance. 
To mitigate these challenges, we present Meta Demonstration Distillation (MEND), where a language model learns to distill any lengthy demonstrations into vectors without retraining for a new downstream task. 
We exploit the knowledge distillation to enhance alignment between MEND and MEND, achieving both efficiency and effectiveness concurrently. 
MEND is endowed with the meta-knowledge of distilling demonstrations through a two-stage training process, which includes meta-distillation pretraining and fine-tuning.
Comprehensive evaluations across seven diverse ICL settings using decoder-only (GPT-2) and encoder-decoder (T5) attest to MEND's prowess.
It not only matches but often outperforms the Vanilla ICL as well as other state-of-the-art distillation models, while significantly reducing the computational demands. 
This innovation promises enhanced scalability and efficiency for the practical deployment of large language models.

Published as a conference paper at ICLR 2024
MEND: META DEMONSTRATION DISTILLATION FOR EFFICIENT AND EFFECTIVE IN-CONTEXT LEARNING
Yichuan L Xiyao Ma², Sixing Lu², Kyumin Lee', Xiaohu Liu", Chenlei Guo?
Worcester Polytechnic Institute, *Amazon Alexa AI {yli29, kmlee @eppi. edu (maxiya, cynrrailyytllll derecliu, g @amazznn.com
ABSTRACT
Large Language models (LLMs) have demonstrated impressive in-context learning (ICL) capabilities, where LLM makes predictions for given test input together with few input-output pairs (demonstrations). Nevertheless, the inclusion of demonstrations leads to quadratic increase in the computational overhead of the self- attention mechanism. Existing solutions attempt to distill lengthy demonstrations into compact vectors. However, they often require task-specific retraining compromise LLM's in-context learning performance. To mitigate these challenges, present Meta N Distillation (MEND), where language model learns to distill any lengthy demonstrations into vectors without retraining for a new downstream task. We exploit the knowledge distillation to enhance alignment between MEND and LLM, achieving both efficiency and effectiveness simultancously. MEND is endowed with the meta-knowledge of distilling demonstrations through a two-stage training process, which includes meta-distillation pretraining and fine-tuning. Comprehensive evaluations across seven diverse ICL task partitions using decoder-only (GPT-2) and mncoder-decoder (T5) attest to MEND's 's prowess. It not only matches but often outperforms the Vanilla ICL as well as other state-of-the-art distillation models, while significantly reducing the computational demands. This innovation promises enhanced scalability and efficiency for the practical deployment of large language models
INTRODUCTION
Large language models (LLMs) have demonstrated exceptional power in in-context learning (Kaplan et al. 2020 Brown et al. 2020 Dong al.) 2023 Min al. 2022a). They can rely limited number of input-output pairs, often termed demonstrations, to generate outputs for a given test input, without parameter updates. However, significant bottleneck arises: incorporating demonstrations exacerbates input length for LLMs. This is concerning, especially considering the self self-attention mechanism inherent in these models, which imposes time and memory complexities that scale quadratically with input length.
Attempts to mitigate this challenge typically focus on trimming the context length by distilling extensive demonstrations into concise vectors as shown in Fig. These vectors then used to prompt the LLX to generate outputs (Phang et al. 2023 Ivison et al. 2022 Mu et al. 2023 Lester et al. 2021). Distillation approaches, however, differ across methodologies. For instance, methods such as prompt tuning (Lester et al., 2021 Wang et al. (023) produce vectors through gradient descent. Nonetheless, these approaches necessitate specific retraining for different demonstrations. In contrast, the introduction of hypernetworks (Ha et al. 2016) offers solution that reduces the reliance on gradient descent for any given demonstrations. Methods like Hypertuning (Phang et al. 2023) and HINT([vison et al.) (022) employ conditional language modeling (CLM) objectives to finetune a language model based distillation model, distilling demonstrations into vectors. Yet, when benchmarked against the Vani laiila ICL method-wli LLMs are prompted directly with the unaltered demonstration text-the performance exhibits discernible degradations using these distilled
* "This work mainly done during Yichuan's internship Amazon. TTee code is avaliable https///gthhuh..o///hhhhiuu/au/u

---

Published as a conference paper at ICLR 2024
vectors. This trend remains consistent, even when distillation models are co-trained with the LLM in ICL data (Ivison et al. 2022). Given that these language model based distillation models inherently possess in-context learning capabilities and can generate meaningful representations, the remaining question is how to optimize them to generate demonstration distillation that rival or even surpass the efficacy of Vanilla ICL. Achieving this would pave the way for enhancing ICL efficiency without compromising its efficacy.
During pretraining, LLMs usually learn using detailed word data. But at demonstration distillation scenario, they have to work with a simplified version of this data distilled vectors. It's like studying with a full textbook but taking the test with only a summary. We think it's really important to make sure that the L.MM can understand and use these summaries just as well as the full textbook. This helps the LLM perform better when it's actually being used for ICL. To address this, we introduce the Meta dEmonstration N Distillation (MEND). Our approach realigns the distillation model, MEND and LLM through knowledge distillation (Hin- ton al. 2015: Snell al. 2022). Here, the LLM, when prompted solely with the distilled vectors (acting as the student), is conditioned to emulate the behavior would exhibit when exposed to the full demonstrations (assuming the role of the teacher). To achieve this, we minimize (KL) divergence between teacher and
Vanilla-ICL
PromptTuning
Dusput
Large longuage Model
Targe languege Model
Delk
Irput
Hypernetwor twork
Outoutt
Demo nxeillation
Medel
Large Language Model
Figure I: Vanilla ICL method uti- lizes the concatenation of demonstra- tions and test input generate the out- put. In contrast, PromptTuning and HyperNetworks employ distilled vec- tors in place of the full demonstrations. The length of these distilled vectors is significantly shorter than that of the demonstrations, contributing to more compact and efficient in-context learning for LLM.
the Kullback-Leein student models' word distributions. Importantly, during this optimization process, we backpropagate the gradients from the LLM to MEND, while ensuring that the LLM remains frozen throughout. The training paradigm for MEND is twofold: meta-distillation pretraining on standard text pre- training data (e.g. C4 (Raffel et al. (019)), followed by finetuning on ICL tasks. This two-stage training equips MEND with the meta- knowllde for distilling demonstra- tions, allowing it to generalize effectively across unseen demonstrations without sacrificing performance.
To demonstrate the feasibility of MEND, we apply it to a variety of LLM architectures, including both decoder-only (c.g.. GPT-2 GPT-2 (Brown et al. (2200) and encoder-decoder configurations (c.g.. T5 (Raffel et | 2019)). In our experiments on the MetaICL dataset (Min et al. (202a)). encompassing 142 unique NL. tasks divided across seven partitions, MEND consistently meets or exceeds the performance of Vanilla ICL, notably outperforming where traditional hypernetwork approaches falter. Across the range of language models we investigated, our distillation strategy results in a substantial reduction of up to 75% in FLOPs and accelerates inference by up to 33%. Beyond standard evaluations, we embarked on an in-depth diagnostic analysis where we tweaked the distillation ratio and added intentional disturbances to the demonstrations. In these scenarios, MEND proved resilient to the disruptions and consistently outpaced standard Vanilla ICL methods.
Summarizing our work, our contributions threefold: (1) The introduction of MEND, an innovative technique aimed at enhancing the LLM's in-context learning efficiency without compromising the performance; (2) An exploration into the benefits of knowledge distillation for aligning the demonstra- tion distillation model with LLM; (3) Comprehensive quantitative and qualitative examinations that highlight the robustness and effectiveness of MEND.
2 PROBLEM DEFINITION
Let D = (I, y:) y, be a demonstration set, where x, and y denote the input and output tokens respectively, and K is the number of input-output pairs or demonstrations. Let D denote the concatenation of demonstration set that is D = concat( V/, ... ... yK F In in-context learning (ICL), given D, and test input x, the large language model (LLM) will compute the conditional
IIn the following sections will use concatenated demonstrations and context interchangeably.
2

---

Published as a conference paper at ICLR 2024
Trainable
LLM Head
Student Distribution
LLH Head
Step1: Pretraining
Data:
Text Sequences (1024)
x
Transformer
Layer
Input:
1024x (1 !
peverrste
Transformer
Transformer
Transformer
4Posttion
Embeedding
Emnoddiing
Step2: Distiliation Finetuning
@t.
1111
11
000-000 D
441
Teather Distribution
Data:
Demou: concat(
Word Embedding f
Word Embedding
Word Embedding
Input:
Demo
Woggh
Me-1
MEND
Student: LLM with
Staree
Teacher: LLM with
Lypeernn - Egrat P.Etsiit
Distilatioon
Original Demonstration
Figure Overview MEND. MEND takes as input demonstrations and distillation placeholder, outputs distillation vectors. To capture the meta-kmen wottaknnwwedgee of demonstration distillation, MEND is trained in two stages: meta-distillation pretraining and fientuning.
probability for each label E C and return the maximum conditional probability as:
argmax aggmax,ccPPuxcccconeeee Puuu(clconcat((E)) Ex)).
(1)
where C is the unique set of {y in classification tasks or answer options in question answering tasks, and (-) is LLM's word embedding.
To improve the efficiency of ICL, many related works (Lester et al. 2021 Phang al. 2023 Ivison
et al. 2022 Wang et al., 2023 Mu et al. (023) aim to reduce the demonstrations length for LLM from [D] into such that << [D]. They synthesize high-fidelity demonstration summary Sp !xd
where d is the hidden size of word embedding, replace D:
argmax cec Pun Ez)).
Prompt tuning approaches (Lester et al. 2021 Wang et al. 2023) consider Sp as learnable parameters.
However, for other tasks' demonstrations like D' it requires additional training time get S
Hyperne twork approaches (Phang et al. 2023 Ivison et al. 2022 Mu et al. 2023) including our MEND address the challenge of retraining for novel, unseen tasks. They achieve this by employing demonstration distillation model, denoted M, which produce distillation vectors: Sp M (Ép) and S M (E These vectors correspond to any arbitrary demonstrations 1 D'. Here E.)) represent the word embedding derived from the demonstration distillation model. Notably, previous Hyperne twork methods has the compatibility issues with LLM, resulting in distillation vectors of suboptimal quality.
METHODS
The whole framework of MEND is illustrated in Fig. We insert / special tokens to the vocabulary set of distillation language model MEND, which act as placeholders for the demonstration distillation. For any demonstrations D, these placeholders embedding ÉS are appended to the demonstrations embedding Ep. fostering versatile distillation strategy suitable for diverse tasks. After multiple transformer layers inside MEND, we can distill the information from lengthy D to compact distillation abbreivated as Sp MEND (Ep).
vectors Sp MEND (conat(EE,, E&)) [-1:]
3.1 KNOWLEDGE DISTILLATION
The goal to knowledge distillation to use a concise demonstration summary, Sp, such that the downstream LLM behaves similar (e.g. output close word distributions) to its version conditioned the full demonstrations D. To realize this, we treat the LLM with full demonstration D as the "teacher" and the version with only the demonstration summary Sp as the "student". Subsequently, KL divergence to assess the difference between the word probability distributions of these two models.
employ
Caistill KI. (z || Pa MEND (EN)))))))
We opted for KL divergence as our distillation objective to ensure the student model does not produce outputs that are too different from the teacher model.

---

Published as a conference paper at ICLR 2024
3.2 OPTIMIZATION
Throughout our two-stage optimization process, LLM remains frozen, assisting in backpropagating the gradient from the loss MEND.
Meta-distillation Pretraining. To help MEND capture the general knowledge of distillation, we pretrain it on a text pretraining data-C4 (Raffel et al. 2019). As illustrated in the right segment of Fig. 2] we extract sequences of 1024 tokens from the pretraining dataset. This sequence divided into two parts: the first 1024 X 3 tokens as demonstrations D and the remainder, 1024 x (1 - -)), as input x, where 3 is the hyperparameter to control the length of demonstrations. We then apply the knowledge distillation approach to pretrain MEND. In contrast with the conditional language modeling objective, where LLM predicts subsequent content based on compressed tokens (Phang et al., 2023 Ivison et al. 2022). our demonstration distillation is trained by minimizing Ldistill and aims ensure the distillation model more accurately captures the intrinsic attributes of MEND. Consequently, it can offer a more faithful demonstration distillation. As evidenced in Tab. 2 and Tab. 4 demonstration distillation consistently outperforms the traditional conditional language modeling CLM approach.
Meta-distillation Finetuning. During this stage, we finetune MEND using ICL relevant tasks, equipping with the ability interpret task's semantics from its demonstrations. This ensures that MEND can effectively generalize to unseen demonstrations in the future. In each iteration, we choose a meta-training task and extract K + demonstrations from The first K demonstrations are concatenated into D, while the remaining pair, (IK+1, VK ) is reserved for test input and output purpose. Similar the pretraining phase, the demonstrations D are fed into the distillation model MEND, yielding the demonstration distillation Sp. The primary purpose of Sp is to instruct the LLM in producing y and guarantee that LLM operates as though was condition on the original demonstrations. The formulation of finetuning is as follows:
Epdd log PLax (y) concat(Sp. E.))
(4)
Linneuee
Lpred + ALdisun
where À is the
to control the importance of distillation in finetuning.
4 EXPERIMENTS
4.1 EXPERIMENT SETTING
Benchmarks. In the section, validate our methodology, we employ the MetaICL dataset introduced by Min et al. (202a)), designed for in-context learning scenarios. MetaICL builds upon existing few-shot datasets, such as CrossFit (Ye et al.) 2021) and UnifiedQA (Khashabi et al. (020). Notably, the MetaICL dataset is divided into two distinct partitions: meta-train and meta-test, with no overlap between them. This setting expect the model first trained meta-train then evaluated on meta-test dataset. Our experiments encompass seven distinct meta-train and meta-test partition: outlined in Tab. I] In ICL, the context length is directly proportional the number of demonstrations, For instance, in the Class->Class, with 16 demonstrations, each demonstration's 's average length is 56.21 tokens. Consequently, during inference, the average context length extends to 899.36 tokens (calculated as 16 56.21) which will bring additional computation compared with no demonstrations length with 56.21.
Following MeL setup (Radford et al. 2019). we utilize whitespace to delineate input and output. In most of our experiments, we have preset the number of demonstrations to K 16. For evaluating model performance, accuracy metrics employed for classification tasks, while Macro-Fl is utilized for classssiifiannonn tasks. In partitions that encompass both classification and non-classification tasks (such LR), we compute the average of Macro-F1 and accuracy to assess overall performance.
Base Models. To illustrate the adaptability of our proposed MEND framework, assess its performance using various backbone large language model architectures, including decoder-only models, such as GPT2 (Radford et al. 2019), and encoder-decoder models, like T5 (Raffel al.)
3The tasks and their corresponding abbreviations can be found in Appendix A
4

---

Published as a conference paper at ICLR 2024
[ We initially experimented with using different architectures for MEND and find that the when MEND and LLM are from the same model family works best. Thus, for GPT-2, we choose gpt2-smal while for T5 we select t5 15-small-Im-an -maal-lm-dappp
Baseline Methods. We compare the perfor- mance of MEND against four primary groups of baseline methodologies: /) shot: This approach utilizes the LLM for direct zero-shot inference. 2) Vanilla ICL: Here, we employ LLM for in-context learning by conditioning on concatenation of K randomly selected demonstrations. 3) PromptTuning (Lester This strategy offers an efficient approach to adapt LLM to new tasks without requiring full retraining. 4) HyperTuning: Phang al. (2023) employs language model to distill demonstrations into condensed vec- tors using conditional language modeling objective. For fairness, Prompt Tuning and HyperTuning use same prompt lengths and hypermodel sizes equivalent those used in MEND. Further details regarding hyperparameter settings and analysis can be found in Fig. 4
Table 1: Statistics of seven different task partitions. Each row indicates meta-training/test task partitions.
meta-train # task
meta-test
Sctting
Avg. Len.
Setting
# task
Avg. Len.
Class
43
44.54
Class 20
56.21
non-Class
91.45
QA
37
91.58
QA
22
non- QA
33
72.50
Don-NLI
54.51
NLI
8
61.61
HR
82.44
LR
26
35.31
noD-Para
55.97
Para
5-1.06
4.2 EXPERIMENT RESULTS
Effectiveness. This section outlines the results from experiments, detailed in Tab. 2 the following observations: Fürstly, the zero-sho shot approach predominantly underperforms, indicating that the inductive biases introduced during meta-training (PromptTuning), meta-testing (Vani lla ICL), or both (HyperTuning and MEND) enhance in-context learning. Secondly, when compared with PromptTuning. both HyperTuning and MEND demonstrate marked improvements. This underscores the effectiveness and generalizability of using hypernetworks to distill the supervising signal from demonstrations to assist LLM. A potential reason for Prompt Tuning's inferior performance that it solely captures inductive bias through gradient descent during meta-training and cannot leverage bias from the meta-test's s demonstrations at meta-test time. Thirdly, Vanilla ICL outperforms HyperTunii Tapprrunnng, while MEND consistently matches or even surpasses Vanilla ICL. This suggests that our approach, incorporating Laistill and Lredd. is adept at capturing the meta meta- -ntwllde facilitating the distillation demonstration to aid LLM.
We make
Inference Efficiency. Inference efficiency remains a fundamental aspect of our study. The core idea of our work is to distill extensive natural language demonstrations, denoted as D, into concise distillation vectors, denoted as Sp thereby reducing computational demands for LLM. To assess the efficiency of our model, we report the computational costs associated with different representation techniques in terms of processing time, memory consumption, and floating-point operations per second (FLOPS). Specifically, for each meta-test partition, we select single task, evaluate it with batch size 1, and measure the aforemennioned metrics. Considering that HyperTuning operates identically to MEND during inference, we have chosen Vanilla ICL and Prompt PromptTuning baseline methods. It is important to note that the inference efficiency of MEND encompasses both the process of obtaining the distilled vectors and the subsequent inference by the LLM using these vectors in conjunction with the test input. Compared with promptTuning MEND bring additional computational cost at compressing demonstrations into compact vectors. As illustrated in Fig. MEND achieves up to 3.5 times greater computational efficiency compared to Vanilla ICL and requires less peak GPU memory. Remarkably, while MEND demonstrates efficiency on par with Prompt t Tuuning, also presents a notable performance enhancement, as evidenced in Tab. 2 These observations indicate our proposed method MEND can improve the LLX's efficiency without sacrificing LLM's effectiveness in in-context learning.
IIn C we have test our proposed method on flat-15-xl and opt-6.7b. hhttps://hugglin htttps://huggingaacc.o//g//g
ottps:///nggunggaae.cooooogg
smalmmmll--- adapt
5

---

Published as a conference paper at ICLR 2024
Table 2: Performance on the MetaICL Dataset: This table shows the average and stand deviation scores from running our evaluation with five distinct random seeds. To enhance readability, we present the meta-train and meta-test pairs the format "meta-tres - meta-test". The best-perfiee best-performing models are highlighted in bold, while the second-best are underlined. The standard deviation values reflect the variability due to different demonstrations retrieved. Note that the "PromptTuning" and "zero-shof" approaches do not require demonstration retrieval, hence their standard deviation is zero
Methods
Class Class
nou- Class + Class
non-NLI NLI
Ron- QA-QQA
QA +
HR - LR
non- Paru Pura
AVG
gpt2-large
zero-shot
34.36
34.36
25.50
44.58
44.58
34.77
34.12
36.04
PromptTuning
37.65
38.78
31.34
38.71
45.77
40.08
34.23
36.17
Vanilla ICL
41.30=2.16
41.30g2.16
39.13-2.50
45.81g154
45.81
11.26.2222
38. 28.9321.15
11.93
HyperTuning
40.4221.64
42.54g1m9
36.49-2.01
41.11 20.52
46 46.20_o.50
41 41.63=1.2
39.63
41.15
MEND
43.35
43.38 21.02
39.96
44.29 g0.30
46.92 20.49
40.92
42.54.0.44
43.05
gpt2-xl
zero-shot
32.08
32.08
25.54
46.09
46.09
33.95
33.61
35.63
PromptTuning
37.65
38.78
36.27
11:45
16.95
10.83
35.52
39.61
Vanilla ICL
20.63_2.8
40.6322.5
37.35
48.32=0.88
48.3220.88
42.27
37.53_1.04
42.15
HyperTuning
40. 30
43.74 11.001
40.71 34.61_1.28
47.41
41 41.83
35.72.0.43
40.61
MEND
22.79
43.37 1.30
37.00
45.95
48.07 -0.00
12.16
g2.53
43.12
15-Im-lange
zero-shot
36.75
36.75
25.72
39.05
39.05
32.09
34.28
31881
PromptTuning
32.56
32.37
25.80
39.48
39.44
32.43
36.44
34.07
Vani ICL
38.40
36.68 38.40_2.7
37
39.26
39.26
38.77
36.31 10.51
38.15
HyperTuning
31. 31.17 27
29. 29.06
33.56 13.56.1.7
39.03
39.03.1.0
41.17_00.s
34.28
37.39
35.09
MEND
11.75
38.9 38.93
1.44
37.15.2.00
11.76.0.00
=2.51
39.07
=2 16
36.96
36.99
39.79
FLoPe (TPLOPS)
Nemoryy
[Sesonds]
IoL.
WEND
0.4
2.0
1.0
QA
16
Clsaa Glava
1R
QA
Classo
-aar
Clars
(a) GPre-Large (774M)
Porward PLOPE (TFLOPS)
Memery
(Becone))
0.4
Dlasa
HR
NL
Pala
Close
GA
MLI
Pana
8.8
Class
MR
QA
ML
Pare
(b) GPT2-XL (1.5B)
Figure 3: Efficient Analysis of In-Context Learning at Inference Time. GPT2-large (774M) and GPT2-XL(1.5B evaluated on the same task with batch size 1. The context length for both PromptTuning and MEND is 100, while for Vanilla ICL varies on the partitions. (Class-Cllass is 469, LIR-+LR 652, QA-QAA is 639, non NLI-NNI is 848, and non Para-Paara 818).
ANALYSIS
In this section, we conduct a comprehensive examination of our distillation approach across various scenarios to gain deeper insights into its behavior and potential limitations. To mitigate computational resource demands, we primarily employ the gpt2-large model as LLM on Class- setting unless mentioned otherwise.

---

Published as a conference paper at ICLR 2024
5.1 VARYING DEMONSTRATION DISTILLATION RATIO
A crucial aspect of our experimental analysis was to comprehend how varying the demonstration distillation ratio impacts the distillation of demonstrations and, consequently, the effectiveness of LLM's in-context learning. The demonstration distillation ratio is defined as the ratio of the number of demonstrations the length of distillation vectors. Specifically, we vary the distillation ratio from two perspectives: the richness of input (the number of demonstration examples) and the compactness of the output (the length of demonstration distillation).
Varying Number of Demonstrations. We assess the effectiveness of our method while altering the value of K (the number of demonstration) while keeping length of the distillation vector I constant. As depicted in Fig. 4a our MEND approach consistently outperforms the Vanilla ICL and HyperTuning methods for various values of (1, 2, 4, 8, and 16). Furthermore, MEND demonstrates consistent performance improvement K increases, whereas Vanilla ICL reaches its peak performance at = 4. This improvement suggests that MEND is excels at extracting supervision information for in-context learning from the selected demonstration examples.
45
MacroFF1 (%)
Macro
Vanilla-ICL
HyperTuning
MEND
35
PromptTuning
HyperTuning
MEND
25
(a) Number of demonstrations.
(b) Length of distillation vectors.
Figure Performance with different demonstration distillation ratio. The distillation ratio is the ratio of the number of demonstration examples to the length of the distillation.
Varying demonstration distillation Length. We manipulate the length of demonstration distillation 16. worth noting that we retrain MEND with two
1 1, 50, 100 and 200 while keeping stages shown in 3.2 for different 1 values. The results in Fig. 4b yield the following observations: Firstly, as the demonstration distillation length increases, the performance improves, except for 200 in the of PromptTuning. This suggests that there may be information loss in demonstration distillation, and increasing the length of the demonstration may help mitigate this issue. However, there exists trade-off between efficiency and effectiveness, as extending the length of the distillation vectors results in a quadratic time complexity increase. Secondly, we observe that our proposed method achieves the best performance among the baseline methods, including HyperTuning. This underscores the significance of our optimization design in providing enhanced inductive bias for in-context learning.
all methods generally
5.2 PERTURBATION TO DEMONSTRATIONS
Given the significant influence of provided demonstrations the performance of in-context learn- ing (Min et al. 2022b), we aim to investigate whether our proposed approach, MEND, can effectively distill and propagate modifications made demonstrations the distilled vectors. To address this, we empirically perturb the demonstrations from both positive and negative perspectives.
Positive Perturbation. In light of previous research Liu al. (2021) emphasizing the value of semantically similar demonstrations and their positive impact on in-context learning, we aim to ascertain whether MEND's advantages are complemented by or enhanced through the use of improved retrieved demonstrations. We transit from a random sampling approach to a more nuanced semantic-base j-NN retrieval method. As indicated in Tab. 3 semantic- based retrieval methods, including dense and bm25, exhibit superior performance compared random selection under the No Perturbation condition. Remarkably, MEND not only matches or even surpass the performance of these advanced retrieval methods and does with reduced context size.
7

---

Published as a conference paper at ICLR 2024
Table Performances when applying perturbations on demonstrations.
Positive Perturbation
Negative Perturbation
Methods
No Perturhation
bm25-eNN
dense-&NN
No Label
No Toput
Random I Label
Wrong Label
Vanilla ICL
11.30
15.38
48.33
30.57
42. 29
37.5 25
28.13
HyperTuning NEND
40.42
13.95
45.13
31.78
38.20
38.72
29.31
43.35
46.82
48.81
32.57
44.29
39.25
30.42
Negative Perturbation. We evaluate the impact of various negative perturbations, including the following scenarios: 1) No Label: This perturbation involves removing the labels while retaining the imputs. 2) No Input: The inputs are removed while keeping the labels intact. 3) Random Label: This perturbation randomly selects one of the valid options as the output. 4) Wrong Label: In this case, one of the incorrect options is randomly selected. The results are presented in Tab. 3/ As anticipated, consistent trend emerges, with No Perturbation outperforming both Random Label and Wrong Label for both theVanilla ICL and our proposed MEND. Moreover, that performance improves in most cases when the No Input perturbation is applied. This not only underscores the significance labels in the context in-context learning but also illustrates MEND's s ability to effectively distill label information into the distilled vectors.
noteworthy
5.3 ATTENTION WEIGHT VISUALIZATION
To gain a deeper understanding of how demonstration distillation impacts LLM, we employ visualization techniques to explore the attention weights of LLM's induction heads, as introduced by Olsson et al. (2022). Induction heads are attention heads known for their prefix matching and copying properties, which play crucial role in the context of in-context learning. They empirically increase the likelihood of [B] given [B] ... [[A] when repeated sequence of tokens. Our objective is to understand whether store the input-output pattern that will activate these induction heads in a manner similar to the original demonstration tokens.
demonstration distillation
We visualize the attention weights of the four induction heads both Vanilla ICL and MEND, as illustrated in Fig. 5] A review of Fig. 5 reveals that the final prediction establishes a constructive association with the demonstration distillations. Given that the length of demonstration tokens (average=91- 911rg 4) and compressed prompt tokens (100) significantly exceed the length of test input, we employ max pooling to map the attention weights of the demonstrations into 20 tokens (Area enclosed by red rectangle). This in-depth analysis further substantiates that the distillation derived from MEND offers valuable context supervision signals for LLM.
(b) Attention Visualization of MEND
Figure 5: Attention visualization. The left red surrounded x-axis denotes either the demonstrations (Vanilla ICL) or the distilled vectors (MEND) and the other part of x-axis are the tokens from the test input. The y-axis corresponds the first token of the output word.
5.4 ABLATION STUDY ON DEMONSTRATION DISTILLATION
To assess the significance of the Laistill, conducted experiment that excluding this term during both the pretraining and finetuning stages on several representative task paritions.
Pretraining. During the pretraining phase, we compare using no-pretraining. conditional language modeling (CLM) (Phang et al.) 2023). and CLM+Laisull We find that (1) pretraining is crucial as it substantially enhances performance compared methods with no-pretraining, except for the no-pretraining baseline; (2) our pretraining approach outperforms the alternatives. We hypothesize that this superiority attributed to our pretraining scheme better align the MEND and LLM.
The details of identifying induction heads be found in C s Mooe analysis about CLM+ Caistill can be found in Appendix
8

---

Published as a conference paper at ICLR 2024
Finetuning. In this phase, we retained the same pretraining objective function but omitted various finetuning components. Examining the lower section of Tab. 4 we observe that the removal of each component leads to decrease in performance. This observation underscores the positive contributions of each component within our proposed method to the overall performance. Table Ablation study of knowledge distillation.
Methods
non: Class Class
non Mon-NLI NI.
Rom-
Q4 - QA
Avg.
QA-QA
Vanilla ICL
41.30
39.13
45.81
45.81 46.92
43.01
KEND
43.38
39.96
44.29
43.65
Ablation Study on Pretraining
No-Pretraining
34.33
40.10
CLX
42.00
39.09
44.13
46.47
42,92
CLM + Lamu
41.38
38.79
43.69
45.10
42.24
Ablation Study Finetuning
MEND w/o Gyred
37 64
33,90
43.97
44.54
40.41
NEND w/o Caistill
39.26
37.22
10.29
15.78
10.64
In this experiment, also observed that both the pretraining and finetuning ablations of MEND significantly underperform compared to Vanilla ICL. This finding underscores the critical role of the two-stage design, encompassing both pretraining and finetuning, in our model's effectiveness. Moreover, it highlights the essential contribution of knowledge distillation in replicating the teacher model's behaviors and harnessing meta-training knowledge. These results collectively illustrate the synergistic impact of these components in enhancing MEND's performance.
6 RELATED WORK
Hypernetwork The concept of Hypernetworks as introduced by Ha et al. (2016). refers to an auxiliary network designed to generate parameters for primary network. similar view, MEND can be perceived as a Hypernetwork, producing distilled vectors (parameters) to tailor LLM for new tasks. Notable efforts like HyperTuning (Phang et al., 2023), HINT (Ivison al., 2022). Hyper & Ren] (202T) have employed a language model-based distillation model to condense demonstrations into distilled vectors. While these methods can adapt to unseen demonstrations, they often degrade with ICL performance. On the other hand, Gist (Mu et al., (023) enhances the LLM with instruction distillation and instruction following. However, given that the distillation model is synonymous with the LLM, the distillation procedure induces computational overhead, especially when compared with our approach that deploys a smaller language model for distillation. A distinctive advantage of MEND over existing wwwerneewwrk-baaee demonstration distillations is its simultaneous realization of efficiency and effectiveness as shown in Tab. 2 and Fig. 3
Knowledge Distillation Knowledge distillation, introduced by Hinton et al. (2015), seeks to transfer insights from a high-capacity model to a model with lower capacity. This methodology is key in ensuring both efficiency and effectiveness for MEND, setting MEND apart from other HyperNetwork techniques. Askell et al. (221)) ) [Sell et al. (022) exploit the knowledge distillation to finetune LLM with the ability function as the language model with a prepended prompt when did not provide any prompt. Nonetheless, given the diverse nature of demonstrations,
illustrated in Tab. 3 these methods fail to include superior demonstrations for better ICL performance. Furthermore, as MEND functions as complementary module for LLM, it doesn't 't hamper LLM's inherent capabilities Furthermore, as MEND functions as a complementary module for LLM, it doesn't 't hamper LLM's inherent capabilities.
7 CONCLUSION
We introduced MEND to not only tackle the inherent efficiency challenges in in-context learning with large language models but also to address the effectiveness limitations of existing demonstration distillation methodologies. Our innovative approach distilled in-context demonstrations into vectors, tailored for downstream large language models. Rigorous evaluations of MEND across seven distinct few-shot task partitions and two major large language model families have underscored its prowess. Notably, MEND consistently matches even surpasses the performance of traditional in-context learning, all while demanding fewer FLOPs. This breakthrough paves the way for more efficient and scalable applications of large language models in real- world scenarios. In the future, aim to distill an even broader spectrum of demonstrations, some potentially surpassing the context window limits of both the demonstration distillation model and LLM.

---

Published as a conference paper at ICLR 2024
REFERENCES
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. arXiv preprint arXiv: 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Aydar Bulatov, Yuri Kuratov, and Mikhail Burtsev. Recurrent memory transformer, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-fiettuedd language models. arXiv preprint arXiv:22100 1416, 2022.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey in-context learning, 2023.
Sylvain Gugger, L Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, and Sourab Mangrulkar. Accelerate: Training and inference scale made simple, efficient and adaptable, 2022.
David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. ArXiv, NaW/1609.09106 2016. URL
https://api. wwtps/seeaaaicccoooooooooooooooa ottp://cccccccooooooooooooooooaa
Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting: Scaling in-context learning to 1,000 examples, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv 2015.
preprint
Hamish Ivison. Akshita Bhagia, Yizhong Wang. Hannanch Hajishirzi, and Matthew Peters. Hint: Hypernetwork instruction tuning for efficient zero-shot generalisation. arXiv preprint 103:5 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint XXix:2001.08300 2020.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with single qa system, 2020.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, 2021.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3?, 2021.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context, 2022a.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work?, 2022b.
Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens. 2023.
Neel Nanda and Joseph Bloom. Transformerlens, 2022. URL https://gi /tt..cccmm. neelnanda- www.aannnaai//rraaffffmerreeeeee
Catherine Olsson, Nelson Elhage. Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan,
Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv: 1895, 2022.
10

---

Published as a conference paper at ICLR 2024
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf. Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, segh-performnnce deep learning library, 2019.
Jason Phang, Yi Mao, Pengcheng He, and Weizhu Chen. Hypertuning: Toward adapting large language models without back propagaaat...... In International Conference on Machine Learning, pp. 27854-27875. PMLR, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context. arXiv preprint 1IXh::209915188, 2022.
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning, 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-ard natural language processing. arXiv preprint drXiv:1910.03771, 2019.
Qinyuan Ye and Xiang Ren. Learning to generate task-specific adapters from task description, 2021.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: few-shot learning challenge for cross-task generalization in nlp. arXiv preprint arXiv:2104.08835, 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre- trained transformer language models. ArXiv, abs/2205.01068 2022. URL https://api. semanticscholar.o0m 2.r.rl..r....roooooo..2222222222

---

Published as a conference paper at ICLR 2024
A DATA, TRAINING, EVALUATION, AND COMPUTE DETAILS
Code and data are available in the supplementary material and will be made public upon paper acceptance via GitHub.
Data. For pretraining stage, we utilize the C4 validation dataset (Raffel et al. 2019) as our training data. We truncate each passage into 1024 tokens. For meta-distillation stage, we limit the context length into 900. Within the demonstrations, any example x, exceeding 256 tokens truncated from the end. However, do not truncate the label ys- If the context length surpasses 900 tokens while i < K, the subsequent demonstrations ((x+1. 3++1)) are omiited.
The tasks and their corresponding abbreviations follows: "Class" for classification, "QA" for question answering, "NLI" for natural language inference, "HR" for high resource, "LR" for low resource, and "Para" for paraphrase.
Training. The complete set of stable hyperparameters for training
be found in Tab. 5
These parameters are adapted from MetaICL (Min et al.) 2022a). Additional hyperparameters that needed exploration and their corresponding search spaces are also detailed in Tab.
For pretraining, we leverage the Class->llass meta-test validation dataset for early stopping. It should be noticed that while determining pretraining hyperparameters, focused our search solely on gpt2-large and subsequently adapted the findings to other downstream MEND.
As for finetuning, use specific meta-test validation data for early stopping. When it comes to the meta-distillation finetuning hyperparameters, we conduct the search for each task split and MEND independently.
The hyperparameter analysis of 3 and X can be found in Fig. and Fig. 6a
Table 5: Hyperparameters for MEND.
Pretraining gpt2-xl
Finetuning gpt2-xl
apt2-large
15-large-Im
gar2-large
15-large-Im
Stable Hyperparameters
num stops
30,000
30,000
5,000
30,000
30,000
30,000
batch learning rate precision optimizer
Se-5
Se-5
Se
Se-5
Se
Se-5
fp16
fp16
fp32
fp16
fp16
adamW
adamW
adam
adam W
adamW True
adam W False
LLM, in Shit early stop patience
True
True
False
True
Seurchalle Hyperparameters
8
[0.1, 0.5, 0.8.
N/A
N/A
N/A
N/A
[0.01.
1,
Compute. We implemented our proposed methodology using PyTorch v1.13.1 (Paszke al. 2019); complemented by the HuggingFace Transformers library v4.24.0 (Wolf et al.) 2019) and Accelerate v0.20.0 (Gugger al. 2022). All experiments were conducted eight A10 NVIDIA GPUs, each equipped with 24GB of memory.
B HYPERPARAMETER ANALYSIS
Pretraining relevant Hyperparameters. During the pretraining stage, there are two important factors greatly influence the distillation models performance for the following Meta-Distillatte fineuning: 3 and 7. , controls the length of demonstrations for distillation during pretraining and controls the importance of knowledge distillation during pretraining. In Tab. 4 show the experiment results of CLM+1 x Ldistill). To comprehensively understand the superiority of sole Laistil1. consider an the hyperparameter analysis the combination of CLM+1 x Laistill- which can be formulated as L Lexx + Caistiil- To save computational resource, different from Tab. we directly report the experiment result after pretraining without further dita-distillation comprehension.
12

---

Published as a conference paper at ICLR 2024
40
40
Macro-F1 (%)
Macro-F1 (%)
38
38
36
36
34
34
0.2
0.4
0.8
0.9
0.01
0.1
1
10
(a) Analysis on B.
(b) Analysis on 7. Dashed indicates no CLM.
Figure 6: Analysis pretraining relevant hyperparameters.
As the result shown in Fig. 6 we have the following observations: 1) MEND achieves the best performance when 0.8. This indicates that during pretraining, proper design the ratio of demonstrations to inputs will achieve better performance than small or large ratios; 2) MEND achieves better performance when increasing the y. This indicates the importance of Laistill (knowledge distillation) in minimize the knowledge gap between the distillation model and downstream language model.
Meta- Distillation relevant Hyperparameters. To understand the importance of knowledge dis- tillation in Meta-distillation finetuning stage, we
gpt2-large
gpt2-xl
t5-large-lm-adapt
vary in Eq. 4] As the result shown in Fig. can observe that MEND achieve beter performance when À >= 1, this also indicates the importance of knowledge distillation.
3.17
2771
Avg Rank
2.29
2.00
1.67
1.71
C ADDITIONAL ANALYSIS
Identify Induction Head. In 5.3 visu- alize the attention weights of induction heads. Here, we introduce how we identify these in- duction heads. Following (Olsson et al.) 2022: Nanda Bloom 2022). we firstly create ran- domly sequences with length 500 then expand them by concatenating with itself for time. Thus we have 10 sequences with length 1000 and for each sequence, the first 500 tokens exact same the rest 500 tokens. Then, inside each self-attention layer, take the diagonal of attention paid from each destination position (position index > 500) to source positions 500 back and get the attention average of each head over these tokens. The average attention score shown in ingF.. We choose the 4 attention head with largest average attention score as the our interested inductive head.
0.1
Figure Hyperparameter analysis on A.
Additional Large Language Model. To assess the efficacy and generalizability of MEND, we conducted evaluations on larger models, specifically opt-6. 7b Zhang et al. (2022) and flan- xxx Chhng et al. (2022). For demonstration distillation, we strategically selected smaller counterparts as backbone models: opt-1255 5m for opt-6. 6. 7b and flan-t5-bass for flan-t5-xl We maintained consistent formatting and training methodologies across these evaluations, using whitespace separate inputs and outputs within and across demonstrations, done with gpr2-large. The results, as detailed in Tab. 6) show that MEND consistently outperforms other baseline methods. This demonstrates its ability to effectively capture and utilize meta neta-knowledge, enhancing the efficiency of demonstration distillation aiding large language models (LLM).
13

---

Published as a conference paper at ICLR 2024
0
5
10
15
5
0.8
10
0.6
15
Layer
20
0.4
25
0.2
30
35
0.0
Head
Figure 8: Average attention weight visualization of attention head from gp2-large.
14

---

Published as a conference paper at ICLR 2024
Robustness towards Template Variations
Table Experiment on advanced large language models.
While the primary objective of our study is to dis- till demonstrations into compact vectors, the ploration of optimal prompt templates is beyond the scope of this paper. In experiments, consistently used whitespace to separate inputs and outputs within and between demonstrations across all models. To assess the robustness of our models against template variations, we con- ducted an additional evaluation. We transferred model trained with a whitespace separator to new template using newline characters (nn) for separating inputs and outputs, and three newlines for differentiating between demonstrations the gpt2-larg LLM. The results, presented in Tab. 7 indicate that MEND exhibits minimal
Methods
Class Class
flan-15-xl
Prompt
33.24
Vanilla ICL
40.63=2
HyperTuning
39.70+1.58
MEND
40.77
opt-6.7b
PromptTuning
38.81 18.81
Vanilla ICL
42.38
HyperTuming MEND
32.67
44.27
sensitivity to these format changes. The performance difference was negligible, with less than 0.3% variance between using spaces and newlines.
D LIMITATIONS
Large Downstream language Models. Due to computational constraints, our experiments use models that are <2B. Whether these demon- stration language distillation techniques gener- alize the largest models (10B+) is unknown. However, given that our method can generalize to different model structures and computation efficiency without hurting the downstream lan- guage model's performance, we believe we are shedding insights for future work.
Table 7: Robustness of template variations. All the method is evaluated Class - Class setting. The Diff. the difference between newline result minus whitespace result.
Methods
whitespace
newline
Diff.
Vanilla ICL
41.30 41.30+2.16
28.00
2.40
HyperTuning MEND
40.42+1.64
40.08
43.35
43.50
+0.15
Language Model dependent. Due to our de- sign of distillation, the MEND may face the adap-
tation problem across different MENDs. This means we need to train new distillation model for any new LLM. addition, because of our optimization design, we need the gradients that back propagate on the top of MENDs. This will bring computation overhead when we try large LLM with larger demonstration encoders.
Limited Context Window. Both MEND and LLM have limited context window. Thus, when demonstrations exceeds the length context, we inevitably need to truncate the demonstration. This will not only lose the information from the discarded tokens and cannot distill large amount of demonstration(e.g
1000 (Hao et al., 2022)). Concurrent work utilizes recurrent memory transformer (Bulatov al. 2022 compress long documents beyond the constraint of context window size into soft prompts. We consider handling extra-long demonstration as our future work.
15


Reviews:
[{"review_id": "yAgvWvP0xC", "reviewer": "Reviewer_Dw9E", "summary": "This work proposes a new method for efficient in-context learning via direct prediction of demonstration context vectors. The proposed method, called MEND, combines hypernetwork training with distillation of regular in-context learning behavior to achieve high-quality \"prompt vector\" synthesis capabilities. Authors validate MEND on the MetaICL dataset using GPT2 and T5 models, showing performance gains and accuracy improvements compared to other in-context learning baselines.\n\n---\n\nPost-rebuttal update: I thank the authors for their response and clarifications, and I am keeping my current score.", "questions": "* In Table 2, what were the standard deviations across runs?\n* How did you format 16 demonstration examples into a single input string for each dataset? For example, there are different ways of joining several demonstrations (space, linebreak etc.). Have you studied the robustness of models/methods to that formatting?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The proposed approach is well-motivated and achieves significant improvements in each of the studied setups.\n* The paper contains a detailed analysis section along with the ablation study for MEND, justifying the necessity of each component of the method.", "weaknesses": "* My primary concern regarding the evaluation is that the models studied in the paper (GPT2-XL, T5-large) are relatively small and not representative of models that actually benefit from in-context learning. In fact, authors acknowledge this limitation in the appendix; I simply believe that having experiments on larger models (for example, training only one distillation model and applying it to larger LMs) would increase the impact of the work.\n* The inference efficiency measurement protocol could likely be improved. First, it is unclear whether key/value caching is used for generation: this should make the impact of additional demonstrations less severe. Also, I think it would be helpful to have a more detailed memory/time breakdown for MEND: measuring only the inference with obtained meta-demonstrations is not sufficient, as the distillation model needs to process input demonstrations into prompts.\n* At times, it was a bit difficult to understand the reasoning of the paper due to grammar errors/typos and word choice. Consider, for examplem, \"the evident an misalignment\" and \"between teacher student's\" on page 2, \"into a condensed vectors\", \"distill the supervisional from demonstrations\", and \"cannot leveraging\" on page 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FFv6n5uSUN", "reviewer": "Reviewer_RDfP", "summary": "This paper proposes MEND: meta-demonstration distillation. The authors design a distillation method to compress a long text demonstration into a short vector. MEND is designed as a meta-distillation method, such that the distillation model can be applied to unseen tasks. Experiments are provided to demonstrate the effectiveness of the proposed method.", "questions": "See above", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The proposed distillation method for prompt-tuning is well-motivated. The presentation is very clear. The proposed distillation method is effective and easy to understand.\n\n* The authors investigate several flavors of models to demonstrate the effectiveness of the proposed method. Specifically, the authors use GPT-2 with different sizes and T5 to show that MEND outperforms existing in context learning approaches.", "weaknesses": "My main concern is about experimental settings.\n\n* Could the authors explain why GPT-2 and T5 are used? These models are usually considered outdated and more recent models should be used.\n  * For the GPT family, GPT-J, GPT-Neo, OPT are all open-sourced. And the LLaMa models are instruction fine-tuned such that they may show different behavior when facing vectorized demonstrations.\n  * For the T5 model, I suggest using Flan-T5, which shows much stronger performance than T5.\n\n* The authors should consider more baselines. For example, Chain-of-Thought (CoT) can demonstrate stronger performance than vanilla ICL. The authors need to at least compare with the vanilla CoT. I also suggest distilling CoT prompts (demonstrations) and see whether this can further improve the performance of MEND.\n\nI will raise the score if the authors can run some experiments on more recent models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "psmuEmugxj", "reviewer": "Reviewer_Uu3b", "summary": "This paper focuses on making in-context learning (ICL) with LLMs more efficient and effective. Vanilla ICL requires one to provide a collection of demonstrations, i.e., task-specific example-label pairs, as context to the LLM while running an inference for a test example. However, this leads to long input sequences, increasing the cost of inference with LLMs where the self-attention cost scales quadratically with input sequence length. This paper proposes to utilize a **demonstration distillation model** to compress the long demonstration sequence into a small number of vectors which can be fed into the LLM as a prompt vector (akin to prompt tuning) during inference. There are existing approaches that adopt such an approach to make ICL efficient. However, those approaches often result in performance degradation. This paper proposes a knowledge distillation-based approach to train the demonstration distillation model. Through extensive empirical evaluations, the paper shows that the resulting demonstration distillation model not only realizes efficient inference by reducing the context length for LLM but often also improves the performance compared to vanilla ICL.", "questions": "1) After Eq. (1), the paper states \"...where $\\mathcal{C}$ is the unique set of $\\{y\\_i\\}\\_i=1^K$...\". Isn't this a restrictive assumption? There could be tasks where test examples may have a label not present in any of the $K$ demonstrations, e.g., factual QA.\n2) In Section 3.2, during the pre-training phase, does one begin with a pre-trained or randomly initialized demonstration distillation model?\n3) Do bars in Figure 3 include the cost of generating $S_D$ (distillate vectors) during inference?\n4) Why is prompt tuning performance missing from Figure 4a? Similarly, why is vanilla ICL (with truncated demonstration) missing from Figure 4b?\n5) On page 8, the paper states ``Moreover, it is noteworthy that performance improves in most cases when the No Input perturbation is applied. This not only **underscores the significance of labels**...`` Could authors expand on this? Why does having only labels in the context help?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper proposes a novel solution to improve the demonstration distillation approach to make ICL efficient. The solution utilizes knowledge distillation in two stages -- pre-training and task-specific finetuning -- to improve the demonstration distillation model.\n2) The paper provides thorough empirical evidence that the proposed approach makes the ICL efficient while also being on par with/improving vanilla ICL. \n3) The paper presents a detailed ablation study to highlight the utility of various components of the proposed approach.", "weaknesses": "There are no major weaknesses in the paper that the reviewer could find. Please see the questions section below for some clarifying questions.\n\nSome minor comments about improving the quality of presentation are as follows: \n\n1) Consider paraphrasing some sentences to make them clearer:\n\n* On page 2, \"Considering the evident a misalignment where the LLM trains on natural language tokens but infers using distillation vectors...\"\n* In Appendix D, \"This will not only lose the information from the discarded tokens and cannot distill demonstration with large $K$ (e.g. $K > 1000$ (Hao et al., 2022)).\"\n\n2) On page 2, \"...we embarked on a in-depth...\" --> \"...we embarked on **an** in-depth...\"\n3) In Table 2, \"0-shot\" --> \"zero-shot\"\n4) In Appendix D, \"...are shading insights for future work.\" -->  \"...shedding insights for future work.\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XkAw6Wffl4", "reviewer": "Reviewer_RPCQ", "summary": "This paper proposes a demonstration compression framework to save retraining LLMs for unseen tasks under in-context scenarios. To be specific, a two-stage training process including knowledge distillation from pre-trained LLMs and fine-tuning on specific tasks endows the framework with both efficiency and effectiveness. Empirical results of decoder-only and encoder-decoder architectures validate the proposed method.", "questions": "- The distillation loss occurs both in the pretraining and fine-tuning stages where a lambda controller balances the influence of distillation. What’s the influence of the lambda? A clear explanation about connecting the training mechanism(two-stage process) and each loss term to the “require task-specific retraining or compromise in-context learning” would help the reviewers better understand the advantage of the method.   \n\n- In Figure 3, is the size of the distillation vectors of MEND the same size as PromptTuning? \nWhat’s the model size of MEND and FLOPs it introduced?  \n\n\nSeveral questions for Table 4:\n- In the ablation of pre-training,  even if the row is labeled as “No pretraining”, the fine-tuning term still contains the distillation loss. So what’s the lamda for this row? \n- What does L_hidn mean in Table 4? Is it L_distill ?\n  If it is L_distll, then the second to the last row is the result of pretraining loss for both stages. The performance looks far from comparable to CLM. Can the authors clearly explain this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The authors proposed an efficient demonstration distillation for in-context learning. \n- The paper is well-organized and easy to follow.", "weaknesses": "- More insightful explanations about why the proposed method does not compromise the in-context learning ability of LLMs could make the paper stronger. \n- self-contained notations may help the readers to understand the results better. \n- Figure 5 can be displayed using more contrasting colors", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)