MEND: Meta Demonstration Distillation for Efficient and Effective In-Context Learning
Authors: Yichuan Li,Xiyao Ma,Sixing Lu,Kyumin Lee,Xiaohu Liu,Chenlei Guo
Presentation: ICLR 2024,Poster

Abstract: Large Language models (LLMs) have demonstrated impressive in-context learning (ICL) capabilities, 
where a LLM makes predictions for a given test input together with a few input-output pairs (demonstrations).
Nevertheless, the inclusion of demonstrations poses a challenge, leading to a quadratic increase in the computational overhead of the self-attention mechanism.
Existing solutions attempt to condense lengthy demonstrations into compact vectors. 
However, they often require task-specific retraining or compromise LLM's in-context learning performance. 
To mitigate these challenges, we present Meta Demonstration Distillation (MEND), where a language model learns to distill any lengthy demonstrations into vectors without retraining for a new downstream task. 
We exploit the knowledge distillation to enhance alignment between MEND and MEND, achieving both efficiency and effectiveness concurrently. 
MEND is endowed with the meta-knowledge of distilling demonstrations through a two-stage training process, which includes meta-distillation pretraining and fine-tuning.
Comprehensive evaluations across seven diverse ICL settings using decoder-only (GPT-2) and encoder-decoder (T5) attest to MEND's prowess.
It not only matches but often outperforms the Vanilla ICL as well as other state-of-the-art distillation models, while significantly reducing the computational demands. 
This innovation promises enhanced scalability and efficiency for the practical deployment of large language models.