# P-Distill: Efficient and Effective Prompt Tuning Using Knowledge Distillation
## 1 Introduction  
Pre-trained language models (PLMs) have been effective in improving the performances of various natural language processing (NLP) tasks [1, 2, 3]. These models are fine-tuned by optimizing all parameters to enhance the performances of specific downstream tasks; however, fine-tuning requires significant computational resources while training. The need for significant computational resources for storage and training becomes a challenge, especially when fine-tuning large language models such as Llama2 [3], which may not be readily available to most users. To reduce computational costs, researchers have explored various methods for efficiently fine-tuning parameters [4, 5, 6]. In contrast to the traditional model of fine-tuning that updates all parameters for a downstream task, P-tuning v2 [6] fixes the pre-trained parameters and only trains the continuous prompts, which are trainable embeddings attached at the beginning or throughout each layer of the model. While P-tuning v2 is computationally efficient, especially for PLMs with a large number of parameters, it does not effectively address the inefficiency of utilizing the input space due to the use of continuous prompts [5]. This attachment increases the attention computation, thereby requiring the truncation of the positional encodings for the attached prompts, which reduces the available input token sequence length. Extending the input token sequence by forcibly modifying the code can lead to issues with attention calculation beyond the model's training [7]. Such modifications often result in performance degradation, as the attention mechanism might struggle with sequences extending beyond its initially intended scope. Similar to the findings in the work [6], more challenging tasks require longer prompt lengths to achieve better performance. In this paper, we propose P-Distill, which is a novel prompt compression method to mitigate the limitations of long prompts. Our method involves a two-step process where we first train a teacher model using P-tuning v2 to achieve superior performance with long prompts. We then transfer this knowledge to a student model with significantly shorter prompts through a distillation process. To ensure stability in training, we first perform prompt initialization based on the teacher model prompts. Then, we focus on distilling knowledge between the teacher and student models, specifically targeting the outputs of their intermediate and prediction layers. This is due to the impact of continuous prompts on the hidden states within these layers, which subsequently influences the model's predictions. This method enables the compression of prompts to shorter lengths without a significant degradation in performance, thereby addressing the inefficiencies inherent in longer prompts. To validate its effectiveness and efficiency, we evaluate P-Distill using various NLP benchmarks. Our results demonstrate that P-Distill exhibits comparable or superior performance to those of the existing state-of-the-art prompt-based models. To the best of our knowledge, this study is the first to train teacher prompts and transfer their knowledge to student prompts for compressing prompts. The main contributions of this study are summarized as follows:  
* We propose a method called P-Distill to compress the continuous prompts, effectively mitigating the limitation of reducing the model's usable sequence length in prompt-based learning. * We introduce a prompt distillation method utilizing the teacher model's hidden-state and prediction outputs, influenced by continuous prompts, and propose a prompt initialization for stable prompt distillation. * We validate P-Distill across multiple NLP benchmarks, demonstrating its ability to maintain or enhance accuracy while reducing prompt lengths by up to eight times.