# P-Distill: Efficient and Effective Prompt Tuning Using Knowledge Distillation
## 2 Preliminaries
### Knowledge Distillation  
In artificial intelligence, knowledge distillation is a technique for reducing the size of large models while preserving their performances [14; 15; 16; 17]. Recent improvements in knowledge distillation have presented parameter-efficient online distillation [18], prompt transfer-based knowledge distillation for efficient model adaptation [19], and retrieval-augmented knowledge distillation for enhanced generalization [20]. Furthermore, pseudo-target training has been investigated for enhancing knowledge distillation in natural language generation tasks [21]. Beyond NLP, knowledge distillation has been effectively applied in other domains such as visible-infrared transmission line detection using contrastive learning [22] and object detection through cross-head distillation [23], gradient-guided knowledge distillation for object detectors [24], and instance, scale, and teacher adaptive knowledge distillation for visual detection in autonomous driving [25]. These advancements demonstrate the versatility of knowledge distillation across diverse domains. During knowledge distillation, a smaller student model is trained to internalize and emulate the complex decision-making patterns and behaviors of a larger teacher model. This process involves the behavior functions of the models, \(f^{T}\) and \(f^{S}\), transforming inputs into informative representations, typically defined as the output of any layer within the model. These representations contain abundant information for model predictions. Knowledge distillation is quantified using loss functions, such as the Kullback-Leibler divergence [26] or Mean Squared Error (MSE) [17], as follows:  
\[L_{KD}=\sum_{x\in X}L(f^{S}(x),f^{T}(x)), \tag{3}\]  
where \(x\) is the input and \(X\) and \(L\) denote the dataset and the loss function, respectively. This approach enables the student model to gain a comprehensive understanding of various classes, enhancing its application in fields such as NLP.  
Table 1 summarizes the main notations applied throughout this paper to help clarity and the simplicity of referencing.  
\begin{table}
\begin{tabular}{c l} \hline \hline
**Symbol** & **Description** \\ \hline \(x\) & Input sequence or data point \\ \hline \(Q,K,V\) & Query, key, and value matrices in attention mechanism \\ \hline \(d_{k}\) & Dimensionality of key vectors; scaling factor for gradient stabilization \\ \hline softmax & Softmax function used for normalization in attention \\ \hline \(P_{k},P_{v}\) & Continuous prompts added to key and value vectors \\ \hline \(n_{g}\) & Length of continuous prompts \\ \hline \(d\) & Dimensionality of hidden states and key/value vectors \\ \hline \(f_{T},f_{S}\) & Behavior functions of teacher and student models in knowledge distillation \\ \hline \(L_{KD}\) & Knowledge distillation loss function \\ \hline \(X\) & Dataset used for training or evaluation \\ \hline \(L\) & Loss function (e.g., KL divergence, MSE) \\ \hline \(n_{t}\) & Prompt length of teacher model \\ \hline \(n_{s}\) & Prompt length of student model \\ \hline \(W_{k},W_{v}\) & Learnable weight matrices for prompt initialization \\ \hline \hline \end{tabular}
\end{table}
Table 1: Notation Table.