# P-Distill: Efficient and Effective Prompt Tuning Using Knowledge Distillation
## 4 Experiments
### Results  
Tables 2 and 3 present the experimental results of fine-tuning, P-tuning v2, and P-Distill. In P-Distill, the prompt length was compressed to one-eighth of that of the teacher model prompts. For fewer than eight teacher model prompts, the length was compressed to 1. For a detailed analysis of prompt compression ratios, refer to Section 4.4.3. In general, the proposed P-Distill method exhibited a comparable or superior performance to those of the other methods while using shorter prompts. \begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline  & \begin{tabular}{c} **BoolQ** \\ **Acc.** \\ \end{tabular} & \begin{tabular}{c} **CB** \\ **Acc.** \\ \end{tabular} & \begin{tabular}{c} **COPA** \\ **Acc.** \\ \end{tabular} & \begin{tabular}{c} **MultiRC** \\ **F1a** \\ \end{tabular} & \begin{tabular}{c} **ReCoRD** \\ **F1** \\ \end{tabular} & \begin{tabular}{c} **RTE** \\ **Acc.** \\ \end{tabular} & \
\begin{tabular}{c} **WiC** \\ **Acc.** \\ \end{tabular} &
\begin{tabular}{c} **WSC** \\ **Acc.** \\ \end{tabular} & **Average** \\ \hline Fine-tuning & \(\mathbf{0.777}\) & \(\underline{0.946}\) & \(0.710\) & \(0.705\) & \(0.706\) & \(0.762\) & \(0.746\) & \(0.683\) & \(0.754\) \\ \hline P-tuning v2 & \(0.764_{(8)}\) & \(0.946_{(32)}\) & \(\underline{0.810}_{(4)}\) & \(0.711_{(16)}\) & \(\underline{0.728}_{(16)}\) & \(\underline{0.794}_{(4)}\) & \(0.756_{(4)}\) & \(\underline{0.731}_{(16)}\) & \(0.780\) \\ \(0.738_{(1)}\) & \(0.929_{(4)}\) & \(0.790_{(1)}\) & \(0.707_{(2)}\) & \(0.721_{(2)}\) & \(0.783_{(1)}\) & \(0.745_{(1)}\) & \(0.692_{(2)}\) & \(0.763\) \\ \hline P-Distill & \(0.726_{(1)}\) & \(\underline{0.964}_{(4)}\) & \(\underline{0.810}_{(1)}\) & \(\underline{0.718}_{(2)}\) & \(\underline{0.726}_{(2)}\) & \(\underline{0.798}_{(1)}\) & \(\underline{0.759}_{(1)}\) & \(\underline{0.721}_{(2)}\) & \(\underline{0.784}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Experimental results on the SuperGLUE validation dataset. For P-Distill, training was performed using a teacher model with the prompt length exhibiting the best performance for P-tuning v2. The numbers in parentheses indicate the lengths of the prompt attached to the model. (Acc. : accuracy; **bold**: the best underline: the second best). \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**NER**} & \multicolumn{3}{c}{**SRL**} & \multicolumn{3}{c}{**QA**} & \multicolumn{3}{c}{**SC**} \\ \cline{2-10}  & **CoNLL03** & **CoNLL04** & \begin{tabular}{c} **OntoNotes** \\ **5.0** \\ \end{tabular} & \begin{tabular}{c} **CoNLL05** \\ **Brown** \\ \end{tabular} & \begin{tabular}{c} **CoNLL05** \\ **WSJ** \\ \end{tabular} & \
\begin{tabular}{c} **SouAD** \\ **1.1 Dev** \\ \end{tabular} &
\begin{tabular}{c} **SOuAD** \\ **2.0 Dev** \\ \end{tabular} & **Allsides** \\ \hline Fine-tuning & \(\mathbf{0.928}\) & \(0.882\) & \(\mathbf{0.890}\) & \(0.827\) & \(0.885\) & \(\mathbf{0.911}\) & \(\mathbf{0.819}\) & \(0.780\) \\ \hline P-tuning v2 & \(\underline{0.919}_{(4)}\) & \(0.880_{(128)}\) & \(0.885_{(128)}\) & \(\underline{0.837}_{(32)}\) & \(\underline{0.890}_{(128)}\) & \(\underline{0.902}_{(64)}\) & \(0.782_{(128)}\) & \(0.775_{(32)}\) \\ \(0.914_{(8)}\) & \(0.866_{(16)}\) & \(0.881_{(16)}\) & \(0.807_{(4)}\) & \(0.877_{(16)}\) & \(0.891_{(8)}\) & \(0.771_{(16)}\) & \(0.772_{(4)}\) \\ \hline P-Distill & \(\underline{0.919}_{(8)}\) & \(\mathbf{0.888}_{(16)}\) & \(\underline{0.886}_{(16)}\) & \(0.817_{(4)}\) & \(\underline{0.885}_{(16)}\) & \(0.896_{(8)}\) & \(0.775_{(16)}\) & \(\underline{0.783}_{(4)}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Experimental results for each method on named entity recognition (NER), question answering (QA), semantic role labeling (SRL), and long-sequence classification (SC). For P-Distill, training was performed using a teacher model with the prompt length exhibiting the best performance for P-tuning v2. The numbers in parentheses indicate the lengths of the prompts attached to the model. All metrics are reported as F1-scores.