# P-Distill: Efficient and Effective Prompt Tuning Using Knowledge Distillation
## 4 Experiments
### Qualitative Analysis in Long-Sequence Classification  
To further examine the effectiveness of expanding the input space by using shorter prompts via P-Distill, we conducted a qualitative analysis on the Allsides dataset. The task was to predict the political perspectives inherent in a news article. Therefore, the dataset consisted of news articles that generally exceeded the model's input capacity of 512 tokens.  
We compared the input text and prediction results of P-tuning v2, trained with 32 prompt tokens, to those of P-Distill, which compressed and appended only four prompt tokens, which are shown in Table 8. Note that while P-tuning v2 could utilize up to 480 input tokens by attaching 32 prompt tokens, P-Distill extended the input capacity by using up to 508 input tokens with only four prompt tokens appended.  
We observed that the P-tuning v2 model could not access the detailed explanation of the Fairness Doctrine due to its limited input space. However, P-Distill could access the remainder of the sentence, namely _'broadcasters provide \"equal time\" to divergent political views'_, which provided the key information for the model in making an accurate prediction: 'Center'. This example verifies that the input space preserved by compressing the prompt length with P-Distill could contribute to accurate prediction with the model, leading to better performance overall.  
\begin{table}
\begin{tabular}{l l l l} \hline  & **Fine-Tuning** & **P-Tuning v2** & **P-Distill** \\ \hline BoolQ & 89.06 & 89.17\({}_{(8)}\) & 89.07\({}_{(1)}\) \\ CB & 53.94 & 54.22\({}_{(32)}\) & 53.97\({}_{(4)}\) \\ COPA & 21.82 & 21.83\({}_{(4)}\) & 21.82\({}_{(1)}\) \\ MultiRC & 226.94 & 227.50\({}_{(16)}\) & 227.01\({}_{(2)}\) \\ ReCoRD & 163.12 & 163.53\({}_{(16)}\) & 163.17\({}_{(2)}\) \\ RTE & 42.78 & 42.81\({}_{(4)}\) & 42.79\({}_{(1)}\) \\ WiC & 18.83 & 18.84\({}_{(4)}\) & 18.83\({}_{(1)}\) \\ WSC & 23.11 & 23.17\({}_{(4)}\) & 23.11\({}_{(1)}\) \\ \hline \end{tabular}
\end{table}
Table 7: Comparing GFLOPs of baseline methods and P-Distill on SuperGLUE using BERT\({}_{large}\).  
**P-tuning v2 Input Text (480 tokens)**: President [PERSON] on Wednesday hashed out over a critical news report and escalated his previous attacks on the media by suggesting that news organizations he disagrees with be shut down, alarming free-speech advocates who compared the tactics to intimidation efforts by the [PERSON] administration.  
[...]  
Last week, angered by the ongoing investigations into his campaign's ties to Russia, [PERSON] suggested that the Senate Intelligence Committee investigate news outlets over \"fake news\". Over the weekend, he expressed disdain at late-night television hosts over their \"anti-Trump\" material and _proposed bringing back the Fairness Doctrine, a rule phased out in 1987 that had required_  
**Prediction**: Left  
**P-Distill Input Text (508 tokens)**: President [PERSON] on Wednesday hashed out over a critical news report and escalated his previous attacks on the media by suggesting that news organizations he disagrees with be shut down, alarming free-speech advocates who compared the tactics to intimidation efforts by the [PERSON] administration.  
[...]  
Last week, angered by the ongoing investigations into his campaign's ties to Russia, [PERSON] suggested that the Senate Intelligence Committee investigate news outlets over \"fake news\". Over the weekend, he expressed disdain at late-night television hosts over their \"anti-Trump\" material and _proposed bringing back the Fairness Doctrine, a rule phased out in 1987 that had required broadcasters to provide \"equal time\" for divergent political views on certain issues_. First Amendment advocates roundly condemned the president over his remarks, calling them an assault  
**Prediction**: Center