# Discriminator-Enhanced Knowledge-Distillation Networks
## 3 Experiments
### Baselines and Results  
We present a comprehensive comparison of our proposed Dis-KD approach with several baseline models in two aspects: QAC systems and knowledge distillation. We consider the following strong QAC models for comparison: LWG [36] and MCG [37], which are both generation models ranked by frequency; CLSM [38] and LSTM, which are commonly used for optimizing the ranking and generated by MCG; and neural query language model (NQLM) [35], an end-to-end approach for both generation and ranking based on LSTM. Table 4 presents the performance of different QAC systems, where our proposed Dis-KD approach outperforms all the baselines. Also, we introduce topic models as query generators, pairwise coupled topic model (PCTM) [39] based on latent Dirichlet allocation (LDA) [40] and a topic-based LSTM model [41]. In Table 4, we present the results of our evaluation of QAC systems based on the word level. We first compare the performance of a 2-layer GPT-2 model with other types of models and find that, not surprisingly, it performs worse than the vanilla LSTM model. We hypothesize that deeper neural network language models could learn better word representations and yield better performance. To further investigate, we repeat the experiments using a 6-layer GPT-2 model. We observe that the 6-layer GPT-2 model outperforms all other models in the baseline. To demonstrate the effectiveness of our proposed framework, we add the discriminator loss and introduce a 2-layer GPT-2 model into the distillation framework. The results show that, after learning from the teacher model in our framework, the 2-layer GPT-2 student model's performance not only far outperforms the original model but also outperforms the 6-layer teacher model; we show all the results in Figure 2.  
In the context of text completion or prediction tasks, the PCTM demonstrates exceptional performance on seen data. However, one of the limitations of PCTM, and topic models in general, is their inability to be effectively transferred to scenarios involving unknown query completion. This means that when encountering out-of-vocabulary words, such as \"Dis-KD\" in a dataset not seen during training, PCTM is unable to generate accurate completions. Consequently, while PCTM may be effective in certain applications, its reliance on known datasets constrains its applicability in scenarios where unseen data may be encountered. \begin{table}
\begin{tabular}{l l c c} \hline \hline \multirow{2}{*}{**Generation**} & \multirow{2}{*}{**Ranking**} & \multicolumn{2}{c}{**MRR**} \\ \cline{3-4}  & & **Seen** & **Unseen** \\ \hline Dis-KD & Dis-KD & 0.5731 & 0.1763 \\ \hline LWG & Frequency & 0.4465 & 0.2241 \\ \hline MCG & Frequency & 0.4469 & 0.2610 \\ MCG & CLSM & 0.4224 & 0.2628 \\ MCG & LSTM & 0.4293 & 0.2669 \\ PCTM & & **0.614** & \\ LSTM-Topic & & 0.564 & \\ NQLM & NQLM & 0.5395 & 0.1580 \\ GPT(2)-2 & GPT-2(2) & 0.4702 & 0.1660 \\ GPT(6)-2 & GPT-2(6) & 0.5696 & 0.1659 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Results of completion generation. In the area of knowledge distillation, various approaches have been proposed to train smaller models with comparable performance to larger models. One such approach is Distil-BERT [42], which employs a triple loss that utilizes supervision from a bigger language model to distill a smaller one. This framework has been shown to achieve performance similar to that of the larger model. Another approach is Patient-KD [43], which utilizes the output from the last layer of the teacher network for distillation and enables the student model to patiently learn from multiple intermediate layers of the teacher model for incremental knowledge extraction. In addition, we add an example in Table 5, respectively showing the recommended candidate words under different methods. As few works have directly applied knowledge distillation to GPT-2, we re-implement the above framework and transfer the BERT-based knowledge distillation to a GPT-2-based framework with the same student modules.