# Discriminator-Enhanced Knowledge-Distillation Networks
## 2 Related Work
### Details of the Model  
In this section, we introduce a standard knowledge-distillation method for compressing GPT-2 and present our proposed Dis-KD in detail. **Problem Definition** The original large teacher network is represented by a function \(f(x;\theta)\), where the \(x\) is the input to the network, and \(\theta\) denotes the model parameters. The goal of knowledge distillation is to learn a new set of parameters \(\theta^{\prime}\) such that the student network can achieve similar performance to the teacher, with a much lower computational cost. The traditional strategy is to force the student model to imitate outputs from the teacher model on the training dataset with a KL divergence. We add an additional discriminator signal to enhance this process. In our experimental configuration, we employ a teacher network denoted as \(f(x;\theta^{t})\), a deep language model, such as GPT-2. On the other hand, the student network \(g(x;\theta^{s})\) is a lighter model comprising fewer layers. We utilize the QAC dataset, which contains a set of sequences denoted as \(\{w_{1},w_{2},...,w_{i}\}\). Here, each \(w_{i}\) corresponds to a word, subword, or character in the sequence. The input instance for GPT-2 is constructed as \(x=\{w_{1},w_{2},...,w_{i-1}\}\), while the corresponding ground truth for the language modeling task is obtained as \(y=\{w_{2},w_{3},...,w_{i}\}\). The input sequence is fed into the GPT-2 model, which generates a contextualized embedding denoted as \(h_{i}=GPT2(w_{i}\mid x_{i-1})\in\mathbb{R}^{d}\), where \(d\) represents the dimension of the contextualized embedding. Furthermore, the model computes the output probability of the last words based on the preceding sequence using a softmax layer. \(W\) is a weight matrix to be learned:  
\[P(w_{i}|x_{i-1})=Softmax(W(h_{1},...,h_{i-1})) \tag{1}\]  
To implement knowledge distillation, we begin by training a teacher network. For instance, in the case of training a 6-layer GPT-2 model as the teacher network, the parameters  
Figure 1: The framework of Dis-KD. The data are initially inputted into both the teacher and student models. During the first iteration, the discriminator is trained to discern which model is the teacher and which is the student. In the second iteration, three losses are used to train the student model. The KL loss function is employed to compute the error between the teacher and student model. The cross-entropy loss is employed to compute the error between the label and the student’s model’s prediction. The discriminator’s loss is employed to give comments from the perspective of the whole sentence. of the teacher model are represented by the superscript \(t\) in the following equation. The objective during training is to maximize the likelihood of the equation given below:  
\[L^{t}(y)=\sum_{i}\log P(w_{i}\mid w_{i-k},\dots,w_{i-1};\theta^{t}) \tag{2}\]  
Upon training the teacher network, we employ it to guide the student network. However, while using the teacher model for inference to guide the student model, we need to modify Equation (2) as follows:  
\[P^{t}(w_{i}|x_{i-1})=softmax(\frac{W(h_{1},...,h_{i-1})}{T}) \tag{3}\]  
Here, \(P^{t}(\cdot|\cdot)\) represents the probability output generated by the teacher model. This probability distribution is considered the soft label of the student model and is fixed during the knowledge-distillation process. Additionally, we use temperature parameter \(T\) during knowledge distillation, which controls the extent to which the student model relies on the teacher's soft predictions. A higher value of temperature leads to a more diverse probability distribution over classes as suggested in previous studies [25]. To evaluate the difference between the probability distribution \(P^{t}(w_{i}|x_{i-1})\) generated by the teacher model and the corresponding distribution \(P^{s}(w_{i}|x_{i-1})\) produced by the student model, we employ cross-entropy loss. This approach encourages the student model to imitate the behavior of the teacher model.