# Discriminator-Enhanced Knowledge-Distillation Networks
## 2 Related Work
### Details of the Model
Let \(\theta^{s}\) and \(\theta^{t}\) represent the parameters of the student and teacher models, respectively. Moreover, \(P^{s}(\cdot|\cdot)\) and \(P^{t}(\cdot|\cdot)\) denote the corresponding probability outputs from the student and teacher models, respectively. Additionally, \(N\) represents the length of the sequence, and \(W\) is the vocabulary size of the model. Therefore, we define the distance between the predictions of the teacher and student models as follows:  
\[L_{CE}=-\sum_{n\in[N]w\in[W]}[P^{t}(w_{i}|x_{i-1};\theta^{t})]\cdot logP^{s}( w_{i}|x_{i-1};\theta^{s}) \tag{4}\]  
To enhance the effectiveness of knowledge distillation, we aim to develop a discriminator that evaluates the entire sentence rather than individual words. To achieve this goal, we introduce an encoder, such as a bidirectional LSTM network, which reads the sequence of contextualized vector representations \(h_{i}\) generated by the language model (either the student or the teacher GPT-2). The objective of the discriminator is to differentiate between \(h_{i}\) generated by the student and teacher models. We aim to train the student model to generate representations that can successfully fool the discriminator. We define the signal received from the discriminator as \(s\):  
\[s_{i}=sigmoid(LSTM(h_{1},...,h_{i})) \tag{5}\]  
Taking inspiration from [PERSON] et al. [30], we simplified the reward computation process in SeqGAN [7] by employing a Monte Carlo search with a roll-out policy to calculate the reward at each time step. In our proposed framework, we use the discriminator not only to evaluate the entire sequence but also to assess sub-sequences generated after each word. For instance, in the sequence \"Say hello world\", we evaluate \"Say\", \"Say hello\", and \"Say hello world\". Furthermore, we consider evaluating the potential generation probability of each word. The reward for the entire sequence is then defined as  
\[Score=\sum_{i=1}^{N}\prod_{1}^{i}P^{t}(w_{i}|x_{i-1};\theta)(1-s_{i}) \tag{6}\]  
The training of the proposed framework involves two steps of backpropagation. In the first step, the discriminator is optimized as a binary classifier. Once the discriminator is trained, Equation (6) is used to calculate the score of the student's output and the teacher's output. A smaller difference between the scores indicates that the output of the student model more closely resembles that of the teacher model. To optimize the generation of the entire sequence, the loss function is defined as follows:  
\[L_{Score}=-(y*log(y_{lable}+(1-y)log(1-y_{label}))) \tag{7}\]  
The discriminator's output is designed to yield a value of 0 if the input is determined to be the output of the teacher network, and 0 if it is deemed the output of the student network. To address the simple dichotomous task, we employ Equation (7) as a straightforward cross-entropy loss measure to optimize the discriminator. Next, we go back to training the discriminator, and this cycle iterates several times. Even though the policy is simplified, the 2-layer distilled model outperforms the 6-layer teacher model. The data flow is described in Appendix A.2. The additional loss used for the student model is the language model loss, which is the same as the loss used in GPT:  
\[L_{LM}=-\frac{1}{N}\sum_{i=1}^{N}logp(w_{i}|x_{i-1}) \tag{8}\]  
Combined with the traditional knowledge-distillation loss, the final objective function can be formulated as follows:  
\[L_{DisKD}=(1-\alpha)L_{CE}^{S}+\alpha\;L_{LM}+\beta\;L_{Score} \tag{9}\]  
where \(\alpha\) and \(\beta\) are hyper-parameters that balance the importance of different objective functions; \(\alpha\) is 0.6 and \(\beta\) is 0.35 in our model.