# MicroBERT: Distilling MoE-Based Knowledge from BERT into a Lighter Model
## 3 Methods
### Discriminator Loss  
A trainable discriminator is utilized to compute the Discriminator Loss (DL). In this context, we introduce the concept of a Generative Adversarial Network (GAN), where the generator takes the role of the student model. The primary objective of the generator is to create adversarial samples. The discriminator is trained using the anticipated output of the  
Figure 3: Efficient Inference on MoE: This diagram illustrates the process within the Mixture of Experts (MoE) layer. It begins with the fusion of multiple expert networks, followed by distillation and compression to consolidate their outputs. The final step involves optimization, where the combined network is fine-tuned for improved performance. instructor model, which serves as its training input, ultimately leading to the acquisition of DL. Ultimately, we obtain a trained discriminator capable of distinguishing between the teacher and student models based on the origin of the projected data. Discrimination is crucial to ensure that the student model learns meaningful representations from the input, rather than merely memorizing the teacher's output. As seen in Equation (10) below, the process of finding a binary function's minimum may be characterized as GAN training. \[\min_{S}\max_{D}\mathcal{L}(D,S)=\min_{S}\max_{D}\mathbb{E}_{x\sim p_{\text{ data}}(x)}[\log D(T(x))]+\mathbb{E}_{x\sim p_{\text{data}}(x)}[\log(1-D(S(x)))] \tag{10}\]  
Two goals are achieved by optimizing the loss function \(\mathcal{L}(D,S)\): first, it motivates the generator \(S\) to provide genuine samples; second, it improves the discriminator \(D\)'s capacity to distinguish between samples from \(Tea\_Pred\) and \(Stu\_Pred\). The generator is associated with the student model \(S\), which attempts to replicate the teacher model \(T\)'s output. Consequently, Equation (11) defines the loss function of the student model \(S\). \[\mathcal{L}_{S}=\mathbb{E}_{x\sim p_{\text{data}}(x)}[\log(1-D(S(x)))] \tag{11}\]  
While the student model's output aims to mimic the instructor model in order to trick the discriminator, the teacher model's output is recognized as authentic data. Thus, Equation (12) may be used to define the discriminator \(D\)'s loss function. \[\mathcal{L}_{D}=\mathbb{E}_{x\sim p_{\text{data}}(x)}[\log D(T(x))]+\mathbb{E }_{x\sim p_{\text{data}}(x)}[\log(1-D(S(x)))] \tag{12}\]  
The average of the binary cross-entropy (BCE) losses corresponding to the teacher and student models' outputs is the discriminator \(D\)'s loss function. The discriminator \(D\) will be trained using this loss function in order to improve the ability to discriminate between the outputs of the student and instructor models. As indicated in Equation (13). \[\max_{D}\mathbb{E}_{x\sim p_{\text{data}}(x)}[\log D(T(x))]+ \mathbb{E}_{x\sim p_{\text{data}}(x)}[\log(1-D(S(x)))] \tag{13}\] \[\Rightarrow \min_{D}\mathbb{E}_{x\sim p_{\text{data}}(x)}[-\log D(T(x))]+ \mathbb{E}_{x\sim p_{\text{data}}(x)}[-\log(1-D(S(x)))]\] \[\Rightarrow \min_{D}\mathbb{E}_{x\sim p_{\text{data}}(x)}[\text{BCE}(D(T(x) ),1)]+\mathbb{E}_{x\sim p_{\text{data}}(x)}[\text{BCE}(D(S(x)),0)]\] \[\Rightarrow \min_{D}\frac{1}{N}\sum_{i=1}^{N}[\text{BCE}(D(T(x_{i})),1)+\text {BCE}(D(S(x_{i})),0)]\]  
where \(T(x)\) represents the output of the teacher model on real data sample \(x\) and \(S(x)\) represents the corresponding output generated by the student model, the discriminator's loss function aims to distinguish between these two sources of information. As derived above, the maximization objective for the discriminator \(D\) in a knowledge-distillation setting can be reformulated as minimizing the sum of two BCE terms. Specifically, the first BCE term quantifies the discriminator's ability to recognize the teacher model's outputs as authentic (i.e., labeled as 1), while the second BCE term measures its performance in identifying the student model's outputs as generated (i.e., labeled as 0).