# MicroBERT: Distilling MoE-Based Knowledge from BERT into a Lighter Model
## 3 Methods
### Distillation Process  
We introduce a comprehensive framework for distillation training that can be applied to various transformer-based teacher and student models. Unlike traditional knowledge-distillation approaches, our framework enables supervision over both intermediate feature layers and the final output. In contrast, conventional approaches primarily focus on word-level features, resulting in redundancy and insufficient attention to sentence vectors. To address these limitations, we propose a distillation process that emphasizes both word vector features and sentence vector features, thereby enhancing overall efficiency. In our method, we assume that the number of layers in the teacher model is \(N\) (\(N>M\)), while we set the number of layers in the student model to \(M\). We want the student model to learn the content of the middle layer of the teacher model. Therefore, we extract the \(M\) layer from the teacher model to distill the knowledge to the \(m\) layer of the student model. The mapping formula is \(n=g(m)\), which represents matching the \(m\) layer of the student model to the \(g(m)\) layer of the teacher model. We specifically specify layer \(M+1\) as the prediction layer and layer 0 as the word embedding layer. Figure 1: Overall Architecture Diagram: This diagram outlines the sophisticated design of the MicroBERT framework. 1 The student model aligns its intermediate features with those of the teacher model using FAL. The predict layer is optimized through the application of soft loss derived from the teacher’s outputs and hard loss based on the ground truth labels. 2 Both models incorporate Mixture of Experts (MoE) layers, guided by routers, to enhance processing capabilities. 3 Furthermore, a discriminator network introduces discriminator loss to ensure the student’s outputs closely resemble those of the teacher. We derived two formulas, \(g(0)=0\) and \(g(M+1)=N+1\), where \(g(m)\) represents the mapping layer between the teacher model and the student model. To determine the optimal number of mapping layers, we conducted a series of comparative experiments to evaluate the impact of different choices for \(g(m)\) on the overall model performance. This layer-mapping knowledge-distillation method effectively reduces the disparity between student and teacher models, enabling selective acquisition of relevant knowledge from the teacher model. As a result, knowledge distillation is performed more efficiently. Our framework provides a layer-mapping approach that significantly enhances the effectiveness of knowledge distillation. We therefore derive the following objectives:  
\[\mathcal{L}_{\text{model}}=\sum_{x\in\mathcal{X}}\sum_{m=0}^{M+1}\lambda_{m} \mathcal{L}_{\text{layer}}\Big{(}f_{g(m)}^{T}(x),f_{m}^{S}(x)\Big{)} \tag{1}\]  
where \(f_{m}(x)\) is the behavior function of \(m\) layer, \(\lambda_{m}\) is the hyperparameter, the main purpose of which is to change the weight of the distillation. \(\mathcal{L}_{\text{layer}}\) is the loss function of the provided model layer. **Hidden Layer Distillation**. Our proposed distillation technique for transformer layers builds upon hidden state distillation. Recent research has shown that the [CLS] token, max pooling and mean pooling, which BERT learns, possess robust sentence vector representations. Consequently, distilling such information becomes a crucial factor for sentence vector-based distillation. This linguistic knowledge encompasses syntactic information and other essential aspects for natural language understanding. We anticipate that the student model (MicroBERT) will learn more semantic knowledge from the teacher model (BERT). Therefore, we introduce a hidden layer-based language-refinement method called FAL. We extract three aspects from FAL: the [CLS] token, mean pooling and max pooling. These aspects are then trained by the students, enabling the student model to focus not only on word vector features but also on sentence vector features.