# MicroBERT: Distilling MoE-Based Knowledge from BERT into a Lighter Model
## 4 Experiments and Results  
### Model Accuracy Comparison Experiment  
We construct a small student model MicroBERT\({}_{4}\) with 14.5 M parameters in total. It contains 12 attention heads (\(h=12\)), 4 encoder layers (\(M=4\)), a feed-forward/filter size of 1200 (\(d_{i}=1200\)) and hidden dimensions of 312 (\(d=312\)). The teacher model, which has 110 M parameters, is based on the original BERT\({}_{\text{base}}\) (\(d\) = 768, \(d_{i}\) = 3072, \(N\) = 12 and \(h\) = 12). We extracted the average taken every third layer as the layer-mapping function in the teacher model. so MicroBERT4 learns by taking the average from every 3 layers of BERT\({}_{\text{base}}\). For training, PyTorch 1.7.0 was used to implement the model. This model was trained on a server configuration of NVIDIA GeForce RTX 3090 Ti purchased from Macao SAR, China. The algorithm was trained on Ubuntu 22.04, a 64-bit operating system. We performed knowledge distillation at the intermediate layer for [5; 10; 20] epochs using a batch size of 32. The batch size was selected from [16; 32] and the learning rate was selected from \(\{1\times 10^{-5},\,2\times 10^{-5},\,3\times 10^{-5}\}\). For the single-sentence task, the maximum sequence length during distillation was set to 64. For the sequence-pair task, the maximum sequence length was set to 128. We propose a novel lightweight framework and new mapping methods. To successfully distill a more lightweight model than TinyBERT. To investigate whether our super lightweight model can maintain competitive accuracy, we first tested it on six different well-known NLP open source dataset tasks in GLUE, namely SST-2 for sentiment classification task, MNLI-M for semantic matching task, MRPC and QQP for semantic judgment task and RTE for binary classification task. In terms of models, in addition to our ultra-lightweight model, we also selected the initial large model, which is also the distilled teacher model BERT\({}_{\text{base}}\), and lightweight models for several different compression methods, and also for a fair comparison, we trained 4-layer BERT-PKD\({}_{4}\), 4-layer DistillBERT\({}_{4}\) and 4-layer MiniLMV2\({}_{4}\) using the published code and fine-tuning these 4-layer baselines using the proposed hyper-parameters. The accuracy results for their dataset tasks are shown in the following Table 1.  
In the five datasets selected for this study, accuracy (Acc) was used as the evaluation metric for the tasks SST-2, MNLI-m and RTE. Conversely, the F1-score was employed as the evaluation metric for the MRPC and QQP tasks. Based on the results we found that in the SST-2 dataset for the sentiment classification task, the best result for DistillBERT reached 91.4%, which exceeded the accuracy of BERT (Teacher) by 0.3%. Within this task, our ultra-lightweight model was only 0.6% lower than the average accuracy of the five lightweight models, maintaining a strong competitive position. The best results in MNLI-m for the semantic type matching task were 81.5% for MobileBERT\({}_{4}\) and 84.4% for BERT (Teacher) and the accuracy of our ultra-lightweight model in this task is about 0.4% higher than the average accuracy of the four lightweight models, which is within our acceptance range. In MRPC for the semantic judgment task, the best models are 88.7% for our ultra-lightweight model, reaching an extremely high level. The 86.6% in QQP is about 6.5% more than the average accuracy of the other lightweight models. Finally, in the RTE task, our model is about 3.6% less accurate than the best BERT (Google), while it is higher than the average accuracy.