# MKD: Mixup-Based Knowledge Distillation for Mandarin End-to-End Speech Recognition
## 3 Methodology and Aim
### MKD: Mixup-Based Knowledge Distillation  
Knowledge distillation is commonly adopted in model compression for speech recognition. Knowledge distillation utilizes a teacher-student network structure that exploits soft labels from the teacher model to guide student network learning. However, this framework is subject to the amount of available data. In particular, tasks with fewer samples provide less opportunity for the student model to learn from the teacher. Even with a well-designed loss function, the student model is still prone to overfitting and effectively mimicking the teacher network on the available data. Existing data augmentation have been explored to combine with teacher-student network, improving the efficiency of knowledge distillation. Unlike other methods of data augmentation, Mixup is a data-agnostic approach which means no prior knowledge is required for augmentation. This brings an convenience for low-resource speech recognition to generate task-specific data. Labels generated by Mixup are smoother than original one-hot labels. However, these soft labels don't include mutual information between each category. On the other hand, soft labels generated from teacher network could reflect the relationship between similar labels, which have more mutual information than one-hot encoding. The two arguments above inspired us to integrate Mixup into teacher-student framework to improve the data efficiency of knowledge distillation. In our teacher-student framework, the architecture of student model \(Model_{S}\) is the same as teacher model \(Model_{T}\). There is only a difference in parameter scale between teacher model and student model in homogeneous neural networks. First, the input \(X_{i}\) goes through the teacher model \(Model_{T}\). The teacher model is trained by original CE loss. The output of Softmax layer in \(Model_{T}\) serves as the soft label of each frame. The student model is encouraged to imitate the prediction from teacher network by minimizing the _KL_ distance:  
\[\mathcal{L}_{KL}=\sum_{i=1}^{n}p(X_{i})log\left(\frac{p(X_{i})}{q(X_{i})}\right) \tag{5}\]Considering that the original cross-entropy is helpful for training the student network. The student network is trained with \(CE\) loss in usual way as well. The final loss \(\mathcal{L}_{KD}\) of student model is the linear combinations of \(CE\) loss and \(KL\) loss:  
\[\mathcal{L}_{KD}=\gamma\cdot\mathcal{L}_{KL}+(1-\gamma)\cdot\mathcal{L}_{CE} \tag{6}\]  
For softmax-level knowledge distillation, the distillation loss \(\mathcal{L}_{KD}^{k}\) is calculated for \(k\)-th frame, and \(\mathcal{L}_{KD}\) is composed of the accumulation of each frame loss. When optimizing the teacher-student framework by Mixup, the label sequences \(Y_{i}\), \(Y_{j}\) are served as one of the input to the Decoder of the teacher model to obtain their soft labels \(\hat{Y}_{i_{t}}\hat{Y}_{i_{j}}\), respectively. As shown in Figure 1, when training the student network, the output of student network \(\hat{Y}_{s_{t}}\), \(\hat{Y}_{s_{j}}\) are used to calculate the \(\mathcal{L}_{CE}\) and \(\mathcal{L}_{KL}\). Equation (4) is applied to generate \(\mathcal{L}_{CE}\) for each frame. However, the mixture of \(\mathcal{L}_{KL}\) is different from that of \(\mathcal{L}_{CE}\) because the KL-divergence \(\mathcal{L}_{KL}\) is not linear for the label \(Y\). To solve this problem, a novel loss function \(\mathcal{L}_{MKL}\) is proposed to approximate \(\mathcal{L}_{KL}\). \[\mathcal{L}_{MKL}=\lambda\cdot\mathcal{L}_{KL}\big{(}\hat{Y},Y_{i}\big{)}+(1- \lambda)\cdot\mathcal{L}_{KL}\big{(}\hat{Y},Y_{j}\big{)} \tag{7}\]  
It could be proved that \(\mathcal{L}_{MKL}\) is an upper bound on the \(\mathcal{L}_{KL}\) using the properties of convex functions. The proof of Equation (7) is given below. **Theorem 2**.