# MKD: Mixup-Based Knowledge Distillation for Mandarin End-to-End Speech Recognition
## 2 Related Work  
Deep neural networks have achieved great success in many fields such as [12; 13], especially in automatic speech recognition [14]. End-to-end speech recognition has become the mainstream method for training ASR models. The size of end-to-end speech recognition models is huge not matter which framework is adopted from CNN [15], RNN [16] or Transformer [17]. However, it requires heavy computation for both training and testing when the model architecture gets deeper. To mitigate this computational burden, there has been a long list of research on model compression. Knowledge Distillation [18] has been demonstrated as an efficient model compression method, which aims at transferring knowledge from a well-trained teacher model to a small student model. With this additional transfer procedure, the student model can perform better compared to naive training. Transferring class probability and transferring the representation of the hidden layer are two typical training strategies proposed from previous research. [PERSON] [4] first introduced the concept of knowledge distillation by minimising the KL-divergence of the softmax outputs of teacher and student model. Generally, the output layer of a classification model adopts softmax as the activation function. The output of softmax is a probability distribution over the label classes, and the sum of the outputs equals 1. The softmax prediction of teacher model has a nonzero probability value for each target class. This soft label is normally considered more informative than the one-hot encoded ground truth. The KD technique mentioned above only considers the output of the teacher model. In the case of transferring the hidden representation, some KD methods [19; 20] proposed transferring the representation-level information of the hidden layers which minimizes the mean squared error between the representation vector. For speech recognition tasks, [PERSON] [21] applied the teacher-student framework to achieve speech knowledge distillation for the first time. [PERSON] [22] investigated heterogeneous knowledge distillation by refining knowledge from RNN-based models into CNN-based models, and [PERSON] [23] optimised a CTC-based [24] heterogeneous knowledge distillation method. [PERSON] [25] proposed sequence-level knowledge distillation in the CTC framework using the N-best hypotheses of teacher model. [PERSON] [26; 27] migrated sequence-level knowledge into a DNN-HMM framework, and [PERSON] [28] optimized sequence-level distillation based on the probability of the output sequences. Data augmentation is a popular tool for training speech recognition models, especially for low-source ASR [29]. The purpose of data augmentation is to constrain the overfitting problem by constructing additional new samples. The traditional data augmentation methods for speech recognition modify the raw audio. [PERSON] [30] investigated three distortion methods: vocal tract length distortion, speech rate distortion, and frequency-axis random distortion. [PERSON] [31] changed the speed of the audio signal, producing three versions of the original signal with speed factors of 0.9, 1.0 and 1.1. [PERSON] [32] imposed a random linear warping along the frequency dimension. Data augmentation schemes [33] was explored for low resource languages. These methods obtain a high increment of accuracy and have low implementation costs for ASR models. SpecAugment untied the model structure and the masked strategy. It regards the masked method as a means of data augmentation and performs random masking on the log-Mel spectrogram of the input speech. A random permuted method SpecSwap was presented to construct new samples. These noise-based methods effectively improve the robustness of the model. More recently, some researchers focused on automatic data augmentation. AutoAugment [34] was proposed to learn a constant policy under a meta-learning setting for many image recognition tasks. Adversarial AutoAugment [35] improved AutoAugment by searching a policy resulting in a higher training loss.