# MKD: Mixup-Based Knowledge Distillation for Mandarin End-to-End Speech Recognition
## 3 Methodology and Aim
### Mixup in Speech-Transformer  
A robust end-to-end speech recognition model is insensitive to noise. Transformer has become the most popular baseline model for sequence-to-sequence problems due to its ability of modeling long-term dependency, but the complexity of Transformer-based model is extremely high. The dependence on data volume of Transformer is much higher than that of recurrent neural network. Thus, overfitting becomes a severe problem for Transformer-based methods in a limited-data setting. Data augmentation is an efficient tool to improve the robustness of neural networks, especially for low-resource speech recognition. Mixup is a prevailing data augmentation method for supervised learning tasks. It trains a model on a linear combination of pairs of inputs and their targets to make the model more robust to adversarial samples. In this setting, the model can achieve more accurate rate under mixed noise. Mixup is computed as follows:  
\[X_{mix} =\lambda\cdot X_{i}+(1-\lambda)\cdot X_{j} \tag{2}\] \[Y_{mix} =\lambda\cdot Y_{i}+(1-\lambda)\cdot Y_{j}\]  
where \(X_{i}\),\(X_{j}\) are the input vectors, and \(Y_{i}\),\(Y_{j}\) are the corresponding targets. \(X_{i}\) and \(X_{j}\) are randomly sampled from dataset \(D=<X,Y>\). \(\lambda\) is sampled by a _Beta_ distribution \(B(\alpha,\alpha)\) with \(\alpha\in(0,\infty)\). The generated pair \(<X_{mix},Y_{mix}>\) is added into training dataset \(D\). For classification problems, Mixup can effectively improve the robustness of the model by smoothing loss landscapes. However, Mixup cannot be directly applied in speech recognition because the length of audio differs from each other which makes it difficult to calculate by Equation (2). Another reason is that the target sequence of each audio is discrete, and the linear combination of discrete data is meaningless. These two issues need to be addressed for most sequence-to-sequence problems. In order to apply Mixup to speech recognition, Mixup is modified at the input level and the loss level, respectively. For input, two raw audios are mixed at the frame level. Before mixture, the shorter input will be padded to the same length as the longer one. Thus, the length of augmented sample \(X_{mix}\) equals \(max(len_{X_{i}},len_{X_{j}})\). A loss level mixture is adopted by mixing two loss function regarding the output. In general, the CTC loss function and the Cross-Entropy (CE) loss function are commonly used in end-to-end speech recognition. CE is adopted in Transformer-based model. For Speech-Transformer, the output of Transformer decoder is sent to a softmax classifier. The result of softmax layer becomes one of the input of CE loss. CE loss is calculated by Equation (3). \[\mathcal{L}_{CE}(\hat{Y},Y)=-\frac{1}{N}\sum_{i=1}^{N}y_{i}\cdot log\hat{y} \tag{3}\]where \(\hat{y}\) denotes the output of ASR model for each character, and \(N\) is the length of \(Y\). The frame sequence and target are aligned by attention mechanism in Speech-Transformer which makes the output of softmax layer synchronized with label sequence. After integrating Mixup into \(CE\), the mixture of the \(CE\) loss becomes:  
\[\mathcal{L}_{CE}(\hat{Y},Y_{mix})=\lambda\cdot\mathcal{L}_{CE}(\hat{Y},Y_{i})+ (1-\lambda)\cdot\mathcal{L}_{CE}(\hat{Y},Y_{j}) \tag{4}\]  
where \(\lambda\) is the same weight as in the input. Theorem 3.1: _For cross-entropy loss function, the mixture of the labels equals to the mixture of the CE loss._  
Proof of Theorem 3.1.: \[\mathcal{L}_{CE}(\hat{Y},Y_{mix})=\mathcal{L}_{CE}(\hat{Y},\lambda Y _{i}+(1-\lambda)Y_{j})\] \[=-\sum\left(\lambda Y_{i}+(1-\lambda)Y_{j}\right)log\hat{Y}\] \[=-\sum\left[\lambda Y_{i}log\hat{Y}+(1-\lambda)Y_{j}log\hat{Y}\right]\] \[=\lambda\mathcal{L}_{CE}(\hat{Y},Y_{i})+(1-\lambda)\mathcal{L}_{ CE}(\hat{Y},Y_{j})\]  
The mixed \(CE\) loss, respectively calculates the \(CE\) loss \(\mathcal{L}_{CE}(\hat{Y},Y_{i})\) and \(\mathcal{L}_{CE}(\hat{Y},Y_{j})\) with label sequence \(Y_{i}\) and \(Y_{j}\).