# Enhancing Far-Field Speech Recognition with Mixer: A Novel Data Augmentation Approach
## 3 Methods
### Mixing Methods  
Mixup [15] generates new augmented data through a process of linear interpolation, as described in the following equations:  
\[\begin{cases}\hat{x}=\lambda x_{i}+(1-\lambda)x_{j}\\ \hat{y}=\lambda y_{i}+(1-\lambda)y_{j}\end{cases} \tag{3}\]  
where \((x_{i},y_{i})\) and \((x_{j},y_{j})\) represent two randomly sampled examples from the training data. The mixed input feature \(\hat{x}\) and the corresponding mixed label \(\hat{y}\) are generated by blending the \(i\)-th and \(j\)-th examples. The mixing weight \(\lambda\sim Beta(\alpha,\alpha)\) with \(\alpha>0\). When \(\alpha=1.0\), \(\lambda\) is uniformly sampled from the interval [0,1], which also means \(\lambda\in U(0,1)\). By leveraging the prior knowledge that linear interpolations of inputs yield linear interpolations of the corresponding labels, Mixup effectively expands the training distribution. Moreover, some studies [28,29] have theoretically demonstrated that Mixup constitutes a form of data-adaptive regularization, which proves to be highly effective in improving the model's generalization capabilities.  
Manifold Mixup [16] is an extension of Mixup [15], allowing for the linear interpolation of representations from any layer within a neural network. Consider a neural network \(f_{k}(g_{k})\), where \(g_{k}\) maps the input data into the hidden representation at \(k\)-th layer, and \(f_{k}\) subsequently maps the hidden representation to the final output \(f(x)\). For a randomly selected \(k\)-th layer, we process two distinct mini-batches of random data \((x_{1},y_{1})\) and \((x_{2},y_{2})\)  
Figure 1: An illustration of Mixer/Mixer-C and its integration with the Whisper model. We freeze the Whisper model and employ the LoRA algorithm to adapt the model to the far-field speech recognition task.  
as usual, until we reach the \(k\)-th layer. This provides us with two intermediate mini-batches \((g_{k}(x_{1}),y_{1})\) and \((g_{k}(x_{2}),y_{2})\). Subsequently, we construct a mixed mini-batch as follows:  
\[(\hat{g}_{k},\hat{g}):=(Mix_{\lambda}(g_{k}(x_{1}),g_{k}(x_{2})),Mix_{\lambda} (y_{1},y_{2})) \tag{4}\]  
where \(Mix_{\lambda}(a,b)=\lambda a+(1-\lambda)b\). The mixing coefficient \(\lambda\sim Beta(\alpha,\alpha)\) with \(\alpha>0\). This interpolation yields a novel training example based on the hidden representation of the model, effectively serving as a powerful data augmentation technique. Notably, Mixup becomes a special case of Manifold Mixup [16] when \(k\) is set to 0.  
Mixup is inherently suited for classification tasks but encounters challenges when applied to speech recognition due to several fundamental reasons: firstly, text sequences \((Y_{i},Y_{j})\) often exhibit varying lengths, making direct mixing impractical; secondly, the discrete nature of text sequences prohibits straightforward addition; thirdly, mixing two text sequences in the embedding space may lead the model to predict a mixture embedding, which could confuse the ASR model and potentially degrade performance, as the ultimate goal is to recognize a single text transcript from the speech input. Therefore, [PERSON] et al. [9] introduced MixSpeech for speech recognition, which blends two input sequences and applies two separate loss functions in the following manner:  
\[\begin{cases}\hat{X}=\lambda X_{i}+(1-\lambda)X_{j}\\ L_{i}=L(\hat{X},Y_{i}),L_{j}=L(\hat{X},Y_{j})\\ \hat{L}=\lambda L_{i}+(1-\lambda)L_{j}\end{cases} \tag{5}\]  
where \(X_{i}\) and \(Y_{i}\) represent the input speech and corresponding label for the \(i\)-th sample, respectively. \(\hat{X}\) denotes the mixed speech obtained by a frame-wise combination of two speech sequences with mixing factor \(\lambda\). \(L\) denotes the ASR loss, and \(\hat{L}\) denotes the mixed loss. Following Mixup, \(\lambda\sim Beta(\alpha,\alpha)\) with \(\alpha>0\).