# Enhancing Far-Field Speech Recognition with Mixer: A Novel Data Augmentation Approach
## 3 Methods  
### Model Structure  
Whisper [2] is a transformer-based encoder-decoder model for handling various speech tasks, including speech recognition, speech translation, language identification, etc. Assuming the input log-Mel spectrogram feature vectors denoted as \(X=\{x_{1},x_{2},\ldots,x_{T}\}\), and a target transcription is represented by a sequence of \(N\) tokens denoted as \(y=\{y_{1},y_{2},\ldots,y_{N}\}\). The input \(X\) is initially processed by two 1-dimensional convolution layers, and is then encoded to the hidden vector \(H\) by the encoder of Whisper. Next, the decoder consumes \(H\), a prompt \(S\), and the previously generated tokens \(y_{<j}\) as inputs to predict the next token \(y_{j}\). The whole process can be described as follows:  
\[\begin{cases}H=Encoder(Conv(X))\\ y_{j}=Decoder(y_{<j},H,S)\end{cases} \tag{1}\]  
where the prompt \(S\) used can be denoted as \([\langle|\text{sop}|\rangle,\text{previous text},\langle|\text{sot}|\rangle, \langle|\text{language}|\rangle,\\ \langle|\text{task}|\rangle,\langle|\text{no times-tamps}|\rangle]\). The parts encapsulated within \(\langle|\cdot|\rangle\) are special tokens. To specify the FSR task, we set the \(\langle|\text{task}|\rangle\) to \"transcribe\", and \(\langle|\text{language}|\rangle\) is set as \"en\". The \"previous text\" is an optional sequence of text tokens for decoding control. However, directly adapting such large models to low-resource scenarios can be challenging due to overfitting issues and high computational costs. Therefore, parameter-efficient fine-tuning [10; 23; 24] has been proposed, which involves freezing the pre-trained model and training specific small additive modules, such as Adapter [24], LoRA [10] and reprogramming [25; 26; 27].  
LoRA [10], introduced in NLP, is designed to adapt large language models (LLMs) for specific domains or tasks more efficiently. Researchers have noted that pre-trained LLM weights occupy a low-dimensional intrinsic space. Leveraging this insight, LoRA keeps the original weights unchanged while updating only the low-rank incremental weight matrices. For instance, consider the \(i\)-th linear projection in the forward pass, represented as \(f_{i}(X)=W_{i}^{T}X+b_{i}\), where \(W_{i}\in\mathbb{R}^{d1\times d2}\) and \(b_{i}\in\mathbb{R}^{d2}\) are the frozen weight and bias. LoRA modifies the forward computation to include updates to only these low-rank matrices \(\Delta W_{i}\), which can be expressed as follows:  
\[f_{i}(X)=(W_{i}+\Delta W_{i})^{T}X+b_{i} \tag{2}\]In this equation, \(\Delta W_{i}\) is calculated as the product of two rank-decomposed matrices, \(B_{i}\) and \(A_{i}\), such that \(\Delta W_{i}=B_{i}A_{i}\). The matrices \(B_{i}\in\mathbb{R}^{d1\times r}\) and \(A_{i}\in\mathbb{R}^{r\times d2}\) are the two trainable matrices, with rank \(r\ll\min\{d1,d2\}\). When deployed in production, we can explicitly compute and store the combined weight matrix \(W\) as \(W=W_{i}+B_{i}A_{i}\), and proceed with inference as normal. Consequently, LoRA is designed to introduce no additional latency during inference compared to a fully fine-tuned model. Therefore, we employ the LoRA algorithm for Whisper as our baseline, with a detailed illustration shown in Figure 1.