# Audio Anti-Spoofing Based on Audio Feature Fusion
## 2 Methods
### Feature Fusion  
Google proposed a transformer in [27]. It relies solely on the attention mechanism and has made significant breakthroughs in NLP. Meanwhile, the transformer has also shown good performance in the computer vision (CV) area. Various variants of the transformer [26; 28] based on the attention mechanism have continuously set new state-of-the-art records in CV, surpassing the SOTA models. The original transformer was primarily based on the self-attention mechanism. The self-attention mechanism first multiplies the input vector by three matrices (representing three different weights) to obtain three new vectors (\(QKV\)). Then, the attention output is obtained using the dot product:  
\[\text{Attention}(\text{Q},\text{K},\text{V})=\text{softmax}(\frac{\text{QK}^{ \text{T}}}{\sqrt{\text{d}_{\text{k}}}})\text{V} \tag{1}\]  
The attention mechanism used in the transformer structure is complex dot-product attention. Instead, a multi-head attention mechanism was designed to improve the perfor  
Figure 2: Schematic diagram of remix. We segmented and summed the features and then restored them through convolution and reshaping. mance of the model. This mechanism divides the \(QKV\) vectors into n segments along the last dimension and performs attention operations on each segment, separately. The results are then concatenated and passed through a fully connected layer to obtain the final output. Multi-head attention consists of n heads, where the weights for the \(i\)-th head are denoted as \(W_{i,d}\), \(W_{i,k}\), and \(W_{i,v}\):  
\[\text{head}_{i}=\text{Attention}(q\times W_{i}^{q},k\times W_{i}^{k},v\times W _{i}^{v}) \tag{2}\]  
\[\text{Multihead}(Q,K,V)=\text{Concat}(\text{head}_{1},\dots,\text{head}_{ \text{h}})W^{O} \tag{3}\]  
Audio is a different kind of data from text sequences and images. In the past, when processing audio signals, most of the analyses would start from both the time domain and the frequency domain, and the audio features extracted by wav2 vec2 also have two dimensions, timeframe and feature dimension. Therefore, after the pre-training model in wav2 vec2 extracts the input audio, the feature map size is (T, F), where T represents the timeframe dimension, and F represents the feature dimension. The original transformer would have focused on the information in the input feature map from only one perspective. This is due to the nature of natural language processing (NLP), where the input data are temporal. However, audio data are not only temporal, but frequency domain information is also important. Transformer variants in the computer vision (CV) domain, such as those of [26; 28], cannot effectively handle audio features with one less dimension. This is because the feature map of image data is three-dimensional, with height, width, and channels (H, W, C), whereas audio features are usually two-dimensional. Therefore, we cannot, in most cases, directly use these models for audio features. Many studies [13; 29] have chosen to adapt audio features to models in the CV domain by increasing the dimensionality of the features, for example, through convolution, so that two-dimensional audio features become three-dimensional and can thus be directly used with models in the CV domain. In most cases, good results have been achieved in this way. In our model, however, we wanted to try to keep the dimensionality of the audio features the same. Therefore, we redesigned the attention blocks in the overall framework of the Swin-transformer. Unlike NLP, which is overly concerned with temporal information, we treated the timeframe and feature dimension equally. Unlike other feature fusion approaches, we could not fuse feature maps from different inputs since we only had one feature map as our input. Inspired by [28], we took a single-input feature map and fused the output of the branch by feeding it into different branches. At the same time, we added part of the self-attention to the weight matrix as additional information.