# Two-Stage Fusion-Based Audiovisual Remote Sensing Scene Classification
## 3 Methods
### Audio Module  
As shown in Figure 2, the CNN structure, or the CNN combined with a the basic self-attention mechanism, is the typical foundation for traditional acoustic scene-classification algorithms. For tasks involving natural language processing, Transformer was initially proposed. For the implementation of our audio module, we adopted a Transformer-based model from the field of visual processing, in contrast to the prior usage of the CNN  
Figure 1: The overall framework of the proposed Two-stage Fusion-based Audiovisual Classification Network (TFAVCNet) on audiovisual remote sensing scene classification.  
architecture or a combination of the CNN and self-attention mechanisms for audio scene classification [16, 62, 63]. We constructed the Audio Transformer (AiT) model to extract the time-domain and frequency-domain features of the audio signals, on which we carried out the tasks for audio scene classification.  
Figure 3 displays our audio module. The AiT and ViT share similar foundational structures, as both are non-convolutional models primarily constructed using attention mechanisms. However, a significant distinction arises in their respective input requirements. While the ViT is designed to process three-channel images, the AiT operates on a single-channel log-Mel spectrogram. To accommodate this different input format, we averaged the channel weights of the patch-embedding layer. Furthermore, given the utilization of pre-trained weights, the input dimensions of the log-Mel spectrogram (batch_size, hop_length, n_mels) do not align with the fixed input size of the ViT-B (batch_size, 384, 384).Therefore, we rescaled the position embedding of the ViT according to the dimension of the spectrogram. First, the audio file's log-Mel spectrogram is calculated and used as the patch embedding layer's input. An audio file with a length of \(t\) seconds will receive an audio input with a size of (batch_size, n_mels, hop_length). Please note that the audio files referenced in this context pertain specifically to ground environment sounds, and each of them corresponds directly to a corresponding remote sensing aerial image. For a more-in-depth explanation of the audio spectrogram computation, please refer to the Experiments Section 4.6. The spectrogram is then divided into a series of patches, each of which is fed into a linear projection layer to produce patch embeddings (batch_size, n_patches, 768). After that, we added learnable positional embeddings and categorical variable \(CLS\) tokens at the start of the patch-embedding sequence before feeding them into the Transformer encoder layer. We averaged the two \(CLS\) tokens (classification token and distillation token) to create an audio spectrogram representation in the Transformer encoder using the DeiT-B pre-trained weights from ImageNet. In order to obtain the audio_result vector, we must first obtain the output of the audio module (\(CLS\) token) as the feature of the audio spectrogram (audio embedding), input it into a fully connected layer with Softmax activation, then map the audio embedding to the label for classification.  
According to strict definitions, the patch-embedding layer can be thought of as a single convolutional layer with a big kernel and stride size, while the projection layer in each  
Figure 2: Two classical audio modules. (**a**) Traditional acoustic scene classification using the CNN; (**b**) self-attention mechanism combined with the CNN.  
Transformer block is comparable to a \(1\times 1\) convolution. The design deviates from typical CNNs, which have many layers and tiny kernel and stride sizes. In order to separate these Transformer models from the CNNs, they are typically referred to as convolution-free.