# Audio-Visual Action Recognition Using Transformer Fusion Network
## 4 Experimental Results
### Results  
The performance of the proposed method was evaluated with several state-of-the-art approaches on the UCF-sound [15] and Kinetics-sound [17] datasets. The experiment was conducted across four distinct cases, examining various combinations of audio-visual data. These cases encompassed the following: (1) all elements, including a single frame (spatial), \(T\) frames (temporal), and audio; (2) single frame and \(T\) frames without audio; (3) single frame and its corresponding audio; and (4) \(T\) frames and its corresponding audio. This facilitates a comprehensive evaluation of the interactions between various audio-visual configurations and their influence on the experimental outcomes. In the UCF-sound dataset, Table 1 represents an ablation study comparing the performance of different versions of the Swin transformer for the four cases (1)-(4). The results show that case (1), which includes all elements (a single frame, \(T\) frames, and audio), consistently outperforms the other cases across all Swin Transformer variants, indicating the importance of integrating all modalities. This indicates the effectiveness of a multi-modal approach. Conversely, case (4), involving only \(T\) frames and their corresponding audio, shows the lowest performance, suggesting that the spatial data (single frame) significantly contributes to the model's accuracy. The performance differences between cases (2) and (3) further emphasize the individual contributions of the spatial and audio modalities. Figure 5: Examples of the actions in the UCF-sound and Kinetics-sound datasets: (**a**) is UCF-sound and (**b**) is Kinetics-sound. The results of the entire Swin Transformer versions show that case (3) outperforms the other two cases, while case (1) has slightly better performance than case (2). Consequently, while the inclusion of audio data does contribute to an overall improvement in performance, it is evident that visual data play a more crucial role in determining the effectiveness of the model on the UCF-sound dataset. The results of the ablation study on the Kinetics-sound dataset are shown in Table 2. This consistency across datasets underscores the importance of integrating spatial, temporal, and audio data for optimal action recognition. Notably, in this dataset, case (4) (temporal frames and audio) outperforms case (2) (single frame and \(T\) frames without audio), suggesting that the combination of motion and audio features is more effective than spatial and temporal features alone. These variations between the two datasets highlight the impact of dataset-specific characteristics on the efficacy of different modality combinations. [PERSON] et al. [15] achieved results by either the early fusion (EF) or late fusion (LF) of spatial, temporal, and audio features and by making predictions using either neural networks (NNs) or SVM (see Table 3). EF involves concatenating features or transforming the feature space at the feature level, while LF refers to fusing different modalities at the decision level. Our Swin-B model achieves a superior performance of 93.00%, which is approximately 11% better than the best result of [PERSON] et al. [15]. \begin{table}
\begin{tabular}{c c c c c} \hline
**Case** & **Swin-T** & **Swin-S** & **Swin-B** & **Swin-L** \\ \hline (1) & **86.63\%** & **91.05\%** & **93.00\%** & **92.53\%** \\ (2) & 84.21\% & 87.79\% & 91.00\% & 91.95\% \\ (3) & 86.63\% & 90.16\% & 91.79\% & 91.84\% \\ (4) & 74.32\% & 79.26\% & 89.05\% & 90.84\% \\ \hline \end{tabular}
\end{table} 
Table 1: The ablation study for Swin transformer variants on the UCF-sound dataset: (1) all elements, including a single frame (spatial), \(T\) frames (temporal), and audio; (2) single frame and \(T\) frames without audio; (3) single frame and its corresponding audio; and (4) \(T\) frames and its corresponding audio.The bold numbers represent the best results for each Swin type.