# Audio-Visual Action Recognition Using Transformer Fusion Network
## 1 Introduction  
As the sources of video data vary across diverse fields such as the military [1], unmanned systems [2], surveillance systems [3], and personal security [4], the range of human actions depicted in videos also becomes highly diversified. As a result, recognizing these actions presents a significant challenge. This challenge necessitates the utilization of more sophisticated methods and additional information sources to accurately differentiate between these actions. We note that the traditional approaches [5; 6; 7; 8; 9; 10] to action recognition in videos have predominantly relied solely on visual information. With significant advancements in video-based deep learning techniques [7; 8; 9; 10; 11; 12; 13; 14], visual features have become the primary clues for tasks related to video understanding. This predominance stems from the visual nature of videos, where actions, movements, and scenes are primarily conveyed through visual cues. Visual features, such as shapes, colors, and motion patterns, provide critical information for interpreting content and discerning different actions or events within a video. In the two-stream approach [7; 8; 11], spatial and temporal features are extracted from each spatial and temporal stream. For capturing temporal features, [7; 11] utilized optical flow, and [8] used dynamic images. By extending input data along the temporal dimensions, [9; 10; 12; 13; 14] enabled the learning of spatio-temporal features. However, relying solely on visual information can be limiting, especially in scenarios where visual cues are ambiguous or obscured. We recognize that most videos inherently comprise both visual and audio components, which have the potential to complement each other. Audio can enrich video understanding by providing additional layers of information not always visible, such as dialogue, background sounds, or audio cues that correspond to off-screen activities. In challenging situations such as occlusions, poor lighting, or ambiguous actions, the audio component can provide indispensable contextual clues, thereby enhancing action recognition. For instance, the sound of footsteps or a vehicle in motion can offer vital insights into an action that is not visually discernible. Recent research [15; 16; 17] has begun exploring the potential of combined audio-visual data, emphasizing its significance in advancing video understanding. [PERSON] et al. [15] proposed a framework to learn from video appearance, motion, and audio, investigating both earlyand late fusion. [PERSON] et al. [16] utilized the correspondence between visual and audio information for training the network. Lastly, [PERSON] et al. [17] utilized a pathway that connects audio features to the layers learning visual features, thereby training a unified representation. This synergy between audio and visual elements allows for a more comprehensive approach to video analysis, leading to improved accuracy and robustness in action recognition tasks. In previous CNN-based approaches, the fusion of different modalities was achieved through straightforward methods, such as early fusion and late fusion. Early fusion, also known as feature-level fusion, involves combining features from different modalities, like audio and visual, at the initial stage before feeding them into a learning model. This method can capitalize on the raw data's synergy but may also introduce noise and complexity. On the other hand, late fusion, or decision-level fusion, occurs at a much later stage, where the decisions or predictions made from each individual modality are combined. While this approach maintains the purity of each modality's data, it may fail to capture deeper, more complex inter-modal interactions. Meanwhile, with the advent of the transformer architecture, more sophisticated attention mechanisms, such as self-attention [18], have proven to be highly effective in capturing complex relationships within a single modality by enabling the model to assign varying degrees of importance to different elements in the input sequence.