# Audio-Visual Action Recognition Using Transformer Fusion Network
## 3 Method
### Transformer-Based Feature Fusion  
To integrate the features extracted by the feature extraction module, a transformer-based modal fusion module (MFM) is proposed, as shown in Figure 1. Our MFM is shown in Figure 3. The MFM fuses each feature through the process of co-attention with the transformer encoders. Rather than fusing all features simultaneously, it performs pairwise fusion. Specifically, the process involves the fusion of spatial and temporal features, audio and spatial features, and temporal and audio features, respectively. This methodical approach ensures that the unique characteristics of each modality are effectively combined, allowing for a more comprehensive understanding of the multi-modal data. As shown in Figure 3, the MFM consists of two transformer encoders, each taking in different modal features \(f_{modal\_1}\) and \(f_{modal\_2}\) as inputs. The transformer encoder, as illustrated in Figure 4, follows the same structural design as the encoder used in the vision transformer [28]. However, unlike the conventional approach where the Query, Key, and Value inputs are identical, our implementation feeds different modal features into these components. Figure 2: Visualization of audio transformation for the Kinetics-sound dataset. Top to bottom: raw audio waveforms followed by their respective log-mel spectrogram representations. Top-left: ‘Blowing Nose’, top-right: ‘Playing Accord’, bottom-left: ‘Playing Trumpet’, bottom-right: ‘Tapping Pen’. One transformer encoder performs modal fusion by taking \(f_{modal\_1}\) as the Key and Value and \(f_{modal\_2}\) as the Query, while the other transformer encoder performs modal fusion by using \(f_{modal\_1}\) as the Query and \(f_{modal\_2}\) as the Key and Value. The fused attention output vector \(f_{modal\_12}\) and \(f_{modal\_21}\) can be represented as follows:  
\[f_{modal\_12} =\text{Attention}(f_{modal\_2},f_{modal\_1},f_{modal\_1})\] \[=\text{Softmax}\Bigg{(}\frac{f_{modal\_2}f_{modal\_1}^{T}}{ \sqrt{d_{k}}}\Bigg{)}f_{modal\_2} \tag{1}\]  
\[f_{modal\_21} =\text{Attention}(f_{modal\_1},f_{modal\_2},f_{modal\_2})\] \[=\text{Softmax}\Bigg{(}\frac{f_{modal\_1}f_{modal\_2}^{T}}{ \sqrt{d_{k}}}\Bigg{)}f_{modal\_1}, \tag{2}\]  
where \(d_{k}\) is the dimensionality of the Key vector and \(T\) denotes the transpose operation. The attention function [18] in Equations (1) and (2) takes in Query, Key, and Value in that order. The fused attention output vectors are combined using concatenation to produce the final output \(Y\). Figure 4: Transformer encoder. Figure 3: The general structure of the modal fusion module (MFM). In the fusion of spatial image and temporal video, image features \((f_{I})\) and video frame features \((f_{V})\) are fed into the MFM, yielding the fused features \(f_{IV}\) and \(f_{VI}\). The fused attention output vectors are combined using concatenation to produce the final output \(Y_{IV}\). The fused attention output vector \(f_{IV}\), \(f_{VI}\), and \(Y_{IV}\) can be represented as follows:  
\[f_{IV}=\text{Attention}(f_{V},f_{I},f_{I})=\text{Softmax}\left(\frac{f_{V}f_{ I}^{T}}{\sqrt{d_{k}}}\right)f_{I} \tag{3}\]  
Figure 3.0  
Figure 3.1  
In the fusion of spatial image and audio, image features \((f_{I})\) and audio features \((f_{A})\) are combined in the MFM, producing the fused features \(f_{IA}\) and \(f_{AI}\). The attention output vectors from these fusion processes are then combined using concatenation to create the final output for the spatial-audio combination, denoted as \(Y_{IA}\). The equations for the fused attention output vectors \(f_{IA}\), \(f_{AI}\), and \(Y_{IA}\) are as follows:  
Figure 3.2  
Figure 3.3  
Figure 3.4  
In the fusion of temporal video and audio, video frame features \((f_{V})\) and audio features \((f_{A})\) are combined in the MFM, producing the fused features \(f_{VA}\) and \(f_{AV}\). The attention output vectors from these fusion processes are then combined using concatenation to create the final output for the temporal-audio combination, denoted as \(Y_{VA}\).