# Fusion Classification Method Based on Audiovisual Information Processing
## 3 Proposed Method
### Mode Fusion  
When fusing audio and visual data, it is essential to ensure alignment between the modalities. The data collected in this study consists of 10 s videos, from which the visual and audio data are separated. The visual data have a sampling rate of 24 frames per second, while the audio data are sampled at 24 Hz. By matching each frame of the video with its corresponding audio segment, this study achieves alignment between the audio and visual modalities. The visual modality contains rich detail information, while the audio modality provides complementary auditory details to the visual modality.  
As discussed in Sections 3.3.1 and 3.3.2, this study represents the high-level semantic features of images and sounds as \(\widetilde{X}_{img}\) and \(\widetilde{Y}_{audio}\), respectively. A multimodal fusion network structure is constructed to fuse these features. The high-level semantic features extracted from images and sounds using the CNN are mapped into a shared semantic subspace through a joint architecture for feature fusion, resulting in the fused high-level semantic feature \(\widetilde{X}_{cat}\), as shown in Equation (8):  
\[\widetilde{X}_{cat}=Concat\left(\widetilde{X}_{img},\widetilde{Y}_{audio} \right). \tag{8}\]  
After obtaining the fused high-level semantic feature \(\widetilde{X}_{cat}\) in the feature fusion stage, this paper introduces a Self-Attention module (SA module) to learn the representations of image features and audio features. The high-level semantic feature \(\widetilde{X}_{cat}\), the image features extracted by the 2D CNN \(\widetilde{X}_{img}\), and the audio features extracted by the 1D CNN \(\widetilde{Y}_{audio}\) are passed through a self-channel attention module and ultimately weighted to obtain the result. The optimized high-level semantic feature \(X_{cat}^{SA}\), the optimized image feature \(X_{img}^{SA}\), and the optimized audio feature \(X_{audio}^{SA}\) are then fused. The attention weights \(\alpha_{1}\), \(\alpha_{2}\), and between each modality are calculated for \(\widetilde{X}_{cat}^{SA}\), \(\widetilde{X}_{img}^{SA}\), and \(\widetilde{Y}_{audio}^{SA}\), respectively, as shown in Equation (9):  
\[\alpha=Soff\text{max}\Big{(}SA\Big{(}X^{SA}\Big{)}\Big{)}\in\mathbb{R}^{2\times H \times W}. \tag{9}\]  
Among them, \(\alpha=[\alpha_{1},\alpha_{2},\alpha_{3}]\) and \(\alpha_{1}+\alpha_{2}+\alpha_{3}=1\).  
In this study, self-attention weights were used to carry out weighted fusion of each modal feature, and finally, the final high-level semantic information \(X_{cat}\) was obtained through modal fusion, as shown in Equation (10):  
\[X_{cat}=\alpha_{1}\widetilde{X}_{cat}^{SA}+\alpha_{2}X_{img}^{SA}+\alpha_{3}X_{ audio}^{SA}. \tag{10}\]  
In order to effectively complete the task of target classification, Softmax function is used to normalize and achieve the probability distribution of classification. Sofatamx's formula is shown in (11):  
\[P(x_{i})=\frac{e^{x_{i}}}{\sum_{j-1}^{n}e^{x_{i}}}. \tag{11}\]  
The audiovisual fusion network proposed in this paper can gradually enhance the matching degree and fusion effect between sound visual features. In the iterative process, the network adjusts and optimizes the feature extraction and fusion methods of each mode according to the current feature representation and interaction weights, so as to obtain the high-level semantic information of the final fusion. This cross-modal interaction and fusion method not only improves the robustness and generalization ability of feature representation but also solves the problem of data heterogeneity among modes. It makes full use of the complementarity between different modes and provides strong support for the subsequent classification output.