## 5 Discussion
### Ablation Study
#### 5.1.2 Multimodal Fusion  
We compared four fusion strategies: concatenate, multi-head attention, Transformer, and Conformer. The evaluation of these fusion strategies was conducted using Acc. (accuracy) and F1 score. The results are presented in Table 3.  
Based on the overall trend observed in Table 3, Conformer emerged as the most effective fusion method, followed by Transformer, multi-head attention, and concatenate. Specifically, the Uni2 Mul-M-Conformer model achieved an Acc. (IUAs) that was 0.22, 0.44, and 1.76 percentage points higher than those of Uni2 Mul-M-Transformer, Uni2 Mul-M-Attention, and Uni2 Mul-M-Concatenate, respectively.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline
**Modality** & **Name of Model** & **Feature** & **Acc. (IMAs)** & **F1 (IMAs)** & **Acc. (IUAs)** & **F1 (IUAs)** \\ \hline \multirow{8}{*}{\(V\)} & CNN & Image & 48.36 & 44.56 & 45.08 & 44.18 \\  & CNN & CLIP & 63.68 & 63.11 & 66.96 & 65.41 \\  & Image & 54.49 & 38.67 & 51.20 & 35.06 \\  & CNN-LSTM & CLIP & 67.40 & 62.05 & 68.05 & 67.58 \\  & CNN-ATTN-LSTM & Image & 54.27 & 38.18 & 51.20 & 34.68 \\  & CILIP & 65.43 & 59.97 & 67.83 & 65.60 \\ \hline \multirow{8}{*}{\(A\)} & BiLSTM & Mel & 51.20 & 44.35 & 50.77 & 42.14 \\  & Wave2 Vec & 52.52 & 44.94 & 50.98 & 46.17 \\  & CNN-BiLSTM & Mel & 51.42 & 42.70 & 51.20 & 42.27 \\  & CNN-ATTN-BiLSTM & Wave2 Vec & 53.61 & 41.62 & 53.17 & 46.84 \\  & CNN-ATTN-BiLSTM & Mel & 52.95 & 43.24 & 49.45 & 44.36 \\  & Wave2 Vec & 54.27 & 40.48 & 53.17 & 50.06 \\ \hline \multirow{8}{*}{\(T\)} & BiLSTM & Word2 Vec & 54.70 & 43.98 & 53.39 & 44.53 \\  & BERT & 69.80 & 65.64 & 74.40 & 73.18 \\ \cline{1-1}  & ATTN-BiLSTM & Word2 Vec & 54.92 & 43.16 & 54.05 & 44.62 \\ \cline{1-1}  & BERT & 70.24 & 65.96 & 74.62 & 74.41 \\ \cline{1-1}  & CNN-ATTN & Word2 Vec & 55.36 & 42.40 & 53.39 & 45.95 \\ \cline{1-1} \cline{2-6}  & BERT & 70.02 & 66.87 & 75.27 & 75.10 \\ \hline \end{tabular}
\end{table}
Table 2: Performance of unimodal models in stage one. “ATTN” in the model names represents attention mechanism. Mel is the Mel spectrogram feature.  
In most cases, Acc. (IUAs) outperformed Acc. (IMAs), and F1 (IUAs) outperformed F1 (IMAs). The Uni2 Mul-M-Conformer model demonstrated an Acc. (IUAs) that was 3.72 percentage points higher than that of Acc. (IMAs), while F1 (IUAs) was 3.58 percentage points higher than that of F1 (IMAs). These results align precisely with our expectations.  
Furthermore, the multimodal fusion models constructed using pre-trained unimodal models exhibited higher Acc. and F1 score compared to those built without pre-training. Additionally, the multi-task framework outperformed the single-task framework, which is consistent with previous research findings.  
Figure 5: Performance comparison chart of unimodal models in stage one. (**a**–**c**) depict the model performance metrics, Acc. and F1 score, of three visual models: CNN, CNN-LSTM, and CNN-ATTN-LSTM. (**d**–**f**) depict the model performance metrics, Acc. and F1 score, of three acoustic models: BiLSTM, CNN-BiLSTM, and CNN-ATTN-BiLSTM. (**g**–**i**) depict the model performance metrics, Acc. and F1 score, of three textual models: BiLSTM, ATTN-BiLSTM, and CNN-ATTN. \(V\): Vision, \(A\): Audio, \(T\): Text.