## 5 Discussion  
### Ablation Study  
#### 5.1.1 Unimodal Representation  
To validate our hypotheses, we conducted the following experiments to examine the impact of different feature extraction methods and neural network structures on the performance of unimodal emotion classification models. We trained individual models for each modality, utilizing two types of annotations: IMAs and IUAs. The evaluation of these models was based on the Acc. and F1 score. The results are shown in Table 2. To compare the performance differences more clearly between IMA and IUA models, we plotted Figure 5 based on the data in Table 2.  
In the visual modality, we compared our unimodal model, the CNN-LSTM-based model, with the CNN-based and CNN-ATTN-LSTM-based models. Two types of features were tested for each model: video images and the outputs of CLIP. As presented in Table 2 and Figure 5a-c, the Acc. (IUAs) consistently outperformed Acc. (IMAs), and our CNN-LSTM-based model yielded the highest performance among the three models. Specifically, the CNN-LSTM-based model demonstrated a 0.65 percentage point improvement in Acc. (IUAs) and a 5.53 percentage point improvement in F1 (IUAs) when compared to Acc. (IMAs) and F1 (IMAs). Furthermore, CLIP proved to be an effective method for visual feature extraction. Taking the CNN-LSTM model trained using IUAs as an example, employing the output of CLIP as the input feature resulted in a 16.85 percentage point increase in Acc. and a 32.52 percentage point increase in F1 score, compared to using images as the input. This improvement can be attributed to the ability of CLIP to leverage \begin{table}
\begin{tabular}{c c} \hline \hline
**Model** & **Acc.** \\ \hline \multirow{8}{*}{baseline} & EF-LSTM [74] & 51.73 \\  & MFN [75] & 63.89 \\  & MULT [76] & 65.03 \\  & LF-DNN [13] & 66.91 \\  & MLF-DNN * [13] & 69.06 \\  & LMF [77] & 64.38 \\  & MLMF * [13] & 67.70 \\  & TFN [78] & 64.46 \\  & MTFN * [13] & 69.02 \\ \hline \multirow{4}{*}{ours} & Uni2 Mul-S-Conformer (w/o pre-train) & 72.21 \\  & Uni2 Mul-S-Conformer & 76.15 \\  & Uni2 Mul-M-Conformer (w/o pre-train) & 73.30 \\  & Uni2 Mul-M-Conformer & 76.81 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The results of the baseline and our models on the CH-SIMS dataset. The models with * are multi-task models, extended from single-task models by introducing independent unimodal annotations. “S” in model names stands for “Single-task”, while “M” denotes “multi-task”. The model names with “w/o pre-train” indicate that the corresponding unimodal models have not been pre-trained, while the ones without “w/o pre-train” indicate that the unimodal models have been pre-trained. “Conformer” in model names means the fusion method of the models. prior knowledge of both vision and text, allowing it to extract textual information along with visual information. For the acoustic modality, we compared our unimodal model, the CNN-ATTN-BiLSTM-based model, with the BiLSTM-based and CNN-BiLSTM-based models. Two types of features were examined for each model: Mel features and outputs of Wav2 vec. As indicated in Table 2 and Figure 5d-f, most models exhibited higher F1 score for IUAs compared to IMAs, and our CNN-ATTN-BiLSTM-based model delivered the best performance among the three models. Additionally, Wav2 Vec 2.0 proved to be an effective method for acoustic feature extraction. Although the CNN-ATTN-BiLSTM-based model had a 1.1 percentage point lower Acc. (IUAs) compared to Acc. (IMAs), it demonstrated a 9.58 percentage point higher F1 score for IUAs compared to F1 score for IMAs. In the textual modality, we compared our unimodal model, the CNN-ATTN-based model, with the BiLSTM-based and ATTN-BiLSTM-based models. Two types of features were tested for each model: outputs of Word2 Vec and BERT. As depicted in Table 2 and Figure 5g-i, in most cases, the F1 score for IUAs outperformed the F1 score for IMAs, and in some instances, the Acc. (IUAs) surpassed Acc. (IMAs). Our CNN-ATTN-based model yielded the best performance among the three models.