## 4 Experiment
### Parameters' Setting  
All experiments are carried out using a single NVIDIA (Santa Clara, CA, USA) Quadro P520 card. We adopt [PERSON] as the optimizer with a learning rate of \(1\times 10^{-4}\), and set the number of epochs to 30 and batch size to 32. All models are trained using sparse categorical cross-entropy on each softmax output. We evaluate the models with accuracy (Acc.) and F1 score. Acc. is equal to the proportion of correctly classified samples to the total number of samples, and can be formulated as follows:  
\[\text{Acc.}=\frac{TP+TN}{TP+FN+TN+FP}, \tag{19}\]  
\(TP\) is true positives, \(TN\) is true negatives, \(FP\) is false positives, and \(FN\) is false negatives.  
Precision is the ability of the classifier not to label as positive a sample that is negative, and recall is the ability of the classifier to find all the positive samples. The F1 score can be interpreted as a weighted harmonic mean of the precision and recall, and can be formulated as follows:  
\[\begin{array}{c}precision=\frac{TP}{TP+FP},\\ recall=\frac{TP+FN}{TP+FN},\\ \text{F}_{1}=\frac{2\times precision\times recall}{precision+recall},\end{array} \tag{20}\]  
Higher values of Acc. and F1 score represent better performance. We use the \"save_best_only = True\" parameter in Keras to save only the best model during training. This parameter ensures that the model weights are saved whenever the monitored metric improves and overwrites the previously saved weights only if there is an improvement. Thus, at the end of training, the saved weights correspond to the best performing model on the validation set.  
In addition to the general parameters mentioned above, personalized parameters for unimodal and multimodal models are set as follows:  
Vision: In the process of selecting the visual timestep, we conduct several pilot experiments to access the impact of different timesteps on the performance of the visual models. We observe that increasing the timestep results in a larger memory footprint for the feature matrix, without a significant improvement in model performance. To strike a balance between computational complexity and performance, we decide to set the visual timestep to 10. The output length of CLIP is 512. Therefore, the input shape of the visual modality is \(10\times 512\). The following three 1D convolution layers have 64 filters, and the kernel sizes are set to 3, 1, and 3, respectively, with strides being set to 1. The subsequent BiLSTM has 32 units.  
Audio: Similarly, in line with our previous findings for the visual timestep, our pilot experiments demonstrate that setting the acoustic timestep to 128 not only maintains a good level of performance but also showcases the effectiveness of this parameter. As a result, we set the acoustic timestep to 128. The output length of Wav2 Vec 2.0 is 512. Thus, the input shape of the acoustic modality is \(128\times 512\). The following 1D convolution layer has 64 filters, and the kernel size and strides are set to 3 and 2, respectively. The subsequent BiLSTM has 32 units, and dropout is used with a rate of 0.5 to prevent overfitting.  
Text: The longest sentence in the CH-SIMS dataset has 36 tokens. The output length of BERT used in this paper is 512. Hence, the input shape of the textual modality is \(36\times 512\). The subsequent two 1D convolution layers have 64 filters, and both the kernel size and strides are set to 1.  
Multimodal: The number of attention heads in our multi-head attention module in Conformer is 2. Dropout is used with a rate of 0.5 to prevent overfitting. The convolution layer in Conformer has 64 filters, and the kernel size is set to 3. The loss weights for multimodal, vision, audio, and text are 0.3, 0.2, 0.2, and 0.3, respectively.