# EFCMF: A Multimodal Robustness Enhancement Framework for Fine-Grained Recognition
## 3 Experiment
### Experiment Results
#### 3.2.2 Performance of the Model under Different Hyperparameters
The parameter \(p\) controls the probability of training the model with visual modal data in the case of missing modality. A large or small probability \(p\) will result in one of the modality feature extraction modules that cannot be trained effectively. Therefore, the value \(q=0.4\) is suitable to meet the theoretical expectation. Table 8 and 9 shows the specific values of Figure 6. Tables show that the maximum number of maximum points for accuracy exists at \(q=0.4\) under the four adversarial examples attacks. The reason the accuracy maximum point appears at \(q=0.4\) when facing the adversarial example attack is that when the visual modality input data of the model is severely attacked, the unattacked linguistic modality features can describe the target better at this time. Lowering the \(p\)-value and making the model more biased to use the features of linguistic modality can improve some accuracy. Moreover, letting \(q=0\) and \(q=1\) both lead to a significant drop in the model's accuracy in the face of modal deficits, which suggests that when training with the random modal deactivation method, it is best not to have the model extremely biased towards training a single modality. A proper bias towards training linguistic modalities enables the model to learn better standard features. Figure 6: The accuracy trend of the visual modal integrity probability \(q\). (**a**) Birds. (**b**) Flowers. (3) Analysis of the impact of pre-training on the model  
The above experiments reveal that the model is more challenging to update the weights of the linguistic feature extraction module without the modal random deactivation method. After training with random modal deactivation, the model's accuracy in the face of language modality still needs to meet the requirements. For this reason, the number of training iterations required for each feature extraction module during the training of the multimodal model is not consistent. The feature extraction module of the visual modality is often already pre-trained on a large dataset, so the feature extraction module of the linguistic modality requires more iterations for training. Experiments were done with \(p=0.8\) and \(q=0.4\) to verify the above idea. Table 10 shows the experimental results, where pre-trained indicates that the language modality of the model has been pre-trained. Table 10 shows that when the language feature extraction module is trained, the model's accuracy is substantially improved due to visual modality deficiency. At the same time, it can achieve some improvement in the face of the adversarial sample attacks of FGSM, BIM, and C&W. This indicates that the standard features of the multimodal fine-grained recognition model based on feature fusion are more likely to be biased to represent modal features that have been pre-trained. Therefore, it is desirable to pre-train each feature extraction module of the multimodal model to ensure that the standard features do not tend to represent a particular modality more often. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline
**Dataset** & \(p\) & \(q\) & **Pretrained** & **Accuracy** & **FGSM** & **BIM** & **PGD** & **C\&W** & **Vision** & **Language** \\ \hline Birds & 0.8 & 0.4 & False & 0.9105 & 0.3639 & 0.2152 & **0.0811** & 0.3639 & 0.1775 & **0.9138** \\  & **0.8** & **0.4** & **True** & **0.9114** & **0.3826** & **0.2231** & 0.0642 & **0.5730** & **0.5051** & 0.9099 \\ \hline Flowers & 0.8 & 0.4 & False & **0.9951** & 0.8451 & 0.7118 & **0.4412** & 0.9098 & 0.0382 & **0.9961** \\  & **0.8** & **0.4** & **True** & 0.9931 & **0.8882** & **0.7656** & 0.3803 & **0.9509** & **0.6441** & 0.9941 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: The effect of pre-training of the language feature extraction module on various aspects of the model.