# MMTD: A Multilingual and Multimodal Spam Detection Model Combining Text and Document Images
## 5 Results
### Experimental Results  
This section presents a comprehensive comparative analysis of the experimental results obtained from four distinct methods. Detailed hyperparameter configurations for these models are provided in Table 3. Notably, hyperparameter selection hinges on hardware constraints, with the GPU memory capacity influencing the choice of batch size. Furthermore, we adhered to the default settings recommended by a hugging face for learning rate scheduling, warm-up steps, and learning rate decay policies. Additionally, we meticulously determined the number of epochs based on the performance of the models on the test dataset. Table 4 shows the experimental results, Table 5 shows the confusion matrix, and Figure 8 shows the ROC curve of models, respectively. The larger the value of the diagonal element in the confusion matrix, the better the performance of the method. From Tables 4 and 5 and Figure 8, the following observations can be drawn:  
1. Despite using the same modal fusion strategy, MMTD achieves better performance than CLIP. This suggests that the multilingual text encoder and document image encoder employed in MMTD are more effective in feature extraction for spam detection. The improved performance of MMTD indicates the significance of utilizing specialized encoders for specific tasks. 2. Vilt, while emphasizing feature fusion, overlooks the importance of early-stage feature extraction from text and images. As a result, MMTD outperforms Vilt in terms of performance. This highlights the advantage of MMTD's holistic approach that combines effective feature extraction with robust fusion techniques. 3. Compared with MMA-MF, because the LSTM text encoder used by MMA-MF cannot handle multilingual messages well, and directly using CNN as an image encoder is not very effective in distinguishing document images, its effect is worse than the other three methods. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Fold** & **Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline \multirow{6}{*}{CLIP} & 1 & 0.954 & 0.950 & 0.950 & 0.950 \\  & 2 & 0.941 & 0.940 & 0.940 & 0.940 \\  & 3 & 0.944 & 0.940 & 0.940 & 0.940 \\  & 4 & 0.954 & 0.950 & 0.950 & 0.950 \\  & 5 & 0.953 & 0.950 & 0.950 & 0.950 \\  & **mean** & **0.949** & **0.946** & **0.946** & **0.946** \\ \hline \multirow{6}{*}{Vilt} & 1 & 0.942 & 0.940 & 0.940 & 0.940 \\  & 2 & 0.952 & 0.950 & 0.950 & 0.950 \\  & 3 & 0.948 & 0.950 & 0.950 & 0.950 \\  & 4 & 0.942 & 0.940 & 0.940 & 0.940 \\  & 5 & 0.954 & 0.950 & 0.950 & 0.950 \\  & **mean** & **0.948** & **0.946** & **0.946** & **0.946** \\ \hline \multirow{6}{*}{MMA-MF} & 1 & 0.930 & 0.930 & 0.930 & 0.930 \\  & 2 & 0.769 & 0.770 & 0.770 & 0.770 \\ \cline{1-1}  & 3 & 0.889 & 0.890 & 0.890 & 0.890 \\ \cline{1-1}  & 4 & 0.938 & 0.940 & 0.940 & 0.940 \\ \cline{1-1}  & 5 & 0.936 & 0.940 & 0.940 & 0.940 \\ \cline{1-1}  & **mean** & **0.892** & **0.894** & **0.894** & **0.894** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Experimental results. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Models** & **Epoch** & **Batch Size** & **Optimizer** & **Learning Rate** \\ \hline MMTD & 3 & 40 & AdamW & 0.0005 \\ CLIP & 10 & 16 & AdamW & 0.00002 \\ Vilt & 10 & 16 & AdamW & 0.00002 \\ MMA-MF & 15 & 32 & AdamW & 0.001 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Hyperparameter settings. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Model** & **Fold** & **Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline \multirow{6}{*}{MMTD} & 1 & 1.000 & 1.000 & 1.000 & 1.000 \\  & 2 & 0.992 & 0.990 & 0.990 & 0.990 \\  & 3 & 0.999 & 1.000 & 1.000 & 1.000 \\  & 4 & 0.999 & 1.000 & 1.000 & 1.000 \\  & 5 & 1.000 & 1.000 & 1.000 & 1.000 \\  & **mean** & **0.998** & **0.998** & **0.998** & **0.998** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: _Cont._  
Figure 8: ROC curve of models.