# A Spam Filtering Method Based on Multi-Modal Fusion
## 4 Experimental Results and Discussion
### Results and Discussion  
In this section, we show our evaluation results on image spam classification, text spam classification and the mixed spam classification. Moreover, we give some analysis and discussions for the experimental results. We use 5-fold cross-validation method to verify the performance of the MMA-MF model on the text dataset, image dataset and the mixed datasets 1 and 2, and obtain the experimental results of the MMA-MF model on the four datasets, as shown in Table 7, in which \(\overline{u}\) means the average value of Accuracy, Recall, F1-Score or Precision after using the 5-fold cross-validation method. From Table 7, we can conclude that the MMA-MF model designed in this paper implements the filtering function of spam, whether it is hidden in the text, in the image, or hidden in the text and image, we are all able to handle it and filter it out pretty well. In conclusion, we have the following observations: for the MMA-MF model, it not only filters well mixed emails, but also filters text emails or image-based emails well. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Type** & **Training Dataset Size** & **Testing Dataset Size** \\ \hline Text & 5000 & 1000 \\ Image & 2322 & 464 \\ Mixed Dataset 1 & 960 & 240 \\ Mixed Dataset 2 & \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Training and testing dataset size. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Fold** & **Accuracy** & **Recall** & **F1-Score** & **Precision** \\ \hline \multicolumn{5}{c}{MMA-MF Model for Text Dataset} \\  
\(\overline{u}\) & 0.9846 & 0.9852 & 0.9845 & 0.9830 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Experimental results in 5-fold cross-validation for the MMA-MF model. In order to further verify the performance of the MMA-MF model, we compare it with a large set of well-performed models by using the four datasets. For the text dataset, we use Character-level CNN (Char-CNN) [31], BiLSTM (Bi-directional LSTM) [31], Naive Bayes, the Immune Cross-Regulation Model (ICRM) [11], and the MMA-MF model for comparison. For the image dataset, we use Naive Bayes, ID3 Decision Tree and the MMA-MF model for comparison. For the mixed dataset 1 and 2, we use [PERSON]'s idea of incorporating the k-NN algorithm into a model [32] to change the fusion model part of the MMA-MF model. Besides using the fusion model designed by the MMA-MF model, we also use a classical k-nearest neighbors (classical k-NN) model, an enhanced k-nearest neighbors (enhanced k-NN) model, which assigns email labels to each email based on the centroid displacement of the classes present in the neighboring points, a SVM model and a random forest model to replace the fusion model designed by the MMA-MF model to prove that the fusion model designed by the MMA-MF model is better than other models. At the same time, the hyperparameters for the classical k-NN, enhanced k-NN, SVM and random forest models are also using the grid search optimization algorithm to select the best hyperparameters for the four models. For instance, through the grid search optimization algorithm, the best hyperparameters C and gamma of the SVM model are equal to 1 and 0.001, respectively, and the best hyperparameter K of the classical k-NN model is equal to 1, etc. From Table 8, we can see that, for the text and image datasets, the MMA-MF model achieves the best performance compared to other models. For the hybrid datasets 1 and 2, the fusion model designed by the MMA-MF model achieves the best performance compared to the classical k-NN, enhanced k-NN, SVM and random forest models. Overall, the MMA-MF model achieves the best performance in the four datasets. An ROC (receiver operating characteristic) chart, which can be considered as the average value of the sensitivity for a test over all possible values of specificity or vice versa. The ROC chart has the largest area under the curve (AUC) and is considered to be optimal.