# A Spam Filtering Method Based on Multi-Modal Fusion
## 1 Introduction
[PERSON] [15] proposed a multi-feature combination method to design a spam filtering system, it extracts the texture features and attribute features of the image part in an email separately, and then classifies them by the SVM method to get two classification probability values, finally entering them into the model designed by the SVM method again to get a real value of the email as spam. These methods can deal with image spam, while it can't handle text spam and hybrid spam. For multi-modal spam detection, [PERSON] [16] proposed a method based on multi-modal feature fusion. The main idea is to use the P-SVM method to construct multi-classifiers for the text and image data in an email respectively, and then adopts the SVM method to fuse the output of the multi-classifier to get a true value of the email as spam. This method can handle with text-based spam, image-based spam and hybrid spam, but the experimental results show that the identifying accuracy of spam is only 90%. Recently, deep learning techniques have been widely applied and obtained huge success in many application domains. Particularly, the convolutional neural network (CNN) has higher accuracy for image classification tasks than other methods [17], and the Long Short-Term Memory (LSTM) neural network has been widely used in natural language processing due to its time and memory characteristics [18]. It inspires us to adopt the CNN and LSTM model to deal with the image and text data for hybrid emails, respectively. Keeping the research line based on multi-modal spam detection, in this paper, we design a multi-modal fusion model for spam detection, which called MMA-MF. For the text part of an email, we use an LSTM model to extract textual semantic relations features and obtain classification probability value of the text part as spam. For the image part of the same email, we construct an CNN model to get classification probability value of the image part as spam. From the two models, we acquire two classification probability values. Then, the two classification probability values are fed into a fusion model designed by the logistic regression method to acquire the eventual classification probability value, which describes the true value of the email as spam. If an email just only has text data or image data, we use dropout cogitation [19] to set the classification probability value of the LSTM model output or the CNN model output become 0.5 to ensure that the MMF-MA model could not only handle text spam and image spam, but also handle hybrid spam. In addition, the grid search optimization algorithm [20] is used to select the training hyperparameters of the model, and k-fold cross-validation method [21] is employed to verify the MMA-MF model performance. The proposed method merges the text and image information in an email, so it can efficiently filter spam whether the junk information is hidden in the text or in the image. That is, the advantage of the MMA-MF model is that it can not only filter hybrid spam, but also filter spam with only text data or image data. The experimental results indicate that our method is better than other methods significantly. The main contribution is that we apply the CNN and LSTM model to handle the image and text data in an email, and combine them into a fusion model by the logistic regression method. To our best knowledge, we firstly shed light on this approach in the email filtering systems. The rest of this paper is organized as follows: Section 2 describes the architecture of the MMA-MF model, we present the design framework of the CNN, LSTM and fusion model, the brief categorization algorithm for text spam and image spam. Section 3 presents evaluation metrics and validation schemes. Section 4 is about experimental results and discussion. In the end, conclusions are given in Section 5.