# A Hybrid CNN-LSTM Model for SMS Spam Detection in Arabic and English Messages
## 3 The Proposed Model
### Machine Learning Modeling
#### 3.3.2 Deep Learning  
In this paragraph, we describe the deep learning-based model for detecting spams from a collection of Arabic and English text messages. To implement this model, we tried three different deep learning architectures. Firstly, we created an architecture using the CNN (Convolutional Neural Network) method. Secondly, we tried a model based on LSTM (Long Short-Term Memory). Finally, we implemented a hybrid model combining the two previous methods CNN and LSTM. Based on the experimental results that we will explain in the next section, the hybrid model gave us the best results in terms of performance. For this reason, here we will limit ourselves to the description of the CNN-LSTM hybrid model. In Figure 3 and Algorithm 1, we describe the architecture and operating principle of the CNN-LSTM model. The output of the Embedding layer, as described above, is connected with a CNN layer with a type \"Convolution 1D\". Subsequently, a MaxPooling layer is applied to reduce the dimensionality of the CNN output. Next, we connect an LSTM layer. Finally, we apply a Dense output layer with a single neuron and a sigmoid activation function to make decision for the two classes spam and not-spam. In the following, we describe, in detail, the role and configuration of each layer. ```
Input:\(M\) (A set of \(n\) text messages, \(M=M_{1},M_{2},...,M_{n}\) ) Output:\(Y\) (Message label : 0 or 1) foreachmessage \(M_{i}\) in \(M\)do \(V_{i}=\text{Word\_Embedding}(M_{i})\)  end foreach\(V_{i}\)do \(C_{i}=\text{CNN}(V_{i})\)  end foreach\(C_{i}\)do \(O_{i}=\text{LSTM}(C_{i})\)  end foreach\(O_{i}\)do \(Y_{i}=\text{\emph{sigmoid}}(O_{i})\) // Dense layer with sigmoid function  end foreach* **Convolution 1D** The main interest of the Convolution layer is to extract relevant features from the text data. This is done by applying the convolution operation on the word vectors generated by the Word Embedding layer. The convolution task that we present in this paragraph is based on the work that was presented in [19]. Let \(x_{i}\in\mathbb{R}^{d}\) be the d-dimensional word vector corresponding to the \(i\)-th word in the message. Let \(x\in\mathbb{R}^{L\times d}\) denote the input message, where \(L\) is the length of the message. For each position \(j\) in the message, consider a window vector \(w_{j}\) with \(k\) consecutive word vectors, represented as: \[w_{j}=[x_{j},x_{j+1},\ldots,x_{j+k-1}]\] (2)

A convolution operation involves a filter \(p\in\mathbb{R}^{k\times d}\), which is applied to the window \(w\) to produce a new feature map \(c\in\mathbb{R}^{L-k+1}\). Each feature element \(c_{j}\) for window vector \(w_{j}\) is calculated, as follows:

\[c_{j}=f(w_{j}\odot p+b) \tag{3}\]

where \(\odot\) is element-wise product, \(b\in\mathbb{R}\) is a bias term, and \(f\) is a nonlinear function. In our case, we used Rectified linear unit (ReLU) as nonlinear function. It is defined as:

\[f(x)=max(0,x) \tag{4}\]

ReLU activation function returns \(x\) if the value is positive, elsewhere returns 0. For the configuration of the convolution layer, we used a one-dimensional convolution associated with a filter window \(k=3\). Algorithm 2 describes the detailed working of the CNN algorithm. Figure 3: Architecture of the CNN-LSTM model. * **MaxPooling** The feature maps that are generated by the Convolution operation are characterized by a high-level vector representation. To reduce this representation, we added a MaxPooling layer after the Convolution layer to help select only important information by removing weak activation information. This is useful to avoid overfitting due to noisy text. * **Lstm** CNN is very useful in extracting relevant features from the text data. However, it is unable to correlate the current information with the past information. This can be done with another deep learning method, which is \"LSTM\". LSTM (Long short-term memory) is a kind of RNN architecture that is capable of learning long-term dependencies. The architecture of LSTM contains a range of repeated units for each time step.