# Phish Responder: A Hybrid Machine Learning Approach to Detect Phishing and Spam Emails
## 3 Materials and Methods
### Architectural Experimentation
#### 3.3.2 Deep Learning  
The aim of the second experiment was to establish whether Keras or PyTorch was the most suitable Python library for deep learning. The UCI Spambase dataset [22] was used as it provided a dataset ideal for binary classification. Figure 2 outlines a basic design of the deep learning model that was used for the deep learning experiment. To establish whether Keras or PyTorch was the more suitable deep learning library, accuracy and precision were the performance metrics considered (see Table 4). The opportunity to identify the optimal parameters for this research arose when experimenting with Keras (see Figure 3). \begin{table}
\begin{tabular}{c c} \hline \hline
**Metric** & **How Was It Measured?** \\ \hline Accuracy & How accurately the detection solution detects \\  & phishing/spam emails. \\  & Added the start and end time to the detection \\ Speed & solution to understand the time it takes to \\  & detect phishing/spam emails. \\  & Determine the proportion of emails that are \\ Precision & phishing/spam to those that were detected as \\  & phishing/spam [6]. \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Performance metrics. Figure 2: Deep learning model design. _Binary_crossentropy_ was the loss function used because [PERSON], [PERSON] and [PERSON] [33] state that it works best for binary classification problems such as phishing detection. Figure 4 outlines various optimizers which produced accuracies higher than 90%. It was expected that the _nadam_ optimizer would be used as [PERSON], [PERSON] and [PERSON] [34] achieve a high accuracy with this optimizer; however, during the Keras experiment, the nadam optimizer achieved the highest accuracy. [PERSON] et al. [35] in their research into fraud detection (a binary classification problem) used the _nadam_ optimizer. Relu and sigmoid were the activation functions considered, as when combined with the optimizers illustrated in Figure 3, high accuracies on average were achieved (see Table 5 below). Figure 4: Keras optimizers. Figure 3: Optimal parameters. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Optimizers** & **Input** & **Hidden Layers (\(\times\)7)** & **Output (\(\times\)2)** &
\begin{tabular}{c} **Average Accuracy (from 5 Runs)** \\ \end{tabular} \\ \hline Nadam & Relu & Relu & Sigmoid & 86\% \\ Adam & Relu & Relu & Sigmoid & 81\% \\ adam & Sigmoid & Sigmoid & Sigmoid & 94\% \\ Adamax & Sigmoid & Sigmoid & Sigmoid & 94\% \\ Nadam & Sigmoid & Sigmoid & Sigmoid & 94\% \\ RMSprop & Sigmoid & Sigmoid & Sigmoid & 94\% \\ Nadam & Relu & Sigmoid & Sigmoid & 95\% \\ RMSprop & Relu & Sigmoid & Sigmoid & 94\% \\ adam & Relu & Sigmoid & Sigmoid & 94\% \\ adamax & Relu & Sigmoid & Sigmoid & 93\% \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Optimizers and activation functions. [PERSON] et al. [32] found that relu was the most effective activation function but they used sigmoid for the end Dense layer as it performs well for binary classification. Although [PERSON] et al. [36] achieved 92% for classifying phishing URLs when using relu for the input and hidden layers and sigmoid for the output layer, these results were not replicated in this research. Using the relu activation function for the input and hidden layers produced a 60% accuracy which is not suitable for phishing detection. Therefore, the green row in Table 3 illustrates the optimal parameters for the model used during the experimentation phase of this research. After choosing the activation functions, the batch size was tested. [PERSON] et al. [37] illustrate that a batch size of 32 is optimal for all deep learning algorithms, hence a batch size of 32 was used. However, to clarify that this batch size was optimal, a batch size of 10 and 64 were also tested as [PERSON] et al. [38] and [PERSON] et al. [30] both used a batch size of 10 for classifying phishing websites. Over five runs, a batch size of 32 produced an average accuracy of 93.78%, whereas a batch size of 64, although faster, achieved an average accuracy of 92.93%.