# Providing Email Privacy by Preventing Webmail from Loading Malicious XSS Payloads
## 5 Experiment
### Experiment and Result Analysis  
In the first experiment, we used a Bidirectional-RNN to train the dataset and compare it with traditional machine learning algorithms. We set the hidden layer number of RNN to 64, the epochs to 200, the batch_size to 72, the loss function to \"mae\", the optimization function to \"adam\", and other parameters to default. After training, we used the test-dataset to evaluate the model, plotted the ROC curve and calculated the AUC value. We selected the SVM algorithm, GMM (Gaussian Mixed Model) algorithm, Naive Bayes algorithm, and Random Forest algorithm for comparison. The experimental results are shown in Figure 6.  
From Figure 6, we saw that the AUC value of the Bidirectional-RNN reaches 0.9979, which is much higher than other traditional machine learning algorithms. Only the random forest who is a multi-classifier can be similar to AUC value to the Bidirectional-RNN algorithm. It showed that the integrated learning algorithm and the deep learning algorithm were better than the traditional machine learning algorithm when training the email XSS dataset. This was because the deep learning algorithm and the random forest algorithm were more efficient in working under a high-dimensional dataset with higher accuracy. A more detailed assessment is shown in Table 8.  
In the second experiment, we attempted to add the Attention mechanism to the Bidirectional-RNN algorithm to compare whether it can improve the performance of the original model. First, we use the Bidirectional-RNN algorithm to configure two identical neural networks. The parameters of network 1 are the same as the previous experiment, and network 2 adds a layer of  
Figure 6: Bidirectional-Recurrent Neural Network (RNN) compared with traditional machine learning in ROC curve.  
on network 1. We used all the collected email XSS dataset to train the two neural networks, and the training results are shown in Figure 7.  
It can be analyzed from Figure 7 that the accuracy of the two neural networks gradually stabilizes after the number of training epoch reaches 100. The highest accuracy of the neural network directly trained by the Bidirectional-RNN is up to 0.9828, and its average value is 0.98. The highest accurate value of the neural network added the attention mechanism is up to 0.9936, and its average value is 0.99. It shows that the Attention mechanism optimizes the process of Bidirectional-RNN training, increases its accuracy limit and has better performance. At the same time, we evaluated the loss of two neural networks during the training process, and the results are shown in Figure 8.  
It can be seen from Figure 8 that the loss of the two neural networks gradually stabilizes after the training epoch reaches 100. The loss of the neural network directly trained by the Bidirectional-RNN is about 0.1, and the loss of the neural network added to the Attention mechanism is about 0.03. It shows  
Figure 8: The loss comparison of RNN and RNN with Attention Mechanism.  
\begin{table}
\begin{tabular}{c c c c} \hline  & **Precision** & **Recall** & **F1-Score** \\ \hline Bidirectional-RNN & 0.99 & 0.98 & 0.99 \\ Random Forest & 0.98 & 0.94 & 0.96 \\ GMM & 0.93 & 0.90 & 0.91 \\ Naive Bayes & 0.83 & 0.94 & 0.88 \\ SVM & 0.84 & 0.86 & 0.85 \\ \hline \end{tabular}
\end{table}
Table 8: More detailed evaluation for Bidirectional-RNN and other Machine learning algorithms.  
Figure 7: The accuracy comparison of RNN and RNN with Attention Mechanism.  
that the Attention mechanism can effectively reduce its loss value during the Bidirectional-RNN training process.  
In the experiments of [PERSON] and [PERSON], the recall rate of the deep learning model reached 98%. Comparing with their experimental results, our experiments proved that the Bidirectional-RNN model is completely available for email XSS detection. After adding the Attention mechanism, the model was optimized, which had higher accuracy and lower loss than the original model.