## 4 Results and Discussion
### Results
The TREC_05 dataset, however, demonstrates the best balance between precision and recall, with the slightest difference between these metrics. Appendices A and B provide detailed experimental results for all datasets and models. Regarding RQ3, both ML and DL models have their respective strengths and weaknesses. Although DL models like BERT and RoBERTa offer superior accuracy, their high \begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **Merged** & **Balanced Merged** \\ \hline BERT & 0.0364 & 0.0329 \\ \hline DistilBERT & 0.0291 & 0.0258 \\ \hline RoBERTa & 0.0379 & 0.0340 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Prediction Time (in Seconds) of Transformer-based Models for Merged and Balanced Merged Datasets. \begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline
**Model** & **Enron** & **SpamAssassian** & **Ling** & **TREC\_05** & **TREC\_06** & **TREC\_07** & **CEAS\_08** & **Nazario\_5** & **Nigerian\_5** & **Merged** & **Balanced** \\ \hline Naive Bayes & 0.4045 & 0.0774 & 0.4096 & 0.7002 & 0.2044 & 0.6891 & 0.5992 & 0.0414 & 0.0639 & 2.7606 & 2.4230 \\ \hline Logistic Regression & 6.0422  
resource requirements make them less feasible for large-scale applications. However, DistilBERT--a more lightweight version of BERT--also delivers strong results, suggesting potential for efficiency improvements. Future work could focus on optimizing these models for practical, real-world deployments. False positives (safe emails incorrectly flagged as phishing, FPs) and false negatives (phishing emails mistakenly classified as safe, FNs) can significantly impact the effectiveness of real-world email security systems. Minimizing these errors is essential to ensure both security and user experience. For best-performing models across all datasets, we have provided the number of FPs (to evaluate the impact on user experience) and FNs (to assess the potential security threats to users). Detailed results are shown in Appendix B. Because the studied datasets span different periods and phishing techniques evolve, older emails may not accurately reflect current phishing strategies. Future work can include a simulation of sophisticated phishing attacks to estimate the models' effectiveness in detecting evasive threats while minimizing FPs. We did not use a dedicated validation set in our experimental setup, and as a result, we do not have a training-validation accuracy and loss plot. Although we did not explicitly use a validation set test, our approach still provides strong evidence of generalization in some models. First, the high test accuracy, especially on the merged dataset, suggests strong generalization since the merged dataset comes from different sources. Models that achieve high accuracy on this dataset tend to generalize well across diverse phishing scenarios. Second, we have tested the top-performing ML and DL models--namely, SGD Classifier, BERT, DistilBERT, and RoBERTa--on an external dataset (phishing validation emails dataset [73]) to validate the models' robustness further. This dataset includes real and synthetic phishing email samples. The results of this experiment are summarized in Table 7. In this experiment, we have trained these models on each of the 11 datasets separately and then evaluated their performance on the external dataset. BERT achieved a test accuracy of 81.30% and 88.80% on the external dataset, which is lower than its performance when tested on the merged-balanced and merged datasets. This discrepancy may be attributed to BERT's Next Sentence Prediction (NSP) mechanism, which is less effective for certain tasks and can limit generalizability. In contrast, RoBERTa, which benefits from extended training and larger datasets, demonstrated superior performance. Therefore, the fact that RoBERTa maintains high accuracy on unseen data supports its ability to generalize beyond the training set.