# Leveraging Adversarial Samples for Enhanced Classification of Malicious and Evasive PDF Files
## 4 Results and Discussion
### Training and Validation Experiments
#### 4.1.4 Experiment 3  
In this experiment, we train a multi-label model that detects evasion and maliciousness concurrently. We test multiple models to select the best one and fine-tune it. Again, we start by using the algorithms with their default hyperparameters, then after selecting the best model, we fine-tune it accordingly. The accuracy, macro-precision, macro-recall, and macro-F1 score for each of the models are summarized in Table 5.  
We can see that the Logistic Regression, SVM and Naive Bayes models achieve a lower performance compared to Random Forest, AdaBoost, and LightGBM. This is expected after the results we obtained in Experiment 2, which showed the weakness of these models to detect evasion. We can clearly see that the Random Forest and LightGBM models outperformed the others. This is why, we fine-tune both and we achieve the best performance (accuracy = 96.23%, macro-precision = 98.89%, macro-recall = 97.21%, macro-f1-score = 98.04%) using the LightGBM algorithm with the hyperparameters as mentioned in the Experimental Setup section.  
The results associated with each of our approaches for malware prediction and evasion prediction are summarized in Table 6 for easier comparison. We also include the performance metrics of the baseline when tested against the mix of evasive and non-evasive validation sets.  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & \begin{tabular}{c} **Logistic** \\ **Regression** \\ \end{tabular} & **SVM (RBF)** & \begin{tabular}{c} **SVM** \\ **(Poly 1)** \\ \end{tabular} & \
\begin{tabular}{c} **SVM** \\ **(Poly 2)** \\ \end{tabular} & **Naïve Bayes** &
\begin{tabular}{c} **Random** \\ **Forest** \\ \end{tabular} & **AdaBoost** & **LightGBM** \\ \hline Accuracy & 76.53\% & 80.59\% & 77.12\% & 78.75\% & 73.97\% & 96.28\% & 84.48\% & 95.19\% \\ TNR & 93.34\% & 98.18\% & 97.70\% & 96.88\% & 83.10\% & 97.82\% & 90.32\% & 97.82\% \\ FPR & 6.66\% & 1.82\% & 2.30\% & 3.12\% & 16.90\% & 2.18\% & 7.99\% & 2.18\% \\ FNR & 40.78\% & 37.51\% & 44.07\% & 39.91\% & 35.42\% & 5.31\% & 23.28\% & 7.53\% \\ TPR & 59.22\% & 62.49\% & 55.93\% & 60.09\% & 64.58\% & 94.69\% & 76.72\% & 92.47\% \\ F1 Score & 71.31\% & 76.04\% & 70.67\% & 73.60\% & 70.98\% & 96.17\% & 82.86\% & 94.98\% \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance of the Evasion Classifiers.  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & \begin{tabular}{c} **Logistic** \\ **Regression** \\ \end{tabular} & **SVM (RBF)** & \begin{tabular}{c} **SVM** \\ **(Poly 1)** \\ \end{tabular} & \
\begin{tabular}{c} **SVM** \\ **(Poly 2)** \\ \end{tabular} & **Naïve Bayes** &
\begin{tabular}{c} **Random** \\ **Forest** \\ \end{tabular} & **AdaBoost** & **LightGBM** \\ \hline Accuracy & 66.37\% & 79.01\% & 69.26\% & 69.55\% & 51.29\% & 95.87\% & 82.66\% & 94.77\% \\ Macro-Precision & 85.04\% & 97.38\% & 90.91\% & 89.54\% & 77.74\% & 98.71\% & 95.24\% & 98.65\% \\ Macro-Recall & 78.57\% & 79.71\% & 77.02\% & 79.17\% & 78.91\% & 96.95\% & 86.32\% & 95.90\% \\ Macro-F1 Score & 79.82\% & 86.67\% & 81.13\% & 82.13\% & 74.09\% & 97.82\% & 90.34\% & 97.24\% \\ \hline \hline \end{tabular}
\end{table}
Table 5: Performance of each multilabel model to detect evasion and maliciousness.