# Leveraging Adversarial Samples for Enhanced Classification of Malicious and Evasive PDF Files
## 4 Results and Discussion
### Training and Validation Experiments
#### 4.1.3 Experiment 2  
In this experiment, we implement the second approach, and we train and validate each of the models separately. We use the same naming convention we followed in the Methodology section.  
1. [leftmargin=*]
2. Evasion Classifier: The goal of this classifier is to detect whether a PDF sample is evasive or not. Thus, this classifier would be trained on a mix of evasive and non-evasive data. To select a suitable model that can detect evasion, we test multiple algorithms. As an initial step, we keep their default hyperparameters as set by the scikit learn python's library. We train the models on the mix of evasive and non-evasive training sets and then evaluate each of them on the mix of evasive and non-evasive validation sets. The results are summarized in Table 4.  
We can see that not all models perform very well on this data, and this is expected, because detecting evasion is not an easy task with the limited features we have, and this is what explains the low performance of simple models and the good performance of the more powerful models (Random Forest, AdaBoost, and LightGBM) that are based on ensemble techniques. From the results, we can see that the most promising models to detect evasion are the Random Forest Classifier and the LightGBM. We fine-tune both and we achieve the best performance (accuracy = 96.84%, TNR = 97.97%, TPR = 95.69%, and f1-score = 96.76%) using the LightGBM algorithm with the set of hyperparameters as described in the Experimental Setup section.  
Figure 4: Performance of the baseline on the non-evasive and evasive validation datasets.  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & \begin{tabular}{c} **Logistic** \\ **Regression** \\ \end{tabular} & **SVM (RBF)** & \begin{tabular}{c} **SVM** \\ **(Poly 1)** \\ \end{tabular} & \
\begin{tabular}{c} **SVM** \\ **(Poly 2)** \\ \end{tabular} & **Naïve Bayes** &
\begin{tabular}{c} **Random** \\ **Forest** \\ \end{tabular} & **AdaBoost** & **LightGBM** \\ \hline Accuracy & 37.73\% & 44.99\% & 38.53\% & 39.15\% & 34.47\% & 39.66\% & 39.96\% & 66.28\% \\ TNR\({}^{\prime}\) & 80.62\% & 91.81\% & 81.55\% & 87.20\% & 88.70\% & 97.00\% & 94.58\% & 93.19\% \\ FPR & 19.38\% & 8.19\% & 18.45\% & 12.80\% & 11.30\% & 3.00\% & 5.42\% & 6.81\% \\ FNR & 74.09\% & 67.92\% & 73.32\% & 74.09\% & 80.48\% & 76.15\% & 75.10\% & 41.14\% \\ TPR & 25.91\% & 32.08\% & 26.68\% & 25.91\% & 19.52\% & 23.85\% & 24.90\% & 58.86\% \\ F1 Score & 39.48\% & 47.76\% & 40.49\% & 40.03\% & 31.84\% & 38.26\% & 39.40\% & 73.23\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance of the tested models on the evasive validation data.  
2. Evasive Malware Classifier: We trained the same baseline architecture on the evasive training set only. We evaluated this model on the evasive validation set, and the results we obtained are: accuracy = 99.45%, TNR = 99.31%, TPR = 99.49%, and f1-score = 99.65%.
3. Non-evasive Malware Classifier: Here we simply used the baseline that has been trained and evaluated on the non-evasive training set and the non-evasive validation set, respectively.  
After training the models individually, we combine the evasive validation set along with the non-evasive validation set and feed them to the pipeline. The malware detection metrics of the system are the following: accuracy = 99.44%, TNR = 99.45%, TPR = 99.43%, and f1-score = 99.55%. We see a slight decrease in performance compared to the first approach, but this is expected because the Evasion Classifier can make mistakes and accordingly, forward an evasive PDF to the Non-evasive Malware Classifier and vice versa.