## 5 Results
### Performance Analysis of Adversarial Training with LGBM and Realistic Windows PE Attack Methods
Metrics for which \(LGBM^{AT}\) outperformed (or performed equally to) \(LGBM^{O}\) are underlined. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{8}{c}{**Test Set (O + A)**} \\ \cline{2-9}  & **tg** & **fm** & **fg** & **tm** & **oa** & **F** & **fmr** & **fpr** \\ \hline \(LGBM^{O}\) & 13,432 & 62 & 95 & 21,398 & 0.9955 & 0.9963 & 0.0044 & 0.0046 \\ \(LGBM^{AT}\) (Extend) & 13,423 & 71 & 69 & 21,424 & 0.9960 & 0.9967 & 0.0032 & 0.0053 \\ \hline \(LGBM^{O}\) & 13,432 & 62 & 84 & 19,571 & 0.9956 & 0.9963 & 0.0043 & 0.0046 \\ \(LGBM^{AT}\) (Full DOS) & 13,422 & 72 & 72 & 19,578 & 0.9955 & 0.9962 & 0.0039 & 0.0053 \\ \hline \(LGBM^{O}\) & 13,432 & 62 & 72 & 16,887 & 0.9954 & 0.9959 & 0.0045 & 0.0046 \\ \(LGBM^{AT}\) (Shift) & 13,423 & 71 & 79 & 16,885 & 0.9951 & 0.9956 & 0.0047 & 0.0053 \\ \hline \(LGBM^{O}\) & 13,432 & 62 & 81 & 17,844 & 0.9954 & 0.9960 & 0.0045 & 0.0046 \\ \(LGBM^{AT}\) (FGSM) & 13,430 & 64 & 83 & 17,842 & 0.9953 & 0.9959 & 0.0046 & 0.0047 \\ \hline \(LGBM^{O}\) & 13,432 & 62 & 135 & 20,265 & 0.9942 & 0.9952 & 0.0066 & 0.0046 \\ \(LGBM^{AT}\) (GAMMA) & 13,425 & 69 & 63 & 20,337 & 0.9961 & 0.9968 & 0.0031 & 0.0051 \\ \hline \hline \end{tabular} \({}^{1}\) O denotes the collection of decisions produced for the study’s collection of original Windows PE files, while A denotes the collection of decisions produced for the collection of realistic adversarial malware produced from the study’s original PE malware recorded in O.  
\end{table} 
Table 5: Detailed accuracy metrics (tg—amount of true goodware, fm—amount of false malware, fg—amount of false goodware and tm—amount of true malware), and summary accuracy metrics (oa—overall accuracy, F—Fscore, fmr—the false negative rate and fpr—the false positive rate) of \(LGBM^{O}\) and \(LGBM^{AT}\) measured in the evaluation configuration O + A 1. The evaluation was conducted through the 3-fold CV. Adversarial Windows PE malware files were produced with Extend, Full DOS, Shift, FGSM padding + slack and GAMMA. Metrics for which \(LGBM^{AT}\) outperformed (or performed equally to) \(LGBM^{O}\) are underlined. These considerations are also supported by the in-depth analysis of the accuracy performance of both \(LGBM^{O}\) and \(LGBM^{AT}\) measured with the subset of the adversarial Windows PE malware files that were used in the evaluation stage of the \(A+O\) setting of each attack method. Table 6 reports the number of adversarial Windows PE malware files correctly classified in the \"malware\" class (true malware--\(tm\)) and the remaining number of adversarial Windows PE malware files wrongly classified in the \"goodware\" class (false goodware--\(fg\)) from both decision models. These results show that the adversarial training strategy is ineffective with attacks generated via Full DOS, Shift and FGSM padding + slack, while it allows us to reduce the number of mis-classified adversarial Windows PE malware files (\(fg\)) when Extend and GAMMA are used as attack methods. In any case, the highest performance improvement was achieved using the adversarial training strategy with GAMMA. In fact, in this case, \(LGBM^{AT}\) achieved the greatest reduction in the number of adversarial Windows PE malware files that were wrongly classified as goodware compared to \(LGBM^{O}\) by passing from 75 to 2 false goodware decisions (\(fg\)). This result better supports our findings about the effectiveness of the adversarial training strategy with adversarial Windows PE malware files produced with GAMMA. To complete the analysis of the performance of adversarial training with GAMMA, we examined explanations computed with SHAP for decisions yielded via \(LGBM^{O}\) and \(LGBM^{AT}\), respectively. We conducted this explanation analysis using the adversarial PE malware produced with GAMMA, which changed from being misclassified as goodware with \(LGBM^{O}\) to being correctly classified as malware with \(LGBM^{AT}\).