###### Abstract
Notably, adversarial attack methods synthesised for imagery and tabular data [7] cannot be used for Windows PE files since realistic adversarial Windows PE malware files must preserve a Windows PE-compatible format (format-preserving property), obtain an executable file (executable-preserving property) and maintain the malicious nature of the executable file (maliciousness-preserving property). The format-preserving property implies that adversarial modifications must generate a binary file that complies with the standard format of Windows PE files. The executable-preserving property ensures that adversarial Windows PE malware loads all the necessary data and can be executed. The maliciousness-preserving property means that adversarial Windows PE malware maintains the same malicious behaviour (e.g., modifying registry items and deleting or encrypting files) as the original Windows PE malware. On the other hand, accounting for these challenges, various adversarial attack methods have been formulated recently in Windows PE malware detection research since 2017. The authors of [8] recently developed a taxonomy for the existing literature on adversarial Windows PE malware from different viewpoints, i.e., an adversary's knowledge, an adversary's space, target malware detection, and the attack strategy. However, while the authors of [8] provided an exhaustive review of the state-of-the-art research efforts on adversarial attacks against Windows PE malware detection, this study illustrates the results of an empirical evaluation of the evasion ability of some representative attack methods producing real adversarial Windows PE malware. In particular, the evaluation study of this article was conducted using two well-known machine learning models, MalConv [9] and LGBM [10]. Specifically, MalConv is a byte-based Convolutional Neural Network (CNN) model. It takes the raw bytes from Windows PE files and uses an embedding layer, followed by several convolutional layers, to learn relevant features for the final classification that is performed with a sigmoid function. LGBM is a Light Gradient-Boosted Decision Tree Model trained on semantically rich features (engineered features) extracted through the static analysis of PE files. Notably, MalConv has recently been considered the target model to fool for several Windows PE malware attack methods [5; 11; 12] due to the good performance achieved by producing a binary code-based decision model. In fact, MalConv allows us to eliminate the use of any complex engineering step performed through the static analysis of the binary code or the dynamic analysis of the software behaviour. On the other hand, a recent study [13] compared the accuracy performance of several machine learning methods (i.e., Random Forest, XGBoost, LGBM and MalConv) used in Windows PE malware detection problems by showing that the decision model learned through LGBM is not only the most accurate model but also the fastest. In this study, as attack methods, we considered the following: Extend [5], Full DOS [5], Shift [5], FGSM padding + slack [14] and GAMMA [15]. The implementation of all of these attack methods is publicly available in the secm-malware library [16]. We note that the selected machine learning models and attack methods have been used recently in the evaluation study conducted in [5]. Extend, Full DOS, Shift, and FGSM padding+slack are white-box attacks, while GAMMA is a black-box attack. Similarly to [5], we used attack methods against MalConv and explored the transferability of adversarial malware from MalConv to LGBM. Our study confirms the conclusions drawn by [5]. The LGBM model remains quite robust to the adversarial malware generated by attacking the MalConv model. The white-box attack Extend is more effective against MalConv, while the black-box attack GAMMA is more effective against LGBM, with Extend serving as the runner-up. Differently from [5], we used a larger and more recent dataset for our evaluation.