# Malicious Office Macro Detection: Combined Features with Obfuscation and Suspicious Keywords
## 5 Evaluation  
After extracting the features from both Dataset1 and Dataset2, we trained four classical machine learning models to detect malicious macros. These models include random forest (RF), multi-layer perceptron (MLP), support vector machine (SVM), and K-nearest neighbor (KNN). The configuration parameters for each model are listed in Table 6. For the SVM and KNN models, their input was standardized before testing and training. \begin{table}
\begin{tabular}{c c} \hline \hline
**Keywords** & **Description** \\ \hline Auto\_Open\({}^{\star}\), AutoOpen\({}^{\star}\), Document\_Open\({}^{\star}\), Workbook\_Open\({}^{\star}\), Document\_Close\({}^{\star}\) & Procedure names will be executed automatically upon opening or closing the document \\ CreateObject\({}^{\star}\), GetObject, Wscript.Shell\({}^{\star}\), Shell.Application\({}^{\star}\) & Methods and parameters can be used to obtain the key object capable of executing commands \\ Shell\({}^{\star}\), Run\({}^{\star}\), Exec, Create, ShellExecute\({}^{\star}\) & Methods can be used to execute a command or launch a program \\ CreateProcessA, CreateThread, CreateUserThread, VirtualAlloc, VirtualAllocEx, RIMoveMemory, WrittenProcessMemory, VirtualProtect, SetContextThread, QueueApCThread, WriteVirtualMemory, Print\({}^{\star}\), FileCopy\({}^{\star}\), Open\({}^{\star}\), Write\({}^{\star}\), Output\({}^{\star}\), SaveToFile\({}^{\star}\), CreateTextFile\({}^{\star}\), Kill\({}^{\star}\), Binary\({}^{\star}\) & Methods related to file creation, opening, writing, copying, deletion, etc. \\ cmd.exe, powershell.exe, vbhide\({}^{\star}\) & Command line tools and suspicious parameters \\ StartupPath, Environ\({}^{\star}\), Windows\({}^{\star}\), ShowWindow\({}^{\star}\), dde\({}^{\star}\), Lib\({}^{\star}\), ExecuteExcel4 Macro\({}^{\star}\), System\({}^{\star}\), Virtual\({}^{\star}\) & Other keywords related to startup Path, environment variables, program windows, dde, function reference of DLL, execution of Excel 4 macro, virtualization \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Features of suspicion to detect malicious macro. We initially trained and validated the aforementioned models on Dataset1 using different feature selections. This includes obfuscation features (F1-F14), suspicious keyword features (F15), and combined features (F1-F15). Eightty percent of Dataset1 was used for training, and the remaining twenty percent was used for validation. Our experiments were conducted on a Huawei 2488V5 server with 2 Xeon (R) Gold 5118 CPUs, 64 GB system memory, and the CentOS 8 Stream operating system. We evaluated the performance of the trained classifiers using standard classification metrics, including precision, recall, accuracy, and F1-score. Considering the imbalance of malicious and benign samples in Dataset1, we also incorporated the false alarm rate (FAR) into the metrics. Table 7 displays the performance of various models obtained through 5-fold cross-validation on Dataset1. To compare with the work presented in [15], we added the best results reported in their study to the final row of Table 7.  
After conducting cross-validation experiments on Dataset1, the entire dataset was used to train our four machine learning classifiers. Subsequently, we conducted the same experiment on Dataset2, which consisted solely of malicious samples. This time, we used precision as the metric for Dataset2. Table 8 shows the results. From Table 7, it is evident that the RF classifier with features F1-F15 achieves the highest F1-score among all models. Furthermore, classifiers using suspicious keywords (F15) always yield better detection performance than obfuscation features (F1-F14). At first glance, incorporating obfuscation features that rely on suspicious keywords does not appear to have a significant impact on detection performance and may even slightly worsen it for MLP and KNN models. Nonetheless, Table 8 clearly shows that models that rely solely on suspicious keywords as features experience a significant decline in performance.