# Malicious Office Macro Detection: Combined Features with Obfuscation and Suspicious Keywords
## 5 Evaluation
Interestingly, however, when considering the combined features, the random forest model achieves a precision of 0.953 on the 2885 new malicious samples. This indicates that integrating obfuscation features and suspicious keywords enhances the resilience of the model. \begin{table}
\begin{tabular}{c c} \hline
**Model** & **Parameter** \\ \hline RF & n\_estimators = 100 \\ MLP & hidden\_layer\_sizes = 150, max\_iter = 500 \\ SVM & kernel = ’rb\(t\)’ \\ KNN & n\_neighbors = 3 \\ \hline \end{tabular}
\end{table} 
Table 6: Configuration of models. \begin{table}
\begin{tabular}{c c c c c c c} \hline \multirow{2}{*}{**Model**} & **Feature Selection** & **FAR** & **Precision** & **Recall** & **Accuracy** & **F1-Score** \\ \hline RF & F1–F14 & 0.083 & 0.982 & 0.994 & 0.981 & 0.988 \\ RF & F15 & 0.012 & 0.997 & 0.995 & 0.993 & 0.996 \\  & F1–F15 & 0  
Compared with the work in [15], our RF classifier trained using suspicious keywords (F15) has a higher recall and F1-score. This is likely due to the inclusion of new suspicious keywords, such as \"GetObject\" and external functions, in our features. These additions enable us to detect more malicious samples. Table 9 displays the top 20 features of the RF classifier, which use combined features to identify malicious samples in Dataset2. It reveals that only half of the prominent features consist of suspicious keywords. Additionally, ten obfuscation features, including four statistical features, demonstrate their significance in detecting malicious macros. This highlights the potential for complementarity between both types of features in improving detection performance. We further analyzed the proportion of benign and malicious samples containing keywords from F14 and F15. Table 10 demonstrates that samples containing these keywords are more likely to be malicious. \begin{table}
\begin{tabular}{c c c} \hline
**Model** & **Feature Selection** & **Precision** \\ \hline \multirow{4}{*}{RF} & F1–F14 & 0.923 \\  & F15 & 0.856 \\  & F1–F15 & **0.953** \\  & F1–F14 & 0.739 \\ MLP & F15 & 0.793 \\  & F1–F15 & **0.907** \\  & F1–F14 & 0.799 \\ SVM & F15 & 0.788 \\  & F1–F15 & **0.817** \\  & F1–F14 & 0.792 \\ KNN & F15 & 0.757 \\  & F1–F15 & **0.735** \\ \hline \end{tabular}
\end{table} 
Table 8: Performance results of models on Dataset2. \begin{table}
\begin{tabular}{c c c} \hline
**Index** & **Feature Name** & **Feature Type** \\ \hline
1 & F15:CreateObject & Suspicious Keywords \\
2 & F15:Document\_Open & Suspicious Keywords \\
3 & F15:Shell & Suspicious Keywords \\
4 & F15:GetObject & Suspicious Keywords \\
5 & F15:Lib & Suspicious Keywords \\
6 & F15:AutoOpen & Suspicious Keywords \\
7 & F15:Auto\_Open & Suspicious Keywords \\
8 & F15:StartupPath & Suspicious Keywords \\
9 & F11:length(P) & Obfuscation \\
10 & F3:max\{count\_arithmetic(l_{i})\} & Obfuscation \\
11 & F14:Chr & Obfuscation \\
12 & F14:Asc & Obfuscation \\
13 & F14:UCase & Obfuscation \\
14 & F7:max\{count\_arithmetic(p_{i})\} & Obfuscation \\
15 & F5:max\{count\_strings(l_{i})\} & Obfuscation \\
16 & F14:Left & Obfuscation \\
17 & F15:Open & Suspicious Keywords \\
18 & F14:Abs & Obfuscation \\
19 & F14:Split & Obfuscation \\
20 & F15:System & Suspicious Keywords \\ \hline \end{tabular}
\end{table} 
Table 9: The top 20 prominent features and their type. We conducted further investigation to understand why there are still some samples in Dataset2 that cannot be detected by the RF classifier when using combined features. It is interesting to note that some of these undetected samples can be identified by the RF classifier using either feature selection F1-F14 or F15 alone. A schematic diagram is included to illustrate sample detection using the RF classifier on Dataset2 (see Figure 9). The samples successfully detected by the corresponding classifier are represented by the points inside the circle, while the points outside the circle depict the undetected samples. It is apparent that each classifier can identify a subset of the samples.