## 4 Experimental Results
### Results
\begin{table}
\begin{tabular}{l l c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Embed**} & \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**Stream-Level**} & \multicolumn{3}{c}{**Stream-Level Performance Metrics**} & \multicolumn{3}{c}{**File-Level**} \\ \cline{3-11}  & & & \multicolumn{3}{c}{**Precision**} & \multicolumn{3}{c}{**Recall**} & \multicolumn{3}{c}{**F1**} & \multirow{2}{*}{**Acc**} & \multirow{2}{*}{**F1**} \\ \cline{3-11}  & & & & **b** & **m** & **b** & **m** & **b** & **m** & \\ \hline Integer & Transformer\({}_{char}\) & 92.93 & 88.76 & 94.91 & 84.46 & 96.31 & 79.51 & 95.60 & 81.91 & 90.46 & 88.76 \\ encoding & Transformer\({}_{word}\) & 96.95 & 95.09 & 97.00 & 96.77 & 99.26 & 87.80 & 98.12 & 92.07 & 95.01 & 94.07 \\ \hline \multirow{3}{*}{MalCode2 Vec} & Transformer\({}_{char}\) & 93.22 & 88.48 & 93.36 & 92.50 & 98.52 & 72.20 & 95.87 & 81.10 & 90.89 & 88.85 \\  & Transformer\({}_{word}\) & 97.15 & 95.39 & 97.00 & 97.83 & 99.51 & 87.80 & 98.24 & 92.54 & 95.66 & 94.83 \\  & CNN\({}_{word}\) & 96.37 & 94.05 & 96.19 & 97.19 & 99.38 & 84.39 & 97.76 & 90.34 & 94.79 & 93.72 \\ \hline WordPiece & BERT\({}_{Bi-LSTM}\) & 69.65 & 69.15 & 80.36 & 22.34 & 82.04 & 20.49 & 81.19 & 21.37 & 63.77 & 64.12 \\  & Gemini-2.0 & 53.73 & 57.27 & 94.53 & 29.02 & 44.65 & 89.76 & 60.65 & 43.86 & 62.69 & 59.88 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Experimental results using integer encoding and MalCode2 Vec model. The accuracy and performance metrics are for the stream level. File-level accuracy is the model’s performance in categorizing the HWP files as benign or malicious. b and m correspond to benign and malicious cases, respectively. \begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **FNR (Benign)** & **FNR (Malicious)** \\ \hline Integer encoding Transformer char & 3.57\% & 20.49\% \\ Integer encoding Transformer word & 0.74\% & 12.20\% \\ MalCode2 Vec Transformer word & 0.49\% & 12.20\% \\ MalCode2 Vec CNN word & 0.62\% & 15.61\% \\ \hline \hline \end{tabular}
\end{table} 
Table 5: False-negative rates (FNRs) for different models. Attention mechanisms improve the detection of a code or behavior indicative of malware; this suggests that the Transformer is powerful, with self-attention as the key. Interestingly, the results obtained from both approaches, integer encoding and the MalCode2 Vec model, exhibit comparative performance. Figure 13 demonstrates this using the confusion matrix results. This can happen because most of the tokens in the malware data have no semantic meaning as normal text in the vector space representation. Word-level embeddings generally achieve higher accuracy than character-level embeddings. This advantage stems from the fact that words inherently convey more meaning, aiding in the differentiation between malicious and benign patterns. In contrast, character-level embeddings can be affected by data sparsity, particularly in languages with complex scripts or where words are formed by combining multiple characters. The MalCode2 Vec model, which utilizes word-level embeddings, delivers optimal performance in stream and file-level classification. This suggests that embedding vectors effectively capture meaningful relationships within malware sequences, allowing words with similar meanings to be closely positioned in the embedding space. As a result, the model excels both semantically (by comparing similar scripts) and syntactically (by learning sequence patterns), as discussed in [13]. In the evaluation of inference time, we found that the Transformer model with integer encoding achieved a faster inference time of 1.981 milliseconds compared with the Code2 Vec model, which took 2.497 milliseconds on the GeForce GTX 1080 Ti GPU (NVIDIA, San Jose, CA, USA). The longer inference time of Code2 Vec can be attributed to its more complex architecture. Code2 Vec employs neural embeddings and context-aware vector representations, which require additional processing steps.