## 5 Discussion  
We conducted experiments to compare with baseline models, namely, CNN, BERT, and Gemini. For BERT, the selected model was cased with a vocabulary size of 32K. Leveraging the success of the MalCode2 Vec model, which performs well with the Transformer model, we adapted it for the CNN classifier. The embedding was performed at the word level. The results presented in Table 4 show that our proposed Transformer model outperformed the baseline models in file-level and binary malware detection. The results of CNN were more accurate than that of the BERT model. Notably, CNNs are known to have a powerful feature extraction network; therefore, they can learn patterns from the code sequences. The BERT model is more advanced, having features such as the ability to learn the contextual relationship between words; however, the sequence length limitation is the challenge in fine-tuning the Transformers on long text datasets. BERT models have a maximum sequence length of 512 tokens. This limits the input of important features to BERT, as the malware sequence has a longer sequence length of more than 10K. The results from the Gemini model indicate that the model excels at the objective of minimizing the false negatives with a higher recall of 89.76%. However, the precision is relatively low at 29.02%, suggesting that while the model is good at recognizing malware, it also misidentifies a considerable number of benign instances as malware. The result reflects that the model is generally precise when identifying benign samples but faces challenges in achieving a balance between precision and recall for the malware class. While Gemini is a powerful large language model trained on a multi-domain dataset, its performance may not be optimal for domain-specific tasks like malware detection. Most LLMs, including Gemini, are trained on diverse, general-purpose datasets, which may not capture the nuances of the cybersecurity domain, such as the specific characteristics of malware or benign data. Therefore, further domain-specific fine-tuning would likely improve the model's ability to distinguish between malware and benign more accurately. Table 8 compares our work with other malware detection works conducted on Hangul Word Processor files [19,37]. All studies used convolutional neural networks with spatial pyramid average pooling in [19]. The studies show strong recall and better precision for malicious cases, indicating a significant number of false negatives. In comparison, our models demonstrate higher performance in benign classification and a fair recall score in malicious class. We believe that the unbalanced data ratio for malicious samples may be one of the contributing factors. The Transformer-based model with MalCode2 Vec word embedding outperforms the CNN approach, showing an improvement of 3.41% in recall and 2.2% in F1 score for the malicious class. Given our aim for detecting the malware attacks within the document file, the false-negative rate is important to be minimized because it represents the ratio of missed mali \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**Stream-Level Performance Metrics**} & \multicolumn{2}{c}{**Data Statistics**} \\ \cline{2-9}
**Study** & \multicolumn{2}{c}{**Precision**} & \multicolumn{2}{c}{**Recall**} & \multicolumn{2}{c}{**F1 Score**} & \multirow{2}{*}{**Benign**} & \multirow{2}{*}{**Malicious**} \\ \cline{2-9} \cline{5-9}  & **b** & **m** & **b** & **m** & **b** & **m** & \\ \hline
[19] (SPAPConv) & 87.28 & 99.46 & 99.20 & 91.07 & 92.86 & 95.08 & 2852 & 3668 \\
[37] (CNN) & 88.22 & 99.09 & 99.07 & 88.42 & 93.33 & 93.45 & 6150 & 7031 \\ \hline
**Ours (CNN-MalCode2 Vec)** & 96.19 & 97.19 & 99.38 & 84.39 & 97.76 & 90.34 & \multirow{2}{*}{5470} & \multirow{2}{*}{1312} \\
**Ours (Transformer-MalCode2 Vec)** & 97.00 & 97.83 & 99.51 & 87.80 & 98.24 & 92.54 & & \\ \hline \hline \end{tabular}
\end{table} 
Table 8: A comparative summary of our proposed method with the previous study. SPAPConv stands for spatial pyramid average pooling CNN model.