# Malware Detection on Byte Streams of Hangul Word Processor Files
## 5 Discussion  
One might argue that there can be better structures than our proposed CNN. As described so far, we conducted a grid search to find the optimal structure, as well as the best training regime. Several samples of grid searching are shown in Table 5. We tried various structures of less or more layers, and found that the two consecutive convolutional layers followed by a pooling layer and two fully connected layers are the optimal. We also conduct, of course, various settings of the training recipe  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Accuracy** & **F1** & **Precision** & **Recall** \\ \hline GNB & 81.31 & 81.79/80.80 & 74.98/89.35 & 89.95/73.74 \\ RF & 80.18 & 82.17/77.68 & 78.98/81.83 & 85.64/73.94 \\ LR & 88.43 & 88.00/88.83 & 98.45/80.84 & 79.56/98.57 \\ CNN & **93.39** & **93.33 / 93.45** & **88.22 / 99.09** & **99.07 / 88.42** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance comparison of different models, where the two values \(b/m\) of each cell correspond to benign and malicious cases, respectively.  
\begin{table}
\begin{tabular}{c l} \hline \hline
**Model** & **Description** \\ \hline Gaussian naive bayes (GNB) & - Naive bayes classifier with Gaussian distribution \\ \hline \multirow{4}{*}{Logistic regression (LR)} & - Probabilistic classifier using linear combination of variables \\  & - penalty = L2 \\  & - C = 2.0 \\  & - Solver = liblinear \\  & - Maximum iteration = 100 \\ \hline \multirow{4}{*}{Random forest (RF)} & - Ensemble model incorporating results of multiple decision-trees \\  & - \# of estimators = 20 \\ \cline{1-1}  & - Criterion = gini algorithm \\ \cline{1-1}  & - Minimum \# of samples to split = 2 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Description and parameter settings of comparative classifiers.  
(e.g., batch normalization [27], drop-out [23], decoy [25], and gradient clipping), but we do not put all cases into the table due to the limited space. When we take account of the F1 score of malicious cases, our optimal setting achieved 93.45%, as shown in Table 4, which is absolutely better than the other remaining settings, as shown in Table 5.  
We believe that our study can prevent the public from malicious actions of HWP files. For example, a company mailing system may check every attached HWP file before the users download and open it. To do so, the proposed model should be fast enough to deal with a plenty of HWP files. We check the elapsed time for making predictions for 2061 instances, and compare it with other models as shown in Table 6. We observe that the LR is the fastest, whereas the slowest model is GNB. Although the total amount of elapsed time might not be linearly proportional to the number of instances, we can conclude that the proposed CNN is applicable to the scenario of company mailing system, as the runtime for a single instance is about 0.82 milliseconds.  
We performed an analysis of failed cases and identified two reasons. The first reason is a small number of samples. We collected only 455 malicious files: 286 files of EPS, 51 files of VBA macro, 3 file of JavaScript, 104 files of iframe tags, and 11 file of HWP vulnerabilities. The samples extracted from the HWP vulnerabilities or JavaScript methods are relatively small compared to others, so they become a weakness of the trained model. The second reason is the fixed length of streams. In many of the samples we collected, we manually identified the presence of malware in 600-byte segments and chose it as the stream length. However, in some cases, short code fractions are scattered throughout the corresponding steam. To address this limitation, it is necessary to keep collecting malicious HWP documents of various attack types, and examine them to find a better setting.