## 5 Discussion  
In this study, we explained the PDF document structure precisely and identified the structural, meta and content characteristics of PDF document. The method and the tool to analyze the document are described in detail to enable readers to duplicate the malicious PDF analysis. Through the analysis of a large collection of PDF documents, two basic PDF file information, four PDF file encoding and four JavaScript insertion information, and 43 keyword information in PDF file are extracted. To enrich the feature set, we further deciphered the JavaScript contained in the stream object. When JavaScript is inserted, 77 names such as method names, object names, attribute names, reserved words including operators, constants, and types, and readable strings in the script were found in the malicious codes. The number of their occurrences was used as a feature. We categorized these features according to components of JavaScript language and further developed a semantic feature set deriving readable strings. This attempt at categorization would make a thorough derivation of the feature from JavaScript and the novel feature set, semantic features excluded in previous works, could improve the detection performance. Using a total of 126 features, we constructed a machine-learning model that could distinguish between benign and malicious documents. We tested the black-box type algorithm to minimize the risk of revealing important features. Experimental results showed that three algorithms have good performance for malware and benign samples in both cases of using only the structure feature set and using the content feature set as well. However, the detection model with the structure feature set has a fair performance for base samples that were benign samples containing multiple JavaScript and the adversarial samples that were variants of base samples after the injection of malicious codes. Regarding the machine-learning algorithm perspective, we found that traditional machine-learning algorithms are working well enough for malware detection, however, the performance decreases for adversarial samples except for the random forest algorithm. The random forest algorithm might have a good performance because of this transferability. Adversarial sample transferability is defined as the property that adversarial samples produced to mislead a specific model that can mislead other models [6]. This property is important in terms of the practical impact of the adversarial attack. In this work, we check the adversarial sample transferability using the random forest model, which trains the model constructing the ensemble trees with the subset of samples and features and validates the built tree with unused samples. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Algorithm** & **Class** & **Precision** & **Recall** & **F-Measure** \\ \hline \multirow{2}{*}{Random Forest} & Benign & 0.997 & 1 & 0.998 \\  & Malware & 1 & 0.997 & 0.999 \\ \hline \multirow{2}{*}{Naïve Bayes} & Benign & 0.878 & 0.929 & 0.903 \\  & Malware & 0.94 & 0.895 & 0.917 \\ \hline \multirow{2}{*}{SVM} & Benign & 0.976 & 0.999 & 0.987 \\  & Malware & 0.999 & 0.98 & 0.989 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: The detection performance of machine-learning algorithms with the full feature model. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Class** & **Algorithms** & **Benign** & **Malware** \\ \hline \multirow{2}{*}{Base samples (Benign)} & Radom Forest & 113 & 2 \\  & Naïve Bayes & 114 & 1 \\  & SVM & 73 & 42 \\ \hline \multirow{2}{*}{Adversarial samples (Malware)} & Random Forest & 2 & 99 \\  & Naïve Bayes & 58 & 43 \\ \multicolumn{2}{c}{SVM} & 13 & 88 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Classification result of base samples and adversarial samples with the full feature model. There is still room for improvement for the proposed algorithm. In our study, we only used keywords found in PDF, but there are various malwares that use JavaScript.