# Define the MLP model
## 5 Experiments and Evaluations
### Article Classification Model  
To select a high-performance feature combination, we trained the classification model on the traditional machine learning algorithms with combinations of features and obtain the classification performance by average the results from running 10-fold cross-validation 10 times. In each fold, a random sample of approximately 10% was hidden from the training algorithm and used as test data. Even though the SVM algorithm is the most common traditional classifier in threat-article classification, other classifiers were also trained as comparisons, such as multi-layer perceptron (MLP), decision trees, and random forest. Moreover, the XGBoost algorithm [35] is also tested on building a threat-article classification model, which is also a popular classification algorithm and can solve many data science problems in a fast and accurate way. The five classifiers are built based on the Python scikit-learn library [36], and we utilized the default parameters specified in the Python scikit-learn library. The top five results are shown in Table 7. We use some feature combinations representing the combination of multi-features. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **Team** & **Individual** & **Company** & **Forum** \\
**OSTIPs** & **OSTIPs** & **OSTIPs** & **OSTIPs** \\ \hline Threat-articles & 90 & 130 & 250 & 30 \\ Non-threat-article & 90 & 130 & 250 & 30 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Labeled datasets for training article classification model. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Feature** & **Feature Name** & **XGBoost** & **MLP** & **SVM** & **Decision Trees** & **Random Forest** \\ \hline \hline \multirow{4}{*}{A} & CSI-candidate number & & & & \\  & Topic word & 94.85\% & 94.29\% & 82.52\% & 92.19\% & 60.13\% \\ \cline{1-1}  & Dictionary-word ratio & & & & \\  & CSI-candidate number & & & & \\ \cline{1-1}  & Security Target-word density & 94.81\% & 94.25\% & 82.79\% & 92.09\% & 60.48\% \\ \cline{1-1}  & Topic word & & & & \\ \cline{1-1}  & Dictionary-word ratio & & & & \\ \cline{1-1}  & CSI-candidate number & 94.75\% & 81.79\% & 72.24\% & 91.36\% & 91.77\% \\ \cline{1-1}  & Article length & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Classification accuracy of five classifiers with different feature combinations. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Number** & **The Type of OSTIPs** & **Threat Intelligence Source** \\ \hline
1 & Team & Checkpoint, 61 ab, f-secure, badcyber \\
2 & Individual & Dynamo, bartblaze, phishtotal, eternaltodo \\
3 & Company & Fireeye, bleepingcomputer, trendmicro \\
4 & Forum & Conta, technetMicrosoft, juniper \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Some sources of threat intelligence dataset. The results clearly indicate that the XGBoost outperforms traditional algorithms. The differences between accuracy values of XGBoost and SVM are significant, as well as decision trees and random forest. While the XGBoost and MLP results are on par with each other, the XGBoost has a slight advantage. Furthermore, according to the result, the CSI-candidate number is a distinguishing feature in building the threat-article classification model. Then we turn our focus to the training time and testing time of the top five classification accuracy feature combination. The training time and testing time are shown in Table 8. As we can see, the MLP classifiers need more training time than the XGBoost classifier but need less testing time in most cases. However, for combination A, the MLP classifiers show better performance including higher accuracy, less training time, and less testing time. Considering the application of the article classifier, we trained threat-article classifier by running the MLP model over the feature combination of CSI-candidate number, topic word, and dictionary-words density. To compare our approach with state-of-the-art classification methods, we select iACE [4] and TTPDrill [6] as comparisons.