# [[https://github.com/opscq/exploit-CVE-2016-10033](https://github.com/opscq/exploit-CVE-2016-10033)]([https://github.com/opscq/exploit-CVE-2016-10033](https://github.com/opscq/exploit-CVE-2016-10033))
## Vulnerability Exploit Predicting Experiment
### Validation of Reference Host as Machine Learning Feature  
To verify our hypothesis, we created four models with the same dataset and features except for reference hosts and used four machine learning algorithms; Random Forest, Linear SVC, Logistic  
\begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt}} \hline
**Feature Group** & **Description** \\ \hline CVSS Score & CVSS (Common Vulnerability Scoring System) V2.0 Score \\ \hline CVSS Metrics & Evaluation item for estimating CVSS score \\ \hline CWE & Identifier of a software weakness type identified in CWE (Common Weakness Enumeration) list \\ \hline Length of Summary & Character length of CVE description \\ \hline N-grams & The frequency of all words that make up a document, separated by delimiters \\ \hline Vendor & Vendor’s name affected by CVE \\ \hline Product & Product’s name affected by CVE \\ \hline Reference host & Domain names hosting related CVE reference \\ \hline \end{tabular}
\end{table}
Table 1: Feature Metric  
Figure 4: Training and test set processing.  
Regression, and Decision Tree. The _All host_ model constructs the feature matrix by including all reference hosts regardless of the existence of PoC codes, which is the same feature matrix that previous studies have used. The _PoC host_ model includes PoC hosts only instead of all reference hosts. The reason for using this model is to see how the prediction performance changes when data containing correct answer are used. The _No-PoC host_ model uses only No-PoC hosts in the feature matrix. Last, _No host_ model does not use any reference host information for the feature matrix.  
Table 2 shows the average of the 5-fold cross validation experiment results of CVE exploitability prediction using the four machine learning algorithms (the prediction results of each test set of the 5-fold cross validation are included in the Appendix A).  
There is no distinct difference among machine learning algorithms, but each algorithm showed the similar pattern as follows. The All host and the PoC host models showed better prediction results than the other two models. The averaged differences between the two groups are 1.11% in accuracy, 1.68% in precision, 1.90% in recall, and 1.84% in F1-score. Let us look carefully at how the inclusion of PoC hosts affected the results. Although the number of reference hosts of the All host model was approximately four times that of the PoC model, its prediction rate is marginally better than that of the PoC model. The averaged difference between the two model are 0.24% in accuracy, 0.34% in precision, 0.46% in recall, and 0.41% in F1-score. In contrast, the No-PoC host model having three times reference hosts showed lower prediction rates than the PoC host model. The averaged difference between the two model are 0.92% in accuracy, 1.40% in precision, 1.45% in recall, and 1.47% in F1-score. This result implies that the inclusion of PoC hosts plays a critical role in the vulnerability exploitability prediction.  
Therefore, we insist that PoC hosts should be excluded since they have the answer information. If the remaining reference hosts, No-PoC hosts, gave a positive effect on prediction rate, they could be included in the feature matrix. Unfortunately, the prediction result of the No-PoC host model was not much different from that of the No-host model. In the Random Forest, the No-PoC host model even showed lower performances in accuracy and precision. Therefore, it is more preferable not to use the reference host as a feature because reference hosts having answer information will ruin the validity of CVE exploitability prediction.