# Machine Learning-Based Malicious X.509 Certificates' Detection
## 6 Experiment
### Results
#### 6.3.2 Importance Analysis  
In addition to the validation and testing accuracy of different models, we analyze the importance values of the features in the best model of DT, RF, XGBoost, LightGBM, and CatBoost. Figure 5 shows the top 10 important features in XGBoost and their scores. The score is the improvement of accuracy brought by a feature to the branches it is on. Figure 6 shows the top 10 important features in LightGBM and their scores. The score is the relative number of times a particular feature occurs in all splits of the model's trees. Figure 7 shows the top 10 important features in CatBoost and their scores. Refer to official document (LightGBM document: [[https://catboost.ai/docs/concepts/fstr.html#fstr_regular-feature-importance](https://catboost.ai/docs/concepts/fstr.html#fstr_regular-feature-importance)]([https://catboost.ai/docs/concepts/fstr.html#fstr_regular-feature-importance](https://catboost.ai/docs/concepts/fstr.html#fstr_regular-feature-importance))), the score is computed with following Formula (1) and (2). \(c_{1}\), \(c_{2}\) are the number of objects in each leaf and \(v_{1}\), \(v_{2}\) are the formula values in the left and right leaves. \[feature\_importance_{F}=\sum_{trees,leaf_{F}}(v_{1}-a\
u r)^{2}*c_{1}+(v_{2}-a \
u r)^{2}*c_{2} \tag{1}\]  
\[a\
u r=\frac{v_{1}*c_{1}+v_{2}*c_{2}}{c_{1}+c_{2}} \tag{2}\]  
Figure 8 shows the top 10 important features in DT and their scores. The score is computed as the (normalized) total reduction of the criterion brought by that feature, which is known as the Gini importance (Gini importance: [[https://medium.com/the-artificial-impostor/feature-importance-measures-for-tree-models-part-i-47f187c1a2c3](https://medium.com/the-artificial-impostor/feature-importance-measures-for-tree-models-part-i-47f187c1a2c3)]([https://medium.com/the-artificial-impostor/feature-importance-measures-for-tree-models-part-i-47f187c1a2c3](https://medium.com/the-artificial-impostor/feature-importance-measures-for-tree-models-part-i-47f187c1a2c3))). \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Source** & **Degree of Freedom (DF)** & **Sum of Squares (SS)** & **Mean Square (MS)** & **F-Statistic** \\ \hline Between Groups & 8 & 0.0989 & 0.0124 & 15.0821 \\ \hline Within Groups & 54 & 0.0443 & 0.0008 \\ \hline Total: & 62 & 0.1432 & \\ \hline \hline \end{tabular}
\end{table} 
Table 13: The ANOVA results of different models. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline
**Accuracy** & **Dataset1** & **Dataset2** & **Dataset3** & **Dataset4** & **Dataset5** & **Dataset6** & **Dataset7** & **Average** \\ \hline LR & 0.938 & 0.930 & 0.928 & 0.934 & 0.933 & 0.938 & 0.934 & 0.934 \\ \hline DT & 0.955 & 0.941 & 0.954 & 0.954 & 0.971 & 0.966 & 0.950 & 0.956 \\ \hline SVM & 0.835 & 0.827 & 0.812 & 0.821 & 0.976 & 0.978 & 0.982 & 0.890 \\ \hline RF & 0.960 & 0.943 & 0.955 & 0.953 & 0.968 & 0.965 & 0.954 & 0.957 \\ \hline XGBoost & 0.963 & 0.947 & 0.958 & 0.957 & 0.968 & 0.971 & 0.965 & 0.961 \\ \hline LightGBM & 0.967 & 0.951 & 0.953 & 0.951 & 0.973 & 0.974 & 0.966 & 0.962 \\ \hline CatBoost & 0.957 & 0.946 & 0.954 & 0.955 & 0.970 & 0.965 & 0.954 & 0.957 \\ \hline CNN & 0.872 & 0.860 & 0.859 & 0.862 & 0.865 & 0.866 & 0.872 & 0.865 \\ \hline LSTM & 0.869 & 0.857 & 0.860 & 0.863 & 0.865 & 0.868 & 0.873 & 0.865 \\ \hline Average & 0.924 & 0.911 & 0.915 & 0.917 & 0.943 & 0.943 & 0.939 & 0.927 \\ \hline \hline \end{tabular}
\end{table} 
Table 12: Testing accuracy of the best model. Figure 9 shows the top 10 important features in RF and their scores. The score is the average feature importance of trees in the forest. The feature importance of each tree is computed with Gini importance. As a result that the feature importance property is not provided in every model implementation, we cannot obtain all models' feature importance. Therefore, we analyze the feature importance of XGBoost, LightGBM, CatBoost, DT, and RF. From the top 10 important features in each model, we can see that \"valid_time_so_far\" is the most important feature in three out of five models.