# CVE2 ATT&CK: BERT-Based Mapping of CVEs to MITRE ATT&CK Techniques
## 3 Results  
This section analyses the results of the empirical experiments performed using the previously detailed models. First, it compares the performance of various models. Second, it assesses the impact of data augmentation on performance and investigates the metrics obtained by the best model. Multiple observations can be made based on the results of our experiments shown in Table 1. From the classical machine learning models, LabelPowerset is the best multi-label strategy and SVC with a linear kernel and C = 32 has the higher F1-score, competing even with our deep-learning models. The SecBERT model has the highest F1-score (42.34%) among all considered models, proving to be the most powerful solution to labeling a CVE. An important observation is that the CNN + Word2 Vec architecture obtained better results than those using simple BERT. Thus, domain-related pre-training on large security databases leads to increased performance by providing better contextualization and partially compensating for the scarce training set. Table 2 points out the appropriateness of employing data augmentation techniques on our dataset for deep learning models (approximately 6% performance gain). Only the best multi-label strategy for classical machine learning algorithms was considered. The F1-score falls considerably by 10% for Naive Bayes, in particular, since Naive Bayes places great importance on the number of appearances of a word in a document; however, swapping a relevant word with synonyms and performing random insertions or deletions (i.e., the strategies employed by the EasyDataAugmenter [28]) only confuse the model. The SVC model had a similar performance, whereas the BERT-based models take advantage of the increased sample size/the decreased class imbalance, and generalize better. Not only is performance increased, but the models also tend to learn faster (see faster convergence in Figure 7 in terms of training loss for each output layer associated with a technique in the multi-output BERT model). Moreover, Figure 7 denotes which techniques are more easily learned by the model. \begin{table}
\begin{tabular}{l l l c c c} \hline \hline
**Model Type** & **Model** & **Multi-Label Strategy** & **Weighted P** & **Weighted R** & **Weighted F1-Score** \\ \hline \multirow{4}{*}{Classical ML} & \multirow{4}{*}{Naive Bayes} & OneVsRestClassifier & 57.35\% & 9.18\% & 14.47\% \\  & & LabelPowerset & 31.40\% & 24.59\% & 24.76\% \\  & & BinaryRelevance & 57.35\% & 9.18\% & 14.47\% \\  & & RakeID & 53.71\% & 9.83\% & 15.31\% \\  & & OneVsRestClassifier & 31.97\% & 35.57\% & 33.32\% \\  & & LabelPowerset & 46.73\% & 34.75\% & 37.98\% \\  & & BinaryRelevance & 33.45\% & 34.91\% & 33.75\% \\  & & RakeID & 36.20\% & 33.77\% & 34.50\% \\ \hline \multirow{4}{*}{Deep Learning} & CNN + Word2 Vec & - & 48.32\% & **35.40\%** & 39.39\% \\  & Multi-Output BERT & - & 46.85\% & 31.47\% & 35.92\% \\ \cline{1-1}  & Multi-label BERT & - & 55.25\% & 30.98\% & 37.43\% \\ \cline{1-1}  & Multi-label SciBERT & - & **59.26\%** & 34.42\% & 41.87\% \\ \cline{1-1}  & Multi-label SecBERT & - & 57.66\% & **35.40\%** & **42.34\%** \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Results for the proposed models (italics marks the best multi-label strategy for classical ML, while bold marks the best model). Since Table 2 only provides a global overview of the average performance of the SciBERT model trained on the augmented data, exploring the particular difference between how the model handles different techniques provides additional insights into our model's behavior. Figure 8 plots the F1-score obtained for each individual technique, for both the original model and the one trained on the augmented dataset. Apart from four exceptions (_Data from Local System_, _Hijack Execution Flow_, _User Execution_ and _File and Directory Discovery_), the model obtains considerably higher or at least equal scores for all the other 27 techniques.