# Helping CNAs Generate CVSS Scores Faster and More Confidently Using XAI
## 5 Experimental Design and Results
### Ablation Study  
Figure 12 illustrates the performance variation observed during our ablation study for each of the hyperparameters mentioned previously. We observe the following:  
* Number of estimators: This shows a peak performance at around 300 estimators, indicating an optimal balance between underfitting and overfitting. * Learning rate: The graph demonstrates fluctuations, indicating the presence of multiple local maxima in the dataset. Notably, the performance values of 0.5 and 0.7 are very close, with a slightly better performance observed for a learning rate of 0.7. * Preprocessing techniques: Both CountVectorizer and TF-IDF Vectorizer exhibit superior performance compared to OneHotEncode Vectorizer. Furthermore, the performance values of CountVectorizer and TF-IDF Vectorizer are almost identical, with a slight advantage observed for the TF-IDF Vectorizer. * Max features: Interestingly, this parameter shows a rapid improvement in performance up to 1000 words, but a further increase to 10,000 words is required to surpass the current state of the art. This implies that in an industry context, a smaller number of words (a few thousand) can still yield satisfactory results while significantly reducing training time. * N-grams: Table 8 presents the performance variation for different N-grams. Our experimentation includes unigrams (one word), bigrams (two words), trigrams (three words), and the combination of bigrams and trigrams at the same time. Unfortunately, due to memory limitations, we were unable to test all combinations from one to three g simultaneously. Nonetheless, the results show that augmenting the sequence length \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Works** & **Added Data** & **Classifier** & **Data Size** \\ \hline Our work & None & XGBoost & 81,889 \\ \hline [PERSON] et al. [3] & None & XLNet & 101,519 \\ \hline [PERSON] et al. [2] & 5K words vocab & DistilBERT & 79,810 \\ \hline [PERSON] et al. [11] & Scraped articles from the web & DistilBERT & 88,979 \\ \hline [PERSON] and & None & BERT-small & 45,926 \\ Debar [10] & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Experimental conditions comparison. does not lead to improved performance, as unigrams obtained the best results on almost all metrics. Considering the best values for each parameter, the optimal configuration for our XGBoost model is as follows:  
* Number of estimators: 300. * Learning rate: 0.5. * Max features: 10,000. * Preprocessing technique: TF-IDF Vectorizer. * N-grams: Unigrams. Since TF-IDF was selected for preprocessing, we demonstrate an example of a step-by-step transformation of a sentence from the dataset in Table 9. The process includes the following steps:  
1. Tokenization: The sentence is split into individual words or tokens. 2. Lowerasing: All tokens are converted to lowercase to ensure consistency. 3. Stop word removal: Common words such as \"the\" and \"and\" are removed as they do not contribute significantly to the meaning of the text. 4. Stemming or lemmatization: Words are reduced to their base or root form to reduce the number of unique words. Figure 12: Model parameters ablation studies.