# Helping CNAs Generate CVSS Scores Faster and More Confidently Using XAI
## 7 Discussion
### SHAP Limitation for Text Datasets  
While SHAP has been widely adopted as a robust tool for explainability, its application to textual data comes with certain limitations. SHAP operates by treating each feature independently, which is effective in structured data but less optimal for textual inputs where word dependencies and order can significantly influence meaning. This limitation is particularly evident when dealing with complex relationships in text, such as context shifts and long-distance dependencies between words. Additionally, the performance of SHAP can degrade when applied to lengthy texts due to an increased number of features (words) being considered, which can introduce noise and reduce the clarity of the explanations. To overcome these challenges, integrating SHAP with advanced natural language processing techniques can improve the quality of explanations. For instance, word embeddings or contextualized language models like BERT [41] and GPT [45] can be used to capture semantic relationships before applying SHAP, thus providing more context-aware explanations. Furthermore, preprocessing techniques such as tokenization, stemming, or lemmatization can reduce textual complexity, making SHAP's word importance scores more meaningful and reliable. These complementary approaches help address SHAP's inherent limitations when dealing with textual data, enabling more accurate and interpretable model explanations. Additionally, certain words, commonly referred to as stop words, are so prevalent that they are routinely excluded from consideration. This non-uniform distribution introduces a challenge because raw Shapley values generated for a text classification problem may be skewed. For instance, a word that coincidentally appears only once or twice, and specifically with a certain class, could receive a high rating, even if it is merely a chance occurrence. In Table 13, we present the occurrence frequencies of the top five most and least common words in the dataset. The difference between the biggest and smallest occurrence is enormous. Yet they are treated the same by SHAP. It is not a design flaw, it was just not meant for this. Let us revisit the Shapley values generated in Tables 11 and 12. We sort the words based on their impact, from most to least impactful. A sample is visualized in Table 14, which lists the top four words for each class of Attack Vector based on the score column. The distribution of scores varies significantly across different classes. For example, the top score for the \"LOCAL\" class is 168.9, while for the \"NETWORK\" class it is 995.6. Taking word frequency into account can provide additional insights. We added the occurrence count of each word in the entire dataset in the frequency column. The frequency does not necessarily correlate with the score, meaning that a higher score does not imply a higher frequency and vice versa. Therefore, comparing word importance based solely on scores or frequencies is not straightforward. However, we can try to rationalize the importance of words by considering both factors. To go further into this analysis, we introduced another column called \"frequency in class\", which represents the number of occurrences of the word in the specific class it is associated with. For instance, the word \"certificate\" appears a total of 3660 times in the entire dataset but only 31 times in the \"local\" class, yet it has the highest score. Based on the score column, \"certificate\" seems to have a greater impact on the \"local\" classification compared to \"intel\" due to its higher score. However, this does not necessarily imply that \"certificate\" is more important overall, as the importance of a word depends on its frequency and context. A more comprehensive understanding of word importance is required. For instance, a word with a high score but a low frequency in a class may be less significant overall than a word with a lower score but a higher frequency in the class.