# Helping CNAs Generate CVSS Scores Faster and More Confidently Using XAI
## 7 Discussion
### Consideration to Take When Analyzing SHAP Exclusive Words  
To prevent misinterpretations and facilitate efficient research, we wish to emphasize that SHAP scores quantify the importance of a word for the model's prediction, not the actual correlation of a word for a class in the dataset. A high cumulative SHAP score for a specific word in a particular class indicates that the word contributes significantly to the model's prediction for that class compared to other words in the input. However, it does not guarantee that the presence of that word alone will always lead to the vulnerability being classified into that class. Therefore, if a word has a 100% impact on one class and 0% on the rest, it does not mean that the existence of that word automatically classifies the vulnerability into the class associated with that exclusive word. One of the reasons is that the model's decision-making process does not rely solely on individual words but takes into account the overall context of the input sequence, considering the interactions between words and how they combine to form meaningful phrases and sentences. To demonstrate our point, we attempted to categorize vulnerabilities by employing straightforward IF rules derived from the exclusivity of words. IF rules, also referred to as \"if-then rules\", are logical constructs utilized in decision making and problem solving. These rules comprise conditions and corresponding actions; if a particular condition holds, a designated action is executed. We conducted empirical tests to evaluate their performance by treating them as Machine Learning models and measuring their accuracy, F1 score, and data coverage. Data coverage refers to the percentage of data that can be tested using the IF rules. Even if the coverage is limited, high precision can still allow us to trust the IF rules for that portion of the data while leaving the rest to Machine Learning models or human analysis. Our key focus for generating these rules is the exclusivity of words to classes. As mentioned earlier, \"exclusive words\" are those that are 100% correlated with one class and do not correlate with others. In these tests, we used only SHAP-based exclusivity, that is, words whose SHAP scores are 0 for all classes except one. The procedure is as follows:  
1. Extract and store exclusive words. 2. Check whether an exclusive word exists in the vulnerability description. 3. If it exists, classify the vulnerability as the class associated with that word. Since we have the score of each word for each class, we can determine the exclusive words by summing the scores for each class and identifying words that have a score of 0 for all classes except one. It is important to note that this exclusivity is based on SHAP scores and not the actual presence of the words in the sentence. Therefore, it is possible to have multiple exclusive words in the same sentence. To handle this conflict, we devised three methods that follow the same mechanism: for each sentence, we create a dictionary to count the occurrences of exclusive words found. Then the word with the highest frequency becomes the winner, and the sentence is classified under that word's exclusive class. The difference among the three methods lies in how they handle cases of a tie (i.e., equal frequency of two or more words). In such cases, we can choose from three possible approaches:  
1. Random pick: Choose randomly from the equal words. Any one of them should be good. 2. Highest single: Choose the class of the word with the highest score. 3. ffHighest sum: Sum up the class scores of all the words and choose the class with the highest total score. We evaluate the performance of these approaches using the accuracy and F1 metrics, similar to those used in Machine Learning, for each of the two types of exclusive words. Since this approach does not involve Machine Learning, we do not need to split the data into training and testing sets. However, we do perform preprocessing on the vulnerability descriptions.