# Embedding Security Awareness in IoT Systems: A Framework for Providing Change Impact Insights
## 5 Performance Analysis
### Fidelity Assessment  
However, we also recognize that the generated explanations require not only performance enhancements but also improvements in the fidelity of their outcomes. Like many post hoc interpretability methods, XAI techniques such as LIME can introduce unintended biases or reinforce spurious correlations. Although these explanations may appear visually or numerically convincing, they do not always reflect the true decision-making logic of the model or the underlying security semantics. In real-world deployments, such misalignments can mislead analysts or result in suboptimal response decisions if not properly validated. We conduct a fidelity assessment to evaluate how well the explanations generated by LIME align with known protocol-specific ground truth features associated with various DDoS attack types. The ground truth was manually constructed based on expert knowledge of critical features relevant to the TCP, UDP, and HTTP, as outlined in [55]. Each LIME explanation was transformed into a binary feature vector and compared against its corresponding ground truth vector using cosine similarity--a metric that captures directional alignment independent of vector magnitude [60; 61]. In addition to cosine similarity, we computed precision, recall, and F1-score to assess the relevance and completeness of the features highlighted by LIME, as summarized in Table 5.  
\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Attack Type** & **Precision** & **Recall** & **F1-Score** & **Cosine Similarity** \\ \hline DDoS\_TCP & 0.85 & 0.51 & 0.63 & 0.65 \\ \hline DDoS\_UDP & 0.19 & 0.50 & 0.27 & 0.31 \\ \hline DDoS\_HTTP & 0.44 & 0.50 & 0.47 & 0.47 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Explanation’s fidelity assessment result.  
The results revealed several important insights:  
* DDoS_TCP explanations exhibited the highest fidelity, with a cosine similarity score of approximately 0.65. This strong alignment is attributed to the richer and more diverse set of TCP-specific features in the ground truth, increasing the likelihood of overlap with LIME-selected features. The model appears to have captured meaningful protocol-level patterns consistent with known attack behaviors.
* DDoS_HTTP explanations achieved moderate fidelity, with a cosine similarity of around 0.47. This reflects partial alignment, likely influenced by the limited number of HTTP-specific features in the ground truth. Some LIME-identified features were weakly relevant or attributable to noise.
* DDoS_UDP explanations demonstrated the lowest fidelity, with a cosine similarity of approximately 0.31. This suggests that the explanations either focused on less relevant features or failed to capture the protocol's limited but critical attributes. The relative simplicity and minimal feature diversity of the UDP may have contributed to this outcome.  
These findings underscore an important consideration: the quality of XAI explanations varies significantly across attack types and is influenced not only by the model's learning capability but also by the structural richness and complexity of the protocol-specific feature space. While protocols like TCP provide ample opportunities for accurate attribution, simpler protocols such as UDP present greater challenges for generating meaningful and interpretable explanations. This variability highlights the need for protocol-aware interpretability validation and the development of tailored XAI strategies for different network contexts.