# Embedding Security Awareness in IoT Systems: A Framework for Providing Change Impact Insights
## 5 Performance Analysis  
### Execution Time  
To evaluate the performance of our proposed framework for embedding security awareness into IoT systems, we conducted a detailed analysis of the deep learning (DL) pipeline. This evaluation measured the average execution time for three key components: model prediction, XAI-based explanation generation, and the mapping of XAI outcomes to the system's security profile via the _GenerateAttackStatus&ConsequenceResult_ process. The experiments were performed on a personal computer running Microsoft Windows 10 Pro, equipped with a 12 th Gen Intel(R) Core(TM) i9-12900H CPU @ 2.50 GHz and 32 GB of RAM (Intel, Santa Clara, CA, USA). A summary of the execution time analysis is presented in Table 4.  
We also assessed performance across different attack types and observed that all three components of the DL pipeline maintained consistent execution times, regardless of the specific attack scenario. However, the XAI explanation generation consistently incurred significantly higher processing time compared to the other components.  
\begin{table}
\begin{tabular}{l l l} \hline \hline  & **Process Name** & **Average Execution Time (in s)** \\ \hline \multirow{3}{*}{Overall} & Model Prediction & 0.154893 \\ \cline{2-3}  & Generate Explanation & 8.426568 \\ \cline{2-3}  & Mapping and Generate Report & 0.151432 \\ \hline \multirow{3}{*}{DDoS\_UDP} & Model Prediction & 0.159036 \\ \cline{2-3}  & Generate Explanation & 8.082033 \\ \cline{2-3}  & Mapping and Generate Report & 0.153765 \\ \hline \multirow{3}{*}{DDoS\_TCP} & Model Prediction & 0.152603 \\ \cline{2-3}  & Generate Explanation & 8.802392 \\ \cline{2-3}  & Mapping and Generate Report & 0.149118 \\ \hline \multirow{3}{*}{DDoS\_HTTP} & Model Prediction & 0.153175 \\ \cline{2-3}  & Generate Explanation & 8.373173 \\ \cline{1-1} \cline{2-3}  & Mapping and Generate Report & 0.151548 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance analysis result.  
We recognize that the high latency of the Generate Explanation process may not be practical for continuous monitoring of network traffic streams. To address this, we configure the explanation process to activate only when the model predicts an attack with a confidence level above 90%. Furthermore, we investigated the cause of the high computational overhead associated with explanation generation. LIME operates by creating numerous perturbed versions of the input instance, then training a surrogate model (a random forest in our case) on these samples using the predicted outputs from the black-box model. The surrogate model is weighted by similarity scores to ensure it accurately approximates the local decision boundary of the deep neural network (DNN). The goal is to identify which features are most influential in the model's decision for that specific instance. However, performing perturbation and surrogate model training on a large dataset leads to significant computational costs.  
To mitigate this, we conducted an additional experiment where the sample size was reduced to 100 perturbed instances. This optimization reduced the average execution time of the Generate Explanation process to 3.3392 s--an improvement over the initial configuration. While this is a step forward, we acknowledge that it still falls short of being suitable for real-time applications. We also experimented with SHAP as an alternative XAI technique to assess its effectiveness in generating explanations. However, SHAP demonstrated even longer execution times due to its more complex computation of feature attributions, particularly when compared to LIME's relatively lightweight perturbation strategy. In future work, we aim to incorporate advanced ML/DL optimization techniques to enhance system efficiency and further reduce the latency associated with explanation generation, thereby moving closer to enabling real-time, interpretable threat detection.