## 5 Experiments and Results
### Explaining Artificial Intelligence Models  
Although model accuracy is of great importance, insights on model rationale are required to assess the main factors driving the forecast are reasonable, and thus the forecast can be trusted. Although much research in the literature was devoted to global explanations, we found few authors taking into account the data or model training process. Furthermore, we found no authors combined insights regarding feature selection, and how relevant the selected features are to the model across a repeated cross-validation. We therefore propose a novel visualization that summarizes the aforementioned insights (see Figure 11). \begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Model**} & **Experiment 1** & **Experiment 2** \\ \cline{2-3}  & **AUC ROC\({}_{\text{mean}}\)** & **AUC ROC\({}_{\text{mean}}\)** \\ \hline Baseline 1 (zero forecast) & 0.5000 & * 0.5000 \\ \hline Baseline 2 (LR) & 0.5656 & \(\uparrow\)0.5675 \\ \hline Model 1 (LR) & 0.6567 & \(\downarrow\)0.6059 \\ \hline Model 2 (SVC) & 0.4491 & * \(\uparrow\)0.4897 \\ \hline Model 3 (MLPC) & _0.6709_ & \(\downarrow\)0.5381 \\ \hline Model 4 (Catboost) & **0.7359** & \(\uparrow\)**0.7670** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Out-of-specification detection results for _Unit A_. Mean ROC AUC values we obtained for different models with a ten-fold cross-validation, repeated 50 times. Best results are bolded, second-best are reported in italics. The results within the same column, which have no statistically significant difference between them when tested with a Wilcoxon paired rank test at a 95% confidence level, are marked with *. The arrows indicate whether the mean result improved (\(\uparrow\)), or degraded (\(\downarrow\)) when compared to Experiment 1. _Unit B_ is not reported, since the dataset did not include out-of-spec measurements for _Unit B_. Although much research work in the literature has been devoted to global explanations, and some related work focuses on the characteristics of the dataset, little research has been done on integrating insights regarding the dataset, the experimental setting, and the resulting model. We therefore propose a novel visualization which combines the three aforementioned parts. Figure 11A provides a brief description regarding the experimental setting. In this particular case, it states that the corresponding plots result from data obtained when training a forecasting model in a repeated 10-fold cross-validation setting, repeating the cross-validation 50 times. Figure 11B shows a horizontal stacked bar plot, where the intensity of _feature concepts_ is presented to the users. We consider _feature concepts_ as semantic abstractions that group certain features based on feature metadata. In our particular case, such grouping was performed for features computed with the same formula, but using sensor data at different points in time (see Figure 3). The shades of gray within the horizontal stacked bars represent how frequently was each feature of the _feature concepts_ abstraction chosen when performing feature selection, within the 50 times 10-fold cross-validation. The line chart overlayed to the horizontal stacked bar plot informs how relevant were those feature concepts to the forecasting machine learning models, on average. Such overlay provides useful information to the machine-learning engineer, who can remove features found not informative to the model, to give room to better ones. In this particular case, we found three such cases (\(p<7.06\), \(p>7.63\), and \(p<7.14\)), referring to features with Boolean values, assessing whether the pressure values obtained from the sensor were above or below certain threshold value. By clicking a particular _feature concept_ in the horizontal bar stacked plot, the section highlighted in Figure 11C is updated. Figure 11C enriches the aforementioned view, detailing each feature's relevance within the specific _feature concept_.