# XAI in the Context of Predictive Process Monitoring: An Empirical Analysis Framework
## 5 Results and Observations
### Global Methods Comparability
#### 5.1.1 Comparability  
Our study on how PFI, ALE and SHAP analysed the effects or contributions of the most important feature for each predictive model, led to some interesting observations:  
_Observation 4: The cardinality of categorical features has an effect on the explanations generated by permutation-based methods, which depend on measuring the prediction errors after changing a feature value. The effect of the used preprocessing combination depends on the used predictive model._  
For features with lower number of categories, PFI is unable to capture the effect of shuffling feature values on predictions. The effect of shuffling a feature value is static in predictions generated using LR on most event logs, approaching zero in most cases. However, a slight change across PFI iterations can be observed in event logs preprocessed with single aggregation techniques. This observation indicates an effect of the encoding technique along with the interactions between the features. Shuffling values in PFI aims to break dependencies between the features [4]. However, with a low cardinality of a feature, the chances of reducing dependencies decreases over a few number of iterations. PFI over XGBoost is presenting slightly higher shuffling effects in prefix-indexed event logs than in LR. This effect is slightly changing over shuffling iterations in the same event logs. However, the increase of changes in XGBoost predictions over iterations of shuffled feature values are observed in prefix-indexed event logs to be higher than in single-aggregated ones. Figure 3 shows PFI scores for the _Traffic_fines_ event log, which was preprocessed with single aggregation (Figure 3(1-a),(2-a)) and prefix index (Figure 3(1-b),(2-b)). Figure 3(1-a),(1-b) represent change in prediction errors for both predictive models while changing the top important features to XGBoost. Finally, Figure 3(2-a),(2-b) represent the same for the top two important features to LR. to XGBoost, there is an interaction between both features resulting in having points with the same _CreditScoreother_ value to obtain different shap values. These shap values decrease as the values of _std_event_nr_ increase for points with the same _CreditScoreother_ value. _Observation 5: In all event logs, ALE plots for LR are linear themselves._  
An exception to this observation is present in the _Traffic_fines_ event logs preprocessed with single aggregation. In Figure 4(a), the importance of the feature _sum_timesincelastevent_, as indicated by LR coefficients, is confirmed by the change in ALE scores with a sudden decrease with increasing feature values. Although Figure 5 also shows a decrease in XGBoost predictions, the expected decrease is stable and not steep after _sum_timesincelastevent_ is reaching a value of (0.5 \(\times\) 10\({}^{6}\)) unlike the case with LR model. However, in both cases ALE plot is interpolating as a result of the big gap in feature distribution. Note that the deciles on the x-axis represent interval edges at which ALE scores are calculated. In turn, in between the deciles are parts in which the ALE plot is interpolating. Absence of data might be an issue in non-linear models, where the ALE plot is interpolating and hence results might be unreliable in such regions. **Observation 6:**_As expected, explaining how a predictive model uses a feature in predicting outcomes of an event log with class imbalance does not reveal a lot of information._  
This observation is confirmed in results of the three XAI methods, especially when explaining LR reliance on important features. Results of event log _Sepsis1_ shown in Figures 6-8 represent an example of this observation. In Figure 6, despite the plotted features are indicated as important to the predictive models, after shuffling feature values over 10 iterations, there is no observed effect on the prediction error. In turn, in Figure 7(1-a),(2-a), the feature is showing no contribution to change in XGBoost predictions.