# XAI in the Context of Predictive Process Monitoring: An Empirical Analysis Framework
## 4 XAI Comparison Framework
### Framework Composition
#### 4.1.4 XAI Dimension  
We choose certain explainability methods (_how_), at both explainability levels (_how much_), presented in a certain form (_presentation_) at a certain stage of the predictive model lifetime (_when_). The XAI methods chosen in this work all fall under the _model-agnostic_ category. In turn, we conduct a complementary study [21] using _model-specific_ methods. To address the research questions of this study (cf. Section 3), we consider only model-agnostic XAI methods. Making choices in this dimension are impacted by certain factors:  
* Ability of the explainability method to overcome the shortcomings of other methods that explain the same aspects of the reasoning process of a predictive model. For example, Accumulated Local Effects (ALE) [12] adopts the same approach as in Partial Dependence Plots (PDP) method [6]. Unlike PDP however, ALE takes the effects of certain data characteristics (e.g., correlations) into account when studying features effects [4]. * Comprehensiveness regarding the explanation coverage when using both local and global explainability methods. Through local explanations the influence of certain features can be observed. In turn, through global explanations the reasoning process a predictive model has followed can be inspected. This approach allows reaching conclusions that may provide a holistic view of both the data and the model applied on the data. It is hard to find a single explainability method that provides explanations at both levels. However, one of the applied methods (i.e., SHAP [5]) starts at the local level by calculating contributions of the features on a prediction. SHAP aggregates these contributions at a global level to give an impression of the impact a feature has on the whole predictions based on a given dataset. * Availability of a reliable implementation of the explainability method. This implementation should enable the integration of the explainability method with the chosen predictive model as well as in the underlying PPM workflow. Table 2 categorizes each explainability method we apply in the experiments according to the dimensions of an explanation (cf. Section 2.2). The information in the _whom_ column which corresponds to the user groups dimension of the explanations, is case-dependent. Information given in this column is initial and is highly flexible according to several factors including users expertise, application domain and the purpose of the explainability experiment. XAI dimension constitutes two levels, which are realized using global and local experiments, based on the coverage of the explainability method applied, and the type of XAI method used. After executing each group of experiments, we compare the results of the different XAI methods at the level they are applied on. Global Explainability Analysis  
In this set of experiments, we aim to understand how a predictive model learns patterns from the training subset of the event log to which it is fitted. We use training \begin{table}
\begin{tabular}{c c c c c c c}
\begin{tabular}{c} Novice users, systems \\ practitioners, decision \\ makers \& experienced users \\ \end{tabular} \\ \hline \end{tabular}
\end{table} 
Table 2: Explainability dimensions applied on inspected XAI methods. subsets in order to understand how much the model relies on each feature for making predictions. This can be achieved by studying the change in model accuracy after modifying a certain feature value. To this end, training subsets are more qualified to provide insights into this aspect. As aforementioned, Permutation Feature Importance (PFI), ALE and SHAP (the global form) are the model-agnostic methods we used. To check stability of the executions, we run the whole comparison framework with different settings twice. Stability checks following this definition are expensive to run, due to expensive computational costs. These costs are affected by the number of datasets with different sizes and the number of explainability methods applied.