## 5 Evaluation
### Results and Discussion  
The objective of the evaluation of the proposed approach is two-fold. First, our aim is to appraise whether the performance of the ARM-based classifier is at least comparable with the one of other ML-based approaches in outcome-based predictive process monitoring. If that were not to be the case, in fact, it would not make sense to employ ARM-based classifiers in this prediction task, even if the model obtained were intrinsically explainable. Second, our aim is to demonstrate the applicability of the proposed local and global level ARM-based model visualisation to real world event logs.  
Regarding the first objective, the performance of the proposed ARM-based classifiers compared against one of ML-based classifiers is shown in Figure 2. Note that the figure shows the value of different performance measures commonly used for classification tasks by prefix length.  
First, we note that, when the AUC measure is considered, the performance of the ARM- and ML-based classifier is comparable at all prefix lengths and for all datasets, with the only exception of the BPIC 2011 event log at a higher prefix length. AUC is the principal measure of performance normally considered by previous work in outcome-based predictive monitoring [10]. As far as precision is concerned, RIPPER achieves, in general, high precision on all event logs. In some cases (BPIC 2011 and BPIC 2012 event logs at low prefix lengths), the precision achieved by RIPPER is higher than ML-based models. This is also true for the BPIC 2017 event log. However, RIPPER achieves, at the same time, consistently lower accuracy with this event log.  
For some event logs (BPIC 2012, but most evidently BPIC 2017), the recall achieved by RIPPER is low, which affects, also, the F-score performance. For the BPIC 2011 event log, the performance of RIPPER is comparable with ML-based classifiers only at low prefix lengths (up to 20). These peculiarities of the results may be due to intrinsic characteristic of the datasets and/or the business process generating them. For instance, the BPIC 2012 and 2017 event logs are extracted from the execution of the same business process (loan requests at a financial institution), but using two different information systems to support the execution (this system has been upgraded before the BPIC 2017 event log was captured).  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**\# Model** & **\# Parameter** & **\# Range** \\ \hline RF & n estimators & \([100,200,300]\) \\  & max features & \([\text{auto},\text{sqrt}]\) \\  & max depth & \((10,1000)\) \\  & min samples split & \([2,6,10]\) \\  & min samples leaf & \([14,47]\) \\  & bootstrap & [True, False] \\ \hline XGB & n estimators & \([100,200,300]\) \\  & learning rate & \([0.1,0.3,0.5]\) \\  & subsample & \([0.3,0.5,0.7]\) \\  & max depth & \([0,1000]\) \\  & colsample bytes & \([0.5,0.6,0.7]\) \\  & min child weight & \([1,2,3,4]\) \\ \hline RIPPER & prune size & \([0.33,0.5]\) \\  & k & \([1,2]\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Search space of hyper-parameters for the models used in the evaluation.  
Based on the results shown in Figure 2, in particular the analysis of the AUC results, we can claim that the ARM-based classifier RIPPER is not particularly worse than RF or XGB for this classification task. Therefore, decision-makers may decide to choose using an ARM-based classifier, since the model obtained is intrinsically explainable, as demonstrated by the proposed visualisations discussed next.