# A Goal-Oriented Evaluation Methodology for Privacy-Preserving Process Mining
## 4 Case Study
### PPPM Analysis and the Derivation of Answers to Questions
#### 4.5.7 PPPM Analysis of DP-Decision Trees (Q2.1.2)  
Result Utility Quality  
Figure 15 shows the DP-Decision Trees model (test) accuracy metric with respect to the epsilon parameter values behavior on our real-life event logs. The results show a decreasing trend in accuracy values according to increase in the privacy consideration for all the event logs. However, due to the nondeterministic nature of differential privacy, random deviations between different epsilon privacy budget parameters can occur, especially parallel with the instance count of the event logs. For supervised learning algorithms, the instance count is an important consideration for the model's accuracy. In our experiments, while the Road Traffic Fines event log has the highest number of events, there is a dramatic reduction (around 60%) in the model's accuracy after differential privacy protection.  
Additionally, to enhance model interpretability and understand the underlying decision logic, decision trees for trained DP-Decision Trees models are visualized with nodes color-coded according to their majority class, facilitating a clearer representation of the decision paths. As an example, Figure 16 illustrates this situation for the Road Traffic Fines event log before and after applying the DP-Decision Trees model. The resulting tree exhibited a structurally compact form, characterized by a reduced number of nodes and shallower depth. However, this simplification incurred a trade-off in predictive performance, as reflected in the lower accuracy metrics observed in the test set. This trade-off aligns with the well-documented bias-variance dilemma in machine learning, wherein simpler models generally demonstrate higher bias and lower variance. While this reduces the likelihood of overfitting, it may also result in underfitting, thereby limiting the model's ability to capture complex patterns in the data.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Event Log** & **Accuracy** & **Time** \\ \hline Sepsis & 0.81 & 0.0729990005493164 \\ BPIC13 & 1.00 & 0.07899904251098633 \\ CoSeLog & 0.59 & 0.095794677734375 \\ Road Traffic Fines & 1.00 & 0.5005433559417725 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Discovered (non-private) DP-Decision Trees model quality and runtime metric(s) results.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Event Log** & **Epsilon = 0.01** & **Epsilon = 0.05** & **Epsilon = 0.1** & **Epsilon = 0.5** & **Epsilon = 1** \\ \hline Sepsis & 3.386 & 0.814 & 0.666 & 0.483 & 0.474 \\ BPIC13 & 0.12163 & 0.06749 & 0.05439 & 0.04909 & 0.04636 \\ CoSeLog & 3.709 & 0.831 & 0.511 & 0.3260 & 0.3261 \\ Road Traffic Fines & 0.40613 & 0.38346 & 0.38048 & 0.373040 & 0.37363 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Discovered DP-Naive Bayes model runtime(s) metric results.  
Time Utility Quality  
Table 7 includes the runtime(s) performance metric results of the applied DP-Decision Trees model on our real-life event logs. During the experimentation with the nondeterministic DP-Decision Trees algorithm, slight fluctuations in model runtime were also observed as a consequence of specific epsilon parameter modifications. A proportional decrease in runtime was observed with increasing privacy levels, attributable to reduced model complexity. However, with the high number instance count of the Road Traffic Fines event log, a strong privacy level (i.e., lowest epsilon value) was gathered in a considerable amount of time, considering its respective scale. No statistically significant variation indicates stability in the measured values.