# POP-ON: Prediction of Process Using One-Way Language Model Based on NLP Approach
## 4 Performance Analysis
### Comparative Analysis  
There were research results in various fields using the BPI Challenge as a dataset, and there were use cases related to process prediction. The Approach in Table 3 provides a working list of different approaches used to validate our experiments. All of these tasks were performed on the BPI Challenge data set and applied a deep learning approach. In this section, we compare the prediction result values for each dataset by referring to Table 3 and show the superiority of the proposed POP-ON approach. Benchmark (accuracy) between POP-ON and existing deep learning-based process prediction studies was compared. The blue numbers in the table represent the best results for each dataset. The attributes used in each dataset are shown in Table 4. The numerical attribute 'time:elapsed' is an attribute extracted from the timestamp of the log, and is the time difference between the event and the timestamp of the previous event. Hyperparameter used Equation (9), where the batch size was 64, the number of epochs was 200, and the learning rate was \(10^{-6}\). \[n_{layer}=8,\ n_{head}=8,\ d_{model}=128,\ d_{attr}=1024,\ d_{ff}=1024 \tag{9}\] \begin{table}
\begin{tabular}{c c} \hline \hline
**Metrics** & **Formula** \\ \hline Accuracy & \(Accuracy=\frac{(Number\ of\ positive\ classifications))}{(Number of\ input\ sequences)} \\ Precision & \(\frac{1}{n}\sum_{i=1}^{l}S_{i}\frac{tp_{i}}{t_{i}+f_{i}}\) \\ Recall & \(\frac{1}{n}\sum_{i=1}^{l}S_{i}\frac{tp_{i}}{t_{i}+f_{i}}\) \\ F-measure & \(\frac{1}{n}\sum_{i=1}^{l}S_{i}\frac{precision\times recall}{precision+ recall}\) \\ MCC (Matthews Correlation Coefficient) & \(\frac{1}{n}\sum_{i=1}^{l}S_{i}\frac{t_{i}p_{i}\times tn_{i}-fp_{i}\times fn_{i}}{ \sqrt{(tp_{i}+fp_{i})(tp_{i}+fn_{i})(tn_{i}+fp_{i})(tn_{i}+fn_{i})}}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Evaluation indicators of multi-class classification models. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multicolumn{1}{c}{**Appron**} & **BPI 2012 A** & \begin{tabular}{c} **BPI 2012 W** \\ **Complete** \\ \end{tabular} & \begin{tabular}{c} **BPI 2012 O** \\ **Closed Problems** \\ \end{tabular} & \begin{tabular}{c} **BPI 2013** \\ **Incidents** \\ \end{tabular} & \
\begin{tabular}{c} **BPI 2013** \\ **BPI 2013** \\ \end{tabular} &
\begin{tabular}{c} **Helpdesk** \\ \end{tabular} \\ \hline [PERSON]. & [38] & 71.47 & 66.14 & 77.51 & 24.35 & 33.10 & 65.84 \\ [PERSON] et al. & [17] & 77.75 & 66.80 & 81.22 & 65.57 & 67.50 & 75.06 \\ [PERSON] et al. & [39] & 78.82 & 65.19 & 85.13 & 60.62 & 68.01 & 76.51 \\ [PERSON] et al. & [40] & 79.27 & 67.24 & 85.51 & 61.14 & 77.95 & **77.90** \\ [PERSON] et al. & [41] & 75.62 & 75.91 & 84.48 & 55.57 & 64.34 & 69.13 \\ [PERSON] et al. & [42] & 74.44 & 65.38 & 79.20 & 55.66 & 68.15 & 70.07 \\ [PERSON] et al. & [43] & 78.09 & 65.01 & 81.52 & 56.97 & 71.09 & 74.77 \\ POP-ON & & **79.49** & **81.36** & **92.15** & **78.36** & **79.89** & 75.53 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Benchmark (accuracy) between existing deep learning-based process prediction research and POP-ON. The bold numbers in the table represent the best results for each dataset. As an evaluation index, the evaluation was conducted based on the accuracy entered for each reference. As a result, POP-ON performed well on all datasets except Helpdesk. In particular, in the BPI 2012 A, BPI 2012 W Complete, BPI 2012 O, and BPI 2013 Closed Problem datasets, the POP-ON model was the best in terms of accuracy. The accuracy of the BPI 2012 A was slightly higher than most previous studies. Compared to the BPI 2012 W Complete and BPI 2012 O, it showed improvements of about 5.45% points and 6.64% points, respectively, compared to previous studies ([PERSON] et al. [41] and [PERSON] et al. [40]), which were the second best results in terms of accuracy. For the BPI 2013 Closed Problem dataset, compared to the second most accurate study ([PERSON] et al. [17]), our approach showed an improvement of about 12.79% points.