# POP-ON: Prediction of Process Using One-Way Language Model Based on NLP Approach
## 4 Performance Analysis
### The Need for an Attribute Linear Layer  
The key-point of POP-ON is to learn the overall structure of the sequence and add ALL to the structure of the GPT to predict the next event. Through this, the attribute information can be reflected in the next event prediction. Using the BPI 2013 Closed Problems data set, we compared the model performance according to the presence or absence of ALL of the POP-ON model. The BPI 2013 Closed Problem data set is a real life data set provided by Volvo IT, and is an event log for problem handling of IT services provided/operated by the company. The problem handling process works with other processes such as problem monitor service, discover, and define opportunity. Therefore, accurately predicting the problem handling process using event logs can reduce disruption to business and improve the availability and quality of IT services. The BPI 2013 Closed Problems consists of 6660 events and 12 columns. Excluding the case name column, event name column, and timestamp column, 9 categorical attributes can \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
\begin{tabular}{c} **Hyperparameter** \\ **Set** \\ \end{tabular} & \(n_{layer}\) & \(n_{head}\) & \(d_{model}\) & \(d_{attr}\) & \(d_{ff}\) & **Accuracy** \\ \hline \multirow{4}{*}{\(K_{1}\)} & 1 & 1 & 16 & 128 & 128 & 0.8296 \\  & 2 & 1 & 16 & 128 & 128 & 0.8593 \\  & 4 & 1 & 16 & 128 & 128 & 0.8984 \\  & 8 & 1 & 16 & 128 & 128 & 0.9064 \\ \hline \multirow{4}{*}{\(K_{2}\)} & 1 & 2 & 32 & 256 & 256 & 0.8934 \\  & 2 & 2 & 32 & 256 & 256 & 0.9039 \\  & 4 & 2 & 32 & 256 & 256 & 0.9121 \\  & 8 & 2 & 32 & 256 & 256 & 0.9169 \\ \hline \multirow{4}{*}{\(K_{3}\)} & 1 & 4 & 64 & 512 & 512 & 0.9051 \\  & 2 & 4 & 64 & 512 & 512 & 0.9110 \\  & 4 & 4 & 64 & 512 & 512 & 0.9171 \\  & 8 & 4 & 64 & 512 & 512 & 0.9194 \\ \hline \multirow{4}{*}{\(K_{4}\)} & 1 & 8 & 128 & 1024 & 1024 & 0.9125 \\  & 2 & 8 & 128 & 1024 & 1024 & 0.9196 \\ \cline{1-1}  & 4 & 8 & 128 & 1024 & 1024 & 0.9197 \\ \cline{1-1}  & 8 & 8 & 128 & 1024 & 1024 & 0.9217 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Experiment results according to the hyperparameter settings. \begin{table}
\begin{tabular}{c c} \hline \hline \
\begin{tabular}{c} **Hyperparameter** \\ \end{tabular} &
\begin{tabular}{c} **Description** \\ \end{tabular} \\ \hline \(d_{model}\) & Dimension of Embedding \\ \(d_{ff}\) & Dimension of Feed-Forward Network \\ \(d_{attr}\) & Dimension of Attribute Linear Layer \\ \(n_{-}\)layer & Number of Transformer Decoder Layers \\ \(n_{-}\)head & Number of Attention Head \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Description of hyperparameter. be used. When 'time:elapsed' attribute is extracted from the timestamp, 10 attributes can be used. Since there are many attributes that can be used, and among them, there are attributes that are estimated to have a significant effect such as impact and product, we selected the BPI 2013 Closed Problems for the effective analysis of ALL. The learning trend and test performance were compared based on five metrics when the attributes were entered in ALL and when attributes were not entered. Figure 9 compares the test performance of the model according to the presence or absence of a linear attribute layer and training epoch. Among all epochs, the model with ALL resulted in the best test accuracy of 0.7836, and the model without ALL achieved the best test accuracy 0.4647. The hyperparameter used for learning was the same as Equation (10), where the batch size was 64, number of epochs was 200, and the learning rate was \(10^{-6}\). \[n_{layer}=8,\ n_{head}=8,\ d_{model}=128,\ d_{attr}=1024,\ d_{ff}=1024 \tag{10}\]  
Figure 9: Model test performance comparison according to the attribute linear layer. Figure 9 compares the test metrics of two models according to the training epoch. As a result of the comparison, the former model showed better performance in all five metrics used for evaluation than the latter model.