# POP-ON: Prediction of Process Using One-Way Language Model Based on NLP Approach
## 3 Pop-On
### Proposed Prediction Model  
We propose a new model modified from GPT-2, which is a one-way language model-based transformer approach in process prediction. It recognizes the name attribute of a  
Figure 4: Example of using one-hot encoding. process event as a kind of natural language. GPT-2, which showed excellent results in NLP, had difficulties in predicting the process, and the experiment was successfully conducted only after various attempts. The name of the process prediction model is POP-ON (POP-ON: Prediction of Process Using One-way Language Model based on NLP Approach). Figure 5 shows an overview of the POP-ON model. \(n_{layer}\) is the number of transformer decoder layers to be stacked, \(bs\) is the batch size, \(n_{attr}\) is the number of input attributes, and \(n_{seq}\) is the length of the input sequence. Like GPT, the structure of a multi-layer transformer decoder was basically used. The basic structure of GPT is a multi-layer stack of Transformer Decoder including Input Embedding Layer, positional encoding, and layer normalization, plus Linear Layer and softmax layer [22]. POP-ON has a structure in which Attribute Linear Layer (ALL) to be described later is added to each Transformer Decoder Layer of the GPT structure. ALL allows the attribute to be used in the next event prediction along with the sequence. Figure 5: POP-ON (Prediction of Process Using One-way Language Model based on NLP Approach) model architecture. The Transformers model uses positional encoding to add information about the position of a word in a sequence. Positional encoding has a dimension of \(d_{model}\) and is added to the input embedding to go to the transformer decoder layer [25]. Attention functions receive inputs as queries, keys, and values. By calculating the dot-product of the query and the keys, the similarity between query-keys is obtained, whereas Scaled Dot-Product Attention is obtained by multiplying the values by values from the softmax function after scaling, where the higher the value is, the more similar it is to the query. This means that 'attention' can be given to important information of the sequence. The attention function is equivalent to Equation (4) [25]:  
\[Attention(Q,\ K,\ V)=softmax\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V \tag{4}\]  
\(Q\) and \(K\), correspond to the \(d_{k}\) dimensional matrix, and \(V\) is the \(d_{v}\) dimensional matrix. \(K^{T}\) is the transposition matrix of \(K\). In the Transformers model, Multi-Head Attention is used. Multi-Head Attention performs linear projection \(n_{head}\) times on \(Q\), \(K\), and \(V\) instead of using one attention. The projected \(Q\), \(K\), and \(V\) output the \(d_{v}\) dimensional output through the attention function in parallel. Then, each head is concatenated and re-projected, and finally the \(d_{model}\) dimension output is obtained. The Multi-Head Attention Layer used in the Transformer Decoder Layer is Masked Multi-Head Attention. When the attention of the \(i\)th position of the sequence is obtained, the position of the position after \(i\)th is masked and attention is not given to the position. MultiHead Attention is obtained as shown in Equations (5) and (6) [25]. The trainable weight matrices are \(W_{i}^{Q}\), \(W_{i}^{K}\), \(W_{i}^{V}\in\mathbb{R}^{d_{model}\times d_{k}}\), and \(W_{i}^{O}\in\mathbb{R}^{n_{head}d_{k}\times d_{model}}\). \[MultiHead(Q,\ K,\ V)Concat=\left(\text{head}_{1},\ \dots,\text{head}_{n_{head}} \right)W^{O} \tag{5}\]  
\[where\ \text{head}_{i}=Attention\left(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}\right) \tag{6}\]  
The ALL is a fully connected FFN (Feed-Forward Network) that uses the Attribute input of a dataset as input. For the Attribute input vector \(x\), the output of the ALL is shown in Equation (7):  
\[ALL(x)=max(0,((xW_{1}+b_{1})W_{2}+b_{2})W_{3}+b_{3}) \tag{7}\]  
Trainable weights are \(W_{1}\in\mathbb{R}^{n_{attr}\times d_{attr}}\), \(W_{2}\in\mathbb{R}^{d_{attr}\times d_{attr}}\), and \(W_{3}\in\mathbb{R}^{d_{attr}\times d_{model}n_{seq}}\).