# Predictive Business Process Monitoring Approach Based on Hierarchical Transformer
## 3 Preliminaries and Methods
### Concept Drift Detection
In addition, the activity, as an important component of the event, has a more significant impact on subsequent tasks such as the next activity prediction, remaining execution time prediction, etc. Therefore, the activity encoding of the event will be explicitly used as another input in the second layer Transformer model, which can be expressed formally as \(\mathbf{A}=\langle\mathbf{a}_{s},\mathbf{a}_{s+1},\ldots,\mathbf{a}_{s+l-1}\rangle \in\mathbb{R}^{l\times d}\). To fully exploit the intrinsic relationship between event and activity, this paper proposes a cross-self-attention mechanism. The basic idea is to use event and activity encoding as queries, respectively, calculate the weights of each other's encoding and then fuse them in a weighted manner. First, to model the sequence relationship between the events in the sub-sequence, position encoding is added to the event and activity encodings, specifically using the traditional Sinusoidal absolute position encoding \(\mathbf{P}\in\mathbb{R}^{l\times d}\) to obtain \(\mathbf{A}^{\prime}=\mathbf{A}+\mathbf{P}\) and \(\mathbf{M}^{\prime}=\mathbf{M}+\mathbf{P}\), where each element in \(\mathbf{P}\) is set to a fixed value [12] as follows. \[\mathbf{P}_{t,i}= \begin{cases}\sin\Bigl{(}10000^{-\frac{i}{d}\cdot t}\Bigr{)}\ \text{if}\ i\ \text{is even number}\\ \cos\Bigl{(}10000^{-\frac{i}{d}\cdot t}\Bigr{)}\ \text{if}\ i\ \text{is odd number}\end{cases}  \tag{23}\]  
After that, the event encoding matrix and the activity encoding matrix are linearly converted to obtain:  
\[\mathbf{Q}^{(A)}=\mathbf{A}^{\prime}\mathbf{W}^{(Q-A)},\ \mathbf{K}^{(M)}=\mathbf{M}^{\prime}\mathbf{W}^ {(K-M)},\ \mathbf{V}^{(M)}=\mathbf{M}^{\prime}\mathbf{W}^{(V-M)} \tag{24}\]  
\[\mathbf{Q}^{(M)}=\mathbf{M}^{\prime}\mathbf{W}^{(Q-M)},\ \mathbf{K}^{(A)}=\mathbf{A}^{\prime}\mathbf{W}^ {(K-A)},\ \mathbf{V}^{(A)}=\mathbf{A}^{\prime}\mathbf{W}^{(V-A)} \tag{25}\]  
Each of the \(\mathbf{W}\) matrices in the above equation represents a linear transformation weight matrix, taking \(\mathbf{W}^{(Q-A)}\) as an example, with \(Q\) denoting the query transformation used for this matrix and \(A\) denoting the transformation applied to the active encoding. Based on the self-attention mechanism shown in Equation (5), the following calculations are performed using each of the transformation matrices in the above equations, respectively. \[S^{(A\text{-}M)} = softmax\left(\frac{\mathbf{Q}^{(A)}\mathbf{K}^{(M)\text{-}T}}{ \sqrt{d^{(K)}}}\right)\mathbf{V}^{(M)} \tag{26}\] \[S^{(M\text{-}A)} = softmax\left(\frac{\mathbf{Q}^{(M)}\mathbf{K}^{(A)\text{-}T}}{ \sqrt{d^{(K)}}}\right)\mathbf{V}^{(A)}\] (27) \[S^{(A\text{-}A)} = softmax\left(\frac{\mathbf{Q}^{(A)}\mathbf{K}^{(A)\text{-}T}}{ \sqrt{d^{(K)}}}\right)\mathbf{V}^{(A)}\] (28) \[S^{(M\text{-}M)} = softmax\left(\frac{\mathbf{Q}^{(M)}\mathbf{K}^{(M)\text{-}T}}{ \sqrt{d^{(K)}}}\right)\mathbf{V}^{(M)} \tag{29}\]  
Each matrix \(S\) in the above equations represents the result of fusing the event encoding with the activity encoding, with each row corresponding to the fusion encoding of an event in the subsequence. Taking \(S^{(A\text{-}M)}\) as an example, \(A\) indicates that the fusion result uses the activity encoding as the query, and \(M\) indicates that the event encoding is fused. \(S^{(A\text{-}M)}\), \(S^{(M\text{-}A)}\) use different encoding as query and key value, so it can be considered as the fusion result of cross-self-attention, where \(S^{(A\text{-}A)}\), \(S^{(M\text{-}M)}\) are the fusion results of traditional self-attention. Next, these four fusion results will be further fused, specifically using a maximum pooling as follows:  
\[\widetilde{S}=max\_pool\left(S^{(A\text{-}M)},S^{(M\text{-}A)},S^{(A\text{-}A )},S^{(M\text{-}M)}\right) \tag{30}\]  
where \(max\_pool\) maximizes each row of the specified parameter matrix separately, such that \(\mathbf{S}^{(sub)}\in\mathbb{R}^{l\times d^{\prime}}\), with \(d^{\prime}\) denotes the dimension of the fused encoding vector.