# Predictive Business Process Monitoring Approach Based on Hierarchical Transformer
## 1 Introduction
The first (bottom) layer is the event modeling layer, which uses the multi-head self-attention mechanism in the Transformer to model the activity of each event and its attribute information in a process instance. The second (middle) layer is the event sequence fragment modeling layer, of which the goal is to explore intra-fragment correlations and inter-fragment dependencies. The whole event sequence corresponding to the process instance is segmented based on the concept drift detection method to obtain a relatively stable event subsequence of process features. The process features within each event subsequence is then captured using cross-attention. The third (top) layer is the process instance modeling layer, where the goal is to further model the entire event sequence of a process instance using Transformer on top of the event subsequence modeling to obtain the representation of the process instance. In addition, for two representative business process prediction tasks, namely next activity prediction and remaining time prediction, the corresponding decoding modules are designed in the HiP-Transformer model to predict the subsequent states of the business process instances, respectively. In this paper, experiments are conducted on several real business process datasets publicly available at the 4 TU Center for Research Data. The results show that the proposed HiP-Transformer model achieves further improvement over traditional Transformer-based models for the two business process prediction tasks. We refer to the benchmark [2] of the deep learning model to validate the validity of the model. Our method is relatively more accurate than most deep learning-based business process prediction models, showing a significant advantage. The main contributions of this paper can be summarized as follows:  
* We propose a novel model called HiP-Transformer, which infers accurate business process representations with three perspectives on the associated features. The internal associations between activities, events, and sub-sequences in the business process make up the comprehensive representation. * We solve the problem of conceptual drift in long business processes. We capture the dependencies of preceding and following subsequences with conceptual drift behavior, which has a higher prediction accuracy and lower time complexity than traditional segmentation strategies. * Comprehensive experiments on seven public datasets demonstrate the effectiveness of our HiP-Transformer model. Compared with deep learning methods, including LSTM [10], GRU [11], CNN [21], DNC [22], and Process Transformer [16], the next activity prediction accuracy improves by 6.32% compared to the average accuracy of the above models, and the average absolute error of remaining time prediction reduces by 21% compared to the average accuracy of the above models. The rest of this paper is organized as follows: Section 2 reviews the related work on business process prediction research. Section 3 describes the problem to be solved formally and proposes the HiP-Transformer method. Section 4 shows the experimental results and analysis. Section 5 summarizes the work of this paper and describes future work.