# Predictive Business Process Monitoring Approach Based on Hierarchical Transformer
## 1 Introduction
As event logs are essentially sequential data, the Recurrent Neural Network (RNN) [8], which is used to process sequential data, is an early deep-learning model for business process prediction. RNN can consider the previous output and apply it to the current output. In addition, Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), solving the problem of gradient disappearance and explosion in RNN, can better capture longer information dependencies in sequential data and have shown to perform well in business process prediction [9; 10; 11]. Transformer is a new type of model that has emerged in deep learning in recent years. It introduces multi-head attention, positional encoding, residual connection, layer normalization, and other mechanisms based on the self-attention model [12]. It solves the problems of recurrent neural networks that are difficult to run in parallel and convolutional neural networks with a restricted field of perception, and it has been used in natural language processing [13], recommender systems [14], computer vision [15], and many other tasks to achieve state-of-the-art results. The success of the Transformer in several fields has also attracted the attention of researchers in the field of business process prediction. [PERSON] et al. [16] were the first to apply the Transformer to business process prediction, and the basic idea is to compare a business process instance to a sentence in natural language processing, using it as input to the Transformer and the subsequent execution state of the instance as the model output. Overall, research related to the Transformer in business process prediction is relatively rare, and most of the existing work is limited to the direct application of the standard Transformer model architecture. We can argue that the potential of Transformer in business process prediction has not been well explored. To this end, this paper will focus on Transformer-based business process prediction and aims to improve the performance of various business process prediction tasks. Firstly, the business process instance data recorded in the event log not only consists of sequences of activities, but each activity also has different attribute information when it is executed, which is a feature important for accurate business process prediction [17; 18]. Transformer-based business process prediction models do not yet deal well with the relationshipsbetween activities and their various types of attribute information. Secondly, realistic business processes often have phenomena such as conceptual drift during execution, and the backend business process models supporting business systems are unstable, making business process instances reflect different process model characteristics [19]. Concept drift can lead to different business process instance distributions. Various machine learning methods, including deep learning, are often based on the assumption of independent and homogeneous data distribution, not taking into account the effect of concept drift on the final result, so concept drift can reduce the accuracy of business process prediction [20]. The existing data-driven business process models do not match the processes where conceptual drift occurs, and outdated models lead to inaccurate predictions of future results. Furthermore, Transformer requires more time and computational resources for model training for its higher time complexity, which to some extent limits the effective application of Transformer-based business process prediction models, although it has a greater advantage over various RNN models in terms of accuracy for various tasks. Based on the above considerations, a new business process prediction model, the Hierarchical Process Transformer (HiP-Transformer for short), is proposed. The basic idea of this model is to design a hierarchical Transformer model structure to better solve the problems mentioned above in predictive business process monitoring. Specifically, the HiP-Transformer model consists of three main layers.