## 4 Dataset Creation of the Rock Tunnel Construction
### Transformer Model Structure  
The constructed transformer model has two encoder layers and two decoder layers (Figure 6). The sequence length of the input encoder is a time series composed of measurement points with two dimensions (the crown settlement and the horizontal convergence of a transverse section). The predicted value output by the decoder has the same measurement points as the inputs with one dimension, which is a time sequence consisting of the monitoring deformation values at the tunnel face. To let the encoder layer learn the rich data information in the input sequence, the transformer model upgrades the features of the input sequence from a low dimension to a high dimension in the input embedding. In this study, the parameter dimension of the input time-series multi-head attention block is upgraded from 2 to 18, and the sequence length is selected as 30. The number of hidden layer units in the feed-forward network module (FFN) is 10. Since the feature dimension is not large, the number of heads in the multi-head attention mechanism is set to three. The decoder layer has a similar structure to the encoder layer, which has an output dimension of one. Using the same approach as shown in the attention-enhanced LSTM, the outputs of the decoder layer are then processed through the linear layer and sigmoid activation function. The output prediction should have the same sequence length as the input. An essential feature in the constructed transformer model is the positional embedding, widely used in the BERT (Bidirectional Encoder Representations from Transformers) model [31; 32] for pre-training bidirectional representations for language understanding. The positional embedding is designed and placed before the encoder but not included in the decoder stage. The positional embedding assigns tags denoting the different process  
Figure 5: Model structure of the attention-enhanced LSTM. Figure 6: Model structure of the transformer. stages of rock tunnel construction, which can adjust the positional encoding amplitude at various construction stages. As a result, the proposed transformer model can incorporate the process information into the model's input and hide the process information from the output. This study's proposed positional tags represent different process stages in rock tunnel construction. For ease of recognition, the excavation-slagging-stand stage is represented by the tag value of one, the advanced support stage is represented by the tag value of two, and the shotcrete lining stage is represented by the tag value of three. The site engineer manually records each construction stage period so that the monitoring data can align with the reordered time to assign the positional tags. The computed inputs (crown settlements and the horizontal convergence values) corresponding to each construction stage for a specific monitoring period are illustrated in Figure 7 with different tag values. It can be seen that the assigned tags can differentiate the deformation values with similar magnitude, which informs the transformer model with additional site information. Hence, the transformer model can know the construction process, whereas the attention-enhanced LSTM model only learns the data features from the monitoring sensors. Based on the assigned positional tags, the amplitude of the positional encoding should be modified to fit the application before the encoder model. By adjusting the amplitude, the position tags of the input data are integrated to preserve the process information in a long time series. In this study, the positional encoding value can be computed as follows:  
\[\text{PE}(pos,2i)=tag\times\sin(\frac{pos}{1000^{2/d_{k}}}) \tag{4}\]  
\[\text{PE}(pos,2i+1)=tag\times\cos(\frac{pos}{1000^{2/d_{k}}}) \tag{5}\]  
where the _tag_ value corresponds to the construction stages mentioned above, _pos_ is the position of the data in the entire input sequence at a specific moment, \(i\) represents the dimension, and _d_\({}_{k}\) is the feature dimension of the model.