# Composite Foundation Settlement Prediction Based on LSTM-Transformer Model for CFG
## 3 The Establishment of a CFG Compound Foundation Settlement Model Based on LSTM-Transformer  
### The Transformer Model  
The Transformer is a network that leverages attention mechanisms to enhance the speed of model training. Its encoder component resembles a large multilayer perceptron (MLP) [28], where data can be input simultaneously and processed in parallel. Its remarkable capability in handling long sequential data makes it effective for predicting settlement. Moreover, it addresses the issue of gradient explosion efficiently, rendering it suitable for settlement prediction tasks.  
The Transformer relies entirely on self-attention mechanisms to extract inherent features. It can learn long-term dependencies and is more easily parallelizable compared to RNN-type models. The original Transformer was used in classification tasks, obtaining high-dimensional features for individual time steps through an embedding layer. It uses positional encoding to retain the positional information from the original sequence [27; 29]. Subsequently, based on the query-key-value concept, the self-attention layer maps the original features into three components: Q, K, and V. It computes the correlation between features at different time steps and the current moment using Q and \(K\), then derives the fea  
Figure 1: The overall framework of the LSTM–Transformer-based foundation prediction model.  
tures for the next time step through a weighted sum of \(V\). The self-attention mechanism's computational formula [30] in the Transformer is shown as follows:  
\[M(Q,K,V)=J(hi,\dots.hh)W \tag{1}\]  
\[hi=A(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}) \tag{2}\]  
\[A(Q,K,V)=softmax(\frac{QK}{\sqrt{d_{k}}})V \tag{3}\]  
In the equations, \(M(Q,K,V)\) represents the output sequence of multi-head attention; \(d_{k}\) denotes the dimension of (\(Q\)), (\(K\)), and (\(V\)); \(\sqrt{d_{k}}\) is the scaling factor; \(W\), \(W_{i}^{Q}\), \(W_{i}^{K}\), and \(W_{i}^{V}\) are matrices for linear transformations; \(Q\) represents the query calculation vector; \(K\) represents the vector for which the query is computed; \(V\) denotes the current actual features; \(A(Q,K,V)\) represents the attention weights, obtained through scaled dot-product output vector sequences.  
The multi-head self-attention mechanism is an improvement of the self-attention mechanism. It divides the self-attention into multiple parallel heads that perform linear mappings multiple times. It calculates attention for different mapping results, processing different levels of information in the input sequence and computing global attention. Essentially, it is an integration of multiple independent self-attentions. The Transformer not only provides one of the predictions for foundation settlement through the self-attention mechanism algorithm, but its multi-head attention mechanism can also repeat the algorithm multiple times to obtain more balanced and goal-aligned results. During the training and prediction process, the Transformer internally learns and trains with the provided data, effectively extracting intrinsic information from the data. In predictive training, it can mask the original data to enhance the persuasiveness of the algorithm's predictions. Figure 2 shows the structural diagram of the multi-head attention mechanism.