# Sequence Deep Learning for Seismic Ground Response Modeling: 1D-CNN, LSTM, and Transformer Approach
## 3 Prediction Models
### Transformer-Based Model  
#### 3.4.1 Overview  
Transformers [28] have significantly advanced the field of deep learning, particularly in sequential data processing based on a self-attention mechanism. Unlike RNN-based models, including LSTM, which may struggle to maintain information over long sequences due to the vanishing gradient problem [44], transformers employ a self-attention mechanism [28] that enables processing entire sequences of data in parallel. This capability allows them to consider the dependencies and relationships across the elements in sequence more efficiently, irrespective of their proximity.  
The self-attention mechanism (Figure 8) operates by comparing each element in the sequence (\(x_{t_{0}},~{}x_{t_{1}},~{}\ldots,~{}x_{t_{k}}\) in Figure 8) to every other element with a set of learnable parameters. This pair-wise operation returns the attention scores, which determine the relative importance between the elements. The attention scores are then used to weight the contributions of the other elements (\(x_{t_{1}},~{}\ldots,~{}x_{t_{k}}\)) to the target element (\(x_{t_{0}}\)). The summation of the weighted sequence forms a new context-aware representation of the target element, \(x_{t_{0}}\). This process allows the model to capture long-range dependencies within the sequence by focusing on relevant information from other elements with adaptive weighting based on learned importance in parallel. The entire operation entails the above process for all elements (\(x_{t_{0}},~{}x_{t_{1}},~{}\ldots,~{}x_{t_{k}}\)).  
Figure 7: LSTM-based model for seismic ground response estimation.  
By leveraging self-attention, transformers offer an efficient way to integrate contextual information, allowing the model to emphasize the most relevant parts of the data sequence for a deeper understanding of its sequential relationships.  
The transformer architecture consists of an encoder-decoder structure (Figure 9). The encoder (left side of Figure 9) processes the input sequence through multiple layers, each applying a self-attention mechanism followed by dense layers. The output from the encoder provides the contextual representation of the input sequence. The decoder (right side of Figure 9), also composed of multiple layers, generates the output sequence by utilizing two types of attention: self-attention to process the output sequence generated so far and encoder-decoder attention to integrate information from the encoded input sequence. Each layer in the decoder also includes a feedforward neural network. The layer normalization [47] and residual connections [48] are applied to both the encoder and decoder to enhance and stabilize the learning process.  
Moreover, multi-head attention is introduced. It extends the self-attention mechanism by using multiple attention heads. Each head learns different parts of the input, and their outputs are concatenated to form the final representation. This allows the model to capture  
Figure 8: Self-attention mechanisms (revised from [PERSON] [46]).  
Figure 9: Transformer encoder–decoder architecture (revised from [28]).  
various aspects of the data in parallel, leading to a richer and more nuanced representation of the sequence. For full technical details, refer to [PERSON] et al. [28].  
[PERSON] et al. [49] utilize a transformer for structural health monitoring, which integrates classification, localization, and segmentation tasks. The transformer network enhances the accuracy of processing images for detecting and assessing structural damages owing to the self-attention mechanism that learns and exploits interdependencies among multiple attributes and tasks in image processing. [PERSON] et al. [50] use a transformer to measure the vibration profiles of high-rise buildings from unmanned aerial vehicle (UAV)-recorded videos. The use of transformer networks helps mitigate the drifting issues induced by UAV movements, ensuring accurate full-field deformation measurements.