## 2 Related Work
### Transformer  
The Transformer neural network framework, rooted in the attention mechanism, excels in handling long-range dependencies [50]. The advent of the VIT in image processing has underscored its robust efficacy in remote sensing classification. The fundamental concept behind VIT involves leveraging the Patch Embedding module to partition the input image into a sequence of non-overlapping patches [26]. These patches are subsequently projected to craft a linear embedding sequence, serving as input to the Transformer for global feature extraction [51]. Notably, the Transformer usually comprises an encoder and a decoder, with our discussion focusing solely on the encoder segment for classification tasks. Each encoder layer encompasses two sublayers: the first sublayer involves the crucial multi-head self-attention mechanism, while the second sublayer features the feedforward layer aimed at deepening the network. Residuals are integrated into each sublayer to mitigate the vanishing gradient issue, complemented by the application of layer normalization [39].  
The incorporation of the self-attention mechanism empowers the Transformer to glean interdependencies between distinct locations within the input sequence, delivering heightened efficacy in processing lengthy sample sequences [50]. The computation of the self-attention mechanism comprises three steps. Initially, query vectors \(Q\), key vectors \(K\), and value vectors \(V\) are computed, followed by the computation of the inner product of each row of query and key vectors. Subsequently, attention scores are derived by normalizing the inner product results using the Softmax function. Ultimately, the self-attention vector is derived through the multiplication of the attention scores with the value vectors. For the specific calculation procedure can be expressed as follows (1) [52].  
\[\text{Attention }(Q,K,V)=\text{softmax}\bigg{(}\frac{QK^{T}}{\sqrt{d}}\bigg{)}V \tag{1}\]  
where, \(d\) is the dimension of the key vector, serving to prevent the inner product from becoming excessively large.  
The feed-forward layer stands out as a cornerstone component in neural networks. Typically comprising a linear transformation layer, an activation function, and a linear output transformation layer, the feedforward layer serves a pivotal role in model expressiveness and semantic feature extraction. The standard representation of the feed-forward layer is depicted in (2):  
\[FFN(x)=(\delta(xW_{1}+b_{1}))W_{2}+b_{2} \tag{2}\]  
where, \(W_{1}\) and \(b_{1}\) denote the weight coefficients and bias coefficients of the initial linear transformation layer, \(delta\) represents the activation function, and \(W_{2}\) and \(b_{2}\) denote the weight coefficients and bias coefficients of the output layer transformation. Apart from enriching the model's expressive capabilities and delving into deeper semantic features, the feedforward layers also contribute to alleviating the issue of vanishing gradients.  
LayerNorm emerges as a data normalization technique focused on the sample dimension. This normalization layer aids in confining data within a specific range to counteract the emergence of vanishing or exploding gradients [53]. In essence, the output of each sublayer can be characterized as \(LayerNorm(X+\sigma(X))\), where \(X\) denotes the input of each sublayer, and \(\sigma(X)\) signifies the corresponding functional transformation within each sublayer.