# A Novel Method for Ground-Based Cloud Image Classification Using Transformer
## 2 Method
### Transformer Architecture  
#### 2.3.1 Vision Transformer  
Vision Transformer (ViT) [39] was the first model to apply Transformer to train large-scale image datasets, and the overall structure is shown in Figure 3a. The input of Transformer is a one-dimensional (1D) sequence token embedding, so it needs to be processed for two-dimensional (2D) images. Firstly, the image, \(\mathrm{X}\in\mathrm{R}^{\mathrm{H}\times\mathrm{W}\times\mathrm{C}}\), is divided into a number of nonoverlapping patches, \(\mathrm{X}_{\mathrm{P}}\in\mathrm{R}^{\mathrm{P}\times\mathrm{P}\times\mathrm{C}}\), where \(\mathrm{H}\times\mathrm{W}\) is the size of the original image, \(\mathrm{C}\) is the number of channels in the image, and \(\mathrm{P}\times\mathrm{P}\) is the size of each patch. Then, each patch is linearly projected into a vector of fixed dimensions using a trainable embedding matrix. To preserve the spatial information in the image, position embedding is added to each embedding vector to collectively form the input of the transformer encoder. In addition, an extra class embedding is also fed into the encoder to distinguish images from different classes.  
Figure 2: The structure of Mobile Inverted Bottleneck Convolution (MBConv).  
The structure of the transformer encoder is shown in Figure 3b. It is mainly composed of a multi-head self-attention (MSA) module and a feed-forward multilayer perceptron (MLP) [33]. MSA is the core module of the encoder, which will be explained in detail below. MLP contains two fully connected layers with a Gaussian Error Linear Unit (GeLU) activation function between the layers [51]. In addition, the two parts of the encoder both use residual connections, while a normalization layer is added before the input of each part. Finally, GCI classification is performed using the MLP head based on the features trained by the transformer encoder.  
Multi-head self-attention (MSA) [33] is the core component of Transformer, and the introduction of an attention mechanism can make the network pay more attention to the relevant information in the input vector. Figure 4b provides an illustration of MSA. It is composed of multiple self-attention connections, and the structure of self-attention is shown in Figure 4a. The input vectors are first transformed into three different vector matrices: query matrix, \(Q\); key matrix, \(K\); and value matrix, \(V\). The weight assigned to each value is determined by calculating the dot product of the query matrix and the key matrix of different input vectors so that the individual input vectors can be connected to each other to achieve the effect of global modeling. The specific calculation equation is:  
\[Attention(Q,K,V)=Softmax\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)\cdot V \tag{1}\]  
where \(d_{k}\) is the dimension of matrix \(K\). The purpose of dividing by \(\sqrt{d_{k}}\) is to provide proper normalization to make the gradient more stable. In order to generate more correlations between different inputs, the input vector is split into several parts by the multi-head self-attention mechanism, which then computes the matrix dot product of each part in parallel and then finally concatenates all the attention output results. The calculation processes are Equations (2) and (3), which are expressed as:  
\[head_{i}=Attention\left(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}\right) \tag{2}\]  
\[MSA(Q,K,V)=Concat(head_{1},\dots,head_{i})W^{o} \tag{3}\]  
Figure 3: The structure of (**a**) the Vision Transformer (ViT) and (**b**) the transformer encoder.  
where \(i\) denotes the number of input vector splits and \(W_{i}^{Q},W_{i}^{K},W_{i}^{V}\), and \(W^{o}\) are all trainable parameter matrices.