# A Novel Method for Ground-Based Cloud Image Classification Using Transformer
## 2 Method
### Transformer Architecture
#### 2.3.2 Swin Transformer  
Swin Transformer [52] is a transformer model recently proposed and is gradually becoming the backbone for various vision tasks. The original Transformer always keeps the same downsampling operation, and each patch needs to compute self-attention with all other patches. This would result in extracting only single-scale features and incurring extremely high computational costs when classifying high-resolution images. Based on the above reasons, Swin Transformer builds a hierarchical structure imitating the CNN model, as shown in Figure 5. The hierarchical representation is achieved by patch merging operations for each stage. The method sets the dimensionality by merging adjacent patches and two linear transformations, and the resolution of the feature map becomes larger and larger as the transformer layer deepens. In addition, the model uses nonoverlapping windows to compute self-attention, thus changing the computational complexity from quadratic to linear with the size of image resolution. However, this approach results in a lack of information interaction between windows, which prevents a better understanding of contextual information. Therefore, Swin Transformer offers the shifted window method, which enables the interaction of information between neighboring windows. This is also the main difference from the original transformer model. Figure 4: The illustration of self-attention in Transformer. (**a**) The self-attention; (**b**) the multi-head self-attention. Figure 5: Illustration of hierarchical feature maps. Swin Transformer provides four versions of this model: Swin-T, Swin-S, Swin-B, and Swin-L. Considering the specificity and computational complexity of GCI, we chose Swin-T, whose overall structure is shown in a part of Figure 1. The model is divided into four stages containing 2, 2, 6, and 2 blocks. Stage 1 contains two Swin Transformer blocks, and a linear embedding layer is added before the block to change the number of channels in the feature map. Stage 2 contains a patch merging layer and two Swin Transformer blocks, where patch merging acts like the pooling layer in CNN for downsampling before each stage to adjust the dimensionality of the feature map. The structure of stage 3 and stage 4 is similar to that of stage 2, except that the amount of downsampling multiplies, and the number of blocks is different. Different from the original MSA, the Swin Transformer block is constructed based on window-based multi-head self-attention (W-MSA) and shifted window-based multi-head self-attention (SW-MSA). W-MSA reduces the computational complexity of the model by computing self-attention within the window, but it lacks communication between the windows as a result. Therefore, SW-MSA is proposed to achieve the interaction between the windows by dividing and merging the feature maps. The specific operation is shown in Figure 6. First, the feature map is partitioned into a number of nonoverlapping windows and then attention between different patches in each window is calculated with W-MSA. Then, the window positions are cyclically shifted to form a new feature map. Next, the feature map is input into SW-MSA for the calculation of self-attention within the windows, thus achieving the effect of information interaction across windows. Finally, the feature map is reversed back to the original state for the next loop operation. Swin Transformer blocks always appear in pairs due to the abovementioned shifted window characteristics. Figure 7 shows the structure of two successive Swin Transformer blocks. The composition of each Swin Transformer block is almost the same as that of the ordinary transformer encoder; the only difference is that the _MSA_ in it is replaced by _W-MSA_ and _SW-MSA_.