# CloudformerV3: Multi-Scale Adapter and Multi-Level Large Window Attention for Cloud Detection
## 3 Materials and Methods
### Multi-Level Large Window Attention and Decoder  
Detecting thin clouds in remote sensing images is frequently intricately linked to the surrounding context. The employment of multi-level feature maps extends a broader array of contextual insights, equipping the model with an enhanced capacity to comprehend the intricate interplay between thin clouds and their neighboring features. Consequently, this advanced comprehension facilitates a more precise identification of thin clouds. Based on the above reasons, this paper designs a novel decoder in conjunction with the multi-level large window attention, so that the high-level feature maps and the low-level feature maps can pay attention to each other and merge with each other.  
Inspired by the concept of large window attention proposed in the Lawin Transformer [32], this paper introduces a multi-level large window attention. In the context of the large window attention, a feature graph is divided into uniformly sized windows, allowing each window denoted as Q to query a larger region represented by C. However, within the framework of the multi-level large window attention, the low-resolution feature map is segmented into \(n\times n\) windows to replace the query window Q, while the high-resolution feature map undergoes a similar partitioning into \(n\times n\) windows to substitute the queried region C. This transforms the operation of focusing on a single feature map into the interaction between feature maps at different levels, as illustrated in Figure 4. As the window slides across the low-resolution feature map, the current segment is permitted to query the corresponding region within the high-resolution map, encompassing more pixel information. This capability enables the decoder to establish connections between feature maps at varying levels, thereby furnishing the model with a more extensive and intricate contextual understanding.  
Figure 3: The structure of DownSample Layer. Bule represents the extractor, pink represents the downsample layer and yellow represents the injector.  
In the decoder, the multi-level feature maps of 1/4, 1/8, 1/16, and 1/32 of the input image size in length and width obtained after passing through the encoder are passed through the multilayer perceptron, respectively, to obtain \(F_{1/4}\in\mathbb{R}^{H/4\times W/4\times D_{1}}\), \(F_{1/8}\in\mathbb{R}^{H/8\times W/8\times D_{2}}\), \(F_{1/16}\in\mathbb{R}^{H/16\times W/16\times D_{3}}\), and \(F_{1/32}\in\mathbb{R}^{H/32\times W/32\times D_{4}}\). In this paper, five parallel branches are designed in conjunction with multi-level large window attention, including a skip connection, a branch pooling \(F_{1/32}\) to obtain \(\hat{F}_{pool}\in\mathbb{R}^{H/32\times W/32\times D_{4}}\), and the three branches will be \(F_{1/16}\), \(F_{1/8}\), and \(F_{1/4}\), respectively, with \(F_{1/32}\) in multi-level large window attention to obtain \(\hat{F}_{1/4}\in\mathbb{R}^{H/32\times W/32\times D_{4}}\), \(\hat{F}_{1/8}\in\mathbb{R}^{H/32\times W/32\times D_{4}}\), and \(\hat{F}_{1/16}\in\mathbb{R}^{H/32\times W/32\times D_{4}}\). All the obtained features are stitched together to obtain a feature map with multi-level information. The process can be formulated as:  
\[\hat{F}_{pool}=Pooling(F_{1/32}) \tag{4}\]  
\[\hat{F}_{i}=Lavin\,Atten(F_{i},F_{1/32}),i\in\{1/4,1/8,1/16\} \tag{5}\]  
\[C_{1/32}=Concat(\hat{F}_{1/4},\hat{F}_{1/8},\hat{F}_{1/16},\hat{F}_{pool},F_{1 /32}) \tag{6}\]  
\(C_{1/32}\) is upsampled layer by layer through the three feature fusion modules and in the process fused with the high-level feature maps \(F_{1/16}\), \(F_{1/8}\), and \(F_{1/4}\), respectively, spliced to recover the image resolution to obtain \(C_{1/4}\). Finally, the result is output after passing through the multilayer perceptron and upsample layer.