## 4 Principle and Implementation
### Water Segmentation Model
#### 4.2.2 Encoder Design  
Transformer models are widely recognized as a highly successful approach in various fields, thanks to their superior performance and robustness [25]. In recent years, a growing number of Visual Transformer (ViT)-based models have emerged in the field of computer vision, indicating a shift in focus from traditional convolutional network architectures [20; 21; 22; 23; 24] to Transformer-based model architectures. Although the Transformer architecture is inherently capable of extracting global features using attention mechanisms, it lacks the inductive bias that traditional convolutional networks have. (Inductive bias refers to the inherent assumptions or biases that a machine learning algorithm is built upon. Essentially, it is the prior knowledge that the algorithm uses to make predictions based on new data.) This makes the Transformer-based network model weaker than the traditional convolutional neural network architecture in capturing local relationships, which is detrimental to the detection of edge locations in the water region in this paper. Network models built entirely from Transformers tend to perform well in large-scale datasets, which means that a large amount of data needs to be collected and labeled upfront, while the training of the model is equally costly in terms of computational resources. This is clearly not the best option for this paper, which requires application in a specific production environment. In order to achieve good results on small datasets, the original SegFormer structure is improved by considering a fusion of traditional convolution and self-attentiveness to make full use of the properties of both. Figure 3: SegFormer_UNet module. Based on the above considerations, the encoder designed in this paper adopts a hierarchical pyramid structure, with the encoder body consisting of two types of modules: the Multiscale Convolutional Attention Block and the Transformer Block. A series of SegFormer-UNet encoders are designed with reference to the original SegFormer network structure, which are named CTA-B0, CTA-B1, CTA-B2, and CTA-B3 in this paper, respectively. The dimensions of the encoder are designed by referring to the parameters of the encoder as part of the original SegFormer network structure, and customizing and optimizing them for the actual task of this paper. Table 1 shows the composition and specific parameters of each encoder building block. In the table, \(K_{i}\) denotes the patch size of the overlapping patch embedding in Stage \(i\), \(S_{i}\) denotes the stride of the overlapping patch embedding in Stage \(i\), \(P_{i}\) denotes the padding size of the overlapping patch embedding in Stage \(i\), \(C_{i}\) denotes the channel number of the output of Stage \(i\), \(L_{i}\) denotes the number of encoder layers in Stage \(i\), \(R_{i}\) denotes the reduction ratio of the Efficient Self-Attention in Stage \(i\), \(N_{i}\) denotes the head number of the Efficient Self-Attention in Stage \(i\), and \(E_{i}\) denotes the expansion ratio of the feed-forward layer in Stage \(i\). In order to make full use of the detailed features provided by the high-resolution feature map and the advanced semantic features provided by the low-resolution feature map, the encoder in this paper is divided into four stages. At the beginning of each stage, it is necessary to first obtain the features \(F_{i}\) of the previous stage using Overlap Patch Embeddings layers. Specifically, suppose that the number of image features is \(H\times W\times 3\), then the number of output features of each stage is \(\frac{H}{2^{i+1}}\times\frac{W}{2^{i+1}}\times C_{i}\), where \(i\in\{1,2,3,4\}\). Usually, \(C_{i+1}>C_{i}\). The patch embedding module in this paper is different from the one commonly used in Transformer, but it is similar to the downsampling process in convolutional neural networks. The specific implementation is to use a stepwise convolution with overlapping regions to perform convolutional operations on the feature map.