# Swin Transformer Assisted Prior Attention Network for Medical Image Segmentation
## 3 Modeling, Methods, and Design
### Swin Transformer and Attention Guiding Decoder  
Different from the typical multi-head self-attention (MSA) mechanism in vision transformer, Swin Transformer [6] is based on shifted windows to implement self-attention mechanism. The shifted windows are equipped between consecutive attention layers, which can enhance modeling power while performing computation-efficient strategy. At the same time, the hierarchical feature maps of the image are constructed on the basis of linear computational complexity. These hierarchical feature maps can make Swin Transformer more suitable as a general backbone for kinds of computer vision tasks. Since the non-local attention mechanism in traditional prior attention network has poor capability for aggregating multi-scale features from different modules and extracting boundary information, we insert Swin Transformer block into the prior attention network for enhancing the modeling power of network. As shown in Figure 2, there are two cascading Swin Transformer modules which construct one complete block. It can be seen that each Swin Transformer module consists of LayerNorm layer (LN), multi-head self-attention module (MSA), multilayer perceptron (MLP) with non-linearity activation function GELU, and twice residual connection between LayerNorm layers (LN). The important difference [6] is the window-based multi-head self-attention (W-MSA) module applied in the first Swin Transformer module, and the shifted window-based multi-head self-attention (SW-MSA) module applied in the next Swin Transformer module. Based on two successive transformer blocks with conventional window and shifted window partitioning mechanism, the attention learning process of the Swin transformer block can be formulated as:  
\[\hat{z}^{l}=\text{W-MSA}(\text{LN}(z^{l-1}))+z^{l-1} \tag{1}\]  
\[z^{l}=\text{MLP}(\text{LN}(\hat{z}^{l}))+\hat{z}^{l} \tag{2}\]  
\[\hat{z}^{l+1}=\text{SW-MSA}(\text{LN}(z^{l}))+z^{l} \tag{3}\]  
\[z^{l+1}=\text{MLP}(\text{LN}(\hat{z}^{l+1}))+\hat{z}^{l+1} \tag{4}\]  
where \(\hat{z}^{l}\) and \(z^{l}\) represent the outputs of the W-MSA of the \(l_{th}\) layer or the SW-MSA of the \((l-1)_{th}\) layer, and the MLP of the \(l_{th}\) layer. Window-based self-attention is calculated in this formula:  
\[Attention=\text{softmax}(\frac{QK^{T}}{\sqrt{d}}+B)V \tag{5}\]  
In the traditional cascaded networks, the first step is to perform a coarse segmentation and find the ROIs in medical images. In the typical prior attention network, the above process of finding coarse ROIs is performed by attention guiding decoder. In the proposed Swin-PANet, the attention guiding decoder is utilized to generate ROI-related attention prediction from the outputs of Swin Transformer. The process of attention guiding decoder plays a role of refining the feature representations and improving the quantity of segmentation. Then the refined features will be sent to the multi-level enhanced attention blocks for performing finer segmentation. The coarse process of Swin Transformer and attention guiding decoder, and the finer process of the enhanced attention blocks are integrated into the proposed coarse-to-fine strategy. As shown in Figure 3, the feature maps E1 to E4 extracted from the one-to-one Swin Transformer block are fed into the attention guiding decoder for feature fusion. As the encoded feature maps have different spatial size, the interpolation with the mode of bilinear will firstly be utilized to perform up-sampling to ensure the same spatial size. Then the feature map after up-sampling is compressed to control the irrelevant information and reduce computational cost. The next step is to concatenate encoded feature in previous level and compressed feature, respectively, in the channel dimension for performing feature fusion. By that analogy, finally, these four encoded feature maps are fused together to generate attention prediction after convolution layer with sigmoid.