# STHarDNet: Swin Transformer with HarDNet for MRI Segmentation
## 2 Related Work
### Swin Transformer Block  
The Swin Transformer consists of Swin transformer blocks, which are created to be suitable for detection and segmentation by introducing the concept of a hierarchical feature map and shifted windows to ViT. In conventional transformers, self-attention is performed by creating tokens with the same patch size, but the Swin Transformer uses a method of merging adjacent patches gradually, starting from a patch size of 4 \(\times\) 4, like the hierarchical structure of the feature pyramid network. This allows using each hierarchical feature map's information, like U-Net. Figure 1 shows the patch merging process, where, in the figure, a red box refers to a window, the small box (1-16) refers to a patch (token) with a size of 4 \(\times\) 4, and patch merging merges 2 \(\times\) 2 patches into one. In the process of patch merging, the feature map's size is down-sampled to (W/2, H/2).  
For Swin transformer blocks, self-attention is performed in respective windows only, and merging is performed in the last feature map. This solves the problem of increasing computations when the image size increases in conventional ViT. However, the positions of the windows are fixed, and the relationship between the windows is not represented because self-attention is performed in the fixed windows only. Therefore, to calculate the relationship between two windows, the window is shifted to the right (\(\rightarrow\)) and down (\(\downarrow\)) directions by window size/2, and the self-attention is performed once more [15]. As a result, the Swin Transformer facilitates the analysis of the entire input image with self-attention alone in respective windows.  
Figure 2 shows an example of the shifted windows approach. In Figure 2, a red box means a window, and a small box (1-16) means a patch (token) with a size of 4 \(\times\) 4. In the figure, there are four windows with a window size of 4. In Swin transformer block 2, the windows are shifted based on Swin transformer block 1. In Swin transformer block 2, the windows are shifted in the right (\(\rightarrow\)) and down (\(\downarrow\)) directions by the window size/2, and the feature parts that the windows lack are supplemented, as shown in colors in the figure.  
Figure 3 below shows a schematic diagram for connecting two Swin transformer blocks. As shown in the figure, the standard windows-based, multi-head, self-attention (W-MSA) module and shifted window-based, multi-head, self-attention (SW-MSA) module are used sequentially in the Swin transformer block. There is LayerNorm (LN) in front and back of S(W)-MSA, and the last MPL consists of two GELU non-linearities. Therefore, the Swin transformer block is used in multiples of two [16].  
Figure 1: Example of patch merging in windows of the Swin Transformer.  
Connections of Swin transformer blocks can be expressed in Equations (2)-(5) below, where \({}^{z}\)\({}^{l}\)\({}^{l}\) is the output of (S)W-MSA, \({}^{z}\)\({}^{l}\)\({}^{l}\) is the output of MLP, and \({}^{l}\)\({}^{l}\) denotes the position of the Swin transformer block.  
\[\hat{z}^{l}=W-MSA\left(LN\left(z^{l-1}\right)\right)+z^{l-1} \tag{2}\]  
\[z^{l}=MLP\left(LN\left(\hat{z}\right)\right)+\hat{z}^{l} \tag{3}\]  
\[\hat{z}^{l+1}=SW-MSA\left(LN\left(z^{l}\right)\right)+z^{l} \tag{4}\]  
\[z^{l+1}=MLP\left(LN\left(z^{l+1}\right)\right)+\hat{z}^{l+1} \tag{5}\]