# Semantic Segmentation of Brain Tumors Using a Local-Global Attention Model
## 3 Methods
### Semantic-Oriented Masked Attention  
In the original Swin UNETR, both the encoder and decoder utilize local feature extraction methods, which are inefficient in capturing the global features of brain tumors. In addition, the layers close to the bottom of this U-shaped model are distant from the loss function, resulting in low precision in feature extraction within these layers. To address these issues, we propose SMA as a replacement for the residual block in the decoder. The SMA comprises two core components: a non-homogeneous embedding module and a masked attention module. Figure 4 illustrates its structure in detail.  
Most Multi-Head Self-Attenttions (MHSAs) use the same embedding method to generate the query, key, and value. Different from them, we design a non-homogeneous embedding module, which contains a residual block and a fully connected layer with a Dropout layer to produce these. The fully connected layer is responsible for spatial reduction. It is a global operation and offers great flexibility in setting the output dimension. The Dropout layer is used to alleviate the overfitting problem. We employ the residual block to extract local features. It preserves the details of brain tumors and is crucial for distinguishing different classes of edge information.  
Deep supervision has been widely used in segmentation tasks to guide the feature extraction of intermediate layers. However, it is computationally expensive, as it requires calculating pixel-wise classification loss at various resolutions [52]. We develop a masked attention module that incorporates semantic information in the calculation of attention. Specifically, we set the number of channels of \(Y_{Q}\) and \(Y_{K}\) to \(T\) using two \(1\times 1\times 1\) convolution layers, where \(T\) is equal to the number of classes. The \(1\times 1\times 1\) convolutions serve as a shared classifier across all positions in the embedded features. Another \(1\times 1\times 1\) convolution is applied to \(Y_{V}\), with the output channel dimension equal to the input channel dimension. This convolution is used to recover the channel dimension of the final output. For more details about the masked attention module, please refer to Figure 5.  
Figure 3: The structure of original residual blocks.  
All of the SMAs share the same structure, except for the bottom one, which only receives features from the last encoder block and the features have a relatively small spatial dimension. Therefore, we only retain the mask attention module in this bottom SMA, as shown in Figure 6. The process of SMA can be summarized as follows:  
\[Y_{I}=F_{1}(Concat(Y_{E},Y_{D})), \tag{1}\]  
\[Y_{Q}=F_{2}(Res(Y_{I})), \tag{2}\]  
\[Y_{K}=F_{3}(FC(Dropout(Y_{I}))), \tag{3}\]  
\[Y_{V}=F_{4}(FC(Dropout(Y_{I}))), \tag{4}\]  
\[Z=Res(Y_{I})+Softmax(Y_{Q}Y_{K}^{\top})Y_{V}, \tag{5}\]  
where \(Y_{E}\in\mathbb{R}^{N\times C}\) and \(Y_{D}\in\mathbb{R}^{N\times C}\) represent the input features from the encoder and decoder, respectively. \(N\) is the length of the image token sequence and \(C\) is the channel dimension. \(F_{1}\), \(F_{2}\), \(F_{3}\), and \(F_{4}\) denote four different \(1\times 1\times 1\) convolution layers. _Concat_ refers to the concatenate operation. _Dropout_ represents the Dropout layer. \(FC\) denotes the fully connected layer, and \(Res\) represents the residual block. \(Z\) is the final output. The dimensions of \(Y_{Q}\), \(Y_{K}\), and \(Y_{V}\) are \(\mathbb{R}^{N\times T}\), \(\mathbb{R}^{\top\times P}\), and \(\mathbb{R}^{P\times C}\), respectively, where \(P\) is the output dimension of the fully connected layer.  
Figure 4: The structure of SMA blocks in the middle layers.  
Figure 5: The structure of Mask Attention.