# Advanced Brain Tumor Segmentation Using SAM2-UNet
## 4 Proposed Approach
### Integrated SAM2-UNet Model  
Even though the Segment Anything Model (SAM-2) was based on the SAM-1 model, including advanced and robust improvements, SAM2 produces class-agnostic results, i.e., identification and object separation is based on images without label requirement when no prompt is given to the model [9]. To tackle this problem, this study has proposed the integration of SAM2 and UNet model as shown in the Figure 12.  
Figure 11: UNet architecture. The image is depicting the architecture of the U-shaped network. This network is a collection of convolutional neural networks, max-pooling layers, and skip connections, along with the bottle neck as the base of the U network.  
In this proposed solution, the SAM2 model acts as an encoder, and the UNet architecture functions as the decoder. This combination leverages SAM2's advanced zero-shot segmentation capabilities, which help in handling complex structures with minimal manual training. The SAM2 encoder provides robust feature extraction, while the UNet decoder refines the features to produce precise segmentation outputs. This study also incorporates Hiera, a pre-trained model that captures multi-scaled features and is well suited for U-shaped networks. Given an input image, \(I\in\mathbb{R}^{3\times H\times W}\), where \(H\) denotes height and \(W\) denotes width, Hiera outputs four hierarchical features, \(X_{i}\in\mathbb{R}^{C_{i}\times 2^{H/(i+1)}\times 2^{W/(i+1)}}\) (\(i\in\{1,2,3,4\}\)). In order to build this hybrid architecture, we have omitted a few modules from the SAM2 model, namely memory attention, prompt encoder, memory encoder, and memory bank, as they are not necessary to build a U-Network architecture [10]. In summary, while the traditional UNet remains as a powerful tool for medical image segmentation, the hybrid SAM2-UNet model offers significant advantages in handling complex structures with minimal manual training. This combination of SAM2's robust feature extraction and UNet's refined decoding process results in precise and efficient segmentation. The proposed architecture consists of four encoders that accept an image with dimensions of 3 \(\times\) 352 \(\times\) 352 and converts them into feature embeddings of distinct channels. These extracted features will then be passed into the corresponding four receptive field blocks, which help in reducing the number of channels to 64 channels in order to enhance the lightweight features [18]. These features will then be passed into the respective decoder blocks, which are inspired by customizable U-shaped structures, which have been proven to be effective in many biomedical applications [19]. These decoders consist of two combinations of a 3 \(\times\) 3 convolutional layer, Batch normalization, and a ReLU activation function. The outputs from these decoders will be passed into a 1 \(\times\) 1 convolutional layer that produces a segmentation result, which is then upsampled and supervised by the ground-truth segmentation mask 'G'. We use the mean weighted Intersection over Union (mIoU) loss and binary cross-entropy loss as our training objectives [20]:  
\[L=L_{\text{IoU}}+L_{\text{RCE}}.\]  
Figure 12: SAM2-UNet architecture.  
Additionally, we apply deep supervision to all segmentation outputs, \(S_{i}\). The total loss for SAM2-UNet is formulated as follows:  
\[L_{\text{total}}=\sum_{i=1}^{3}L(G,S_{i}).\]  
Each adapter in the proposed framework consists of a linear layer for downsampling, a GeLU activation function, which is followed by another linear layer for upsampling, and a final GeLU activation function, similarly to in [21].  
Table 2 depicts the training hyperparameters used for training our SAM2-UNet model on the brain tumor segmentation 2020 dataset.