# Automatic Medical Image Segmentation with Vision Transformer
## 3 Method
### Transformers Encoder  
Following the vision transformer (ViT) [45], we first tokenized the input image \(x\) by reshaping it into a sequence of flattened 2D patches, \(x_{p}^{i}\in\mathbb{R}^{p^{2}\cdot C}\big{|}i=1,\cdots,N\), and every image patch was \(P\times P\) size. The patches number \(N=\frac{H\times W}{P^{2}}\) was as the input sequence length.  
We employed a trainable linear projection layer to map these image patches \(x_{p}\) into a \(D\)-dimensional latent space. To incorporate spatial information, we added positional embeddings to the patch embeddings before passing them through the projection layer:  
\[z_{0}=\left[x_{p}^{1}E;x_{p}^{2}E;\ldots;x_{p}^{N}E\right]+E_{pos} \tag{1}\]  
Here, \(E\in\mathbb{R}^{(p^{2}\cdot C)\times D}\) is the linear projection of patch embedding, and \(E\in\mathbb{R}^{(p^{2}\cdot C)\times D}\) is the position embedding. The input image is divided into patches, which are then projected into an embedding space. Learned positional encodings are provided through learned embedding vectors. This tokenization and embedding process allows the transformer to model interactions between image patches for global context modeling. In other words, the image is divided into local patches and each patch is mapped to a vector space that retains spatial relationships. This tokenization presents the transformer with visual concepts and their arrangements, priming it to model semantic and positional interactions. The output patch embeddings summarized identity and layout to inform the downstream decoding.  
The transformer encoder contains \(L\) identical layers, with each layer containing a multi-layer perceptron (MLP) block and a multi-head self-attention (MSA) block (Equations (2) and (3)). The MSA block first projects the embeddings into query, key, and value vectors. It then calculates attention based on the dot-product over all tokens using these projections. This models the global relationships in the image. The MLP then processes each token individually with non-linear transformations to encode semantic concepts. Layer normalization and residual connections are employed around each block. The \(\mathcal{L}-layer\) output can be obtained by:  
\[z^{\prime}{}_{\mathcal{L}}=MSA(LN(z_{\mathcal{L}-1}))+z_{\mathcal{L}-1^{\prime}} \tag{2}\]  
\[z_{\mathcal{L}}=MLP(LN(z^{\prime}{}_{\mathcal{L}}))+z^{\prime}{}_{\mathcal{L}} \tag{3}\]  
Here, \(LN(\cdot)\) represents applying layer normalization, and \(z_{L}\) is the transformer output encoding the image features after \(L\) layers. The stacked MSA and MLP blocks enable complex reasoning through multiple rounds of global self-attentive processing and local non-linear transformations.  
Figure 1a illustrates the structure of the transformer layers. There are 12 transformer layers in the second convolutional block of the CNN encoder, and each transformer encoder contains multiple identical layers stacked on top of each other, with each layer containing two sublayers, a multi-head self-attention (MSA) layer and a multi-layer perceptron (MLP) layer. Since we mapped the vectorized patches into a latent \(D\)-dimensional embedding space using a trainable linear projection, the number of hidden dimensions and attention  
heads can adjust based on the input image dimensions, reflecting the transformer's capability to accommodate images of varying sizes--a crucial and practical feature in clinical settings. The self-attention mechanism models the global interactions between image patches, while the MLP provides non-linear transformations to encode semantic concepts. Stacking these layers allows modeling of the global context and long-range dependencies present in the input image.