# Identifying Malignant Breast Ultrasound Images Using ViT-Patch
## 2 Methods  
### Standard ViT for Malignant Identification  
The ViT replaces the traditional convolutional computation with a self-attentive mechanism in vision tasks. Figure 1 shows the standard ViT, which is composed of three modules: patch and position embedding, transformer encoder, and MLP head. _Patch and position embedding:_ Since the input to transformer is a sequence of vectors, the ViT first needs to divide the input image into different patches and encode them to form a sequence of embedded vectors, i.e., the initial patch tokens. Suppose that the input image is \(\mathbf{I}\in\mathcal{R}^{H\times W\times C}\), where \(C\) is the number of channels, and \(H\) and \(W\) represent the height and weight of the image, respectively. The size of each patch is \(S\times S\). There are \(N=HW/S^{2}\) patches in total, and by patch and position embedding, \(N\) embedded vectors (patch tokens) are obtained. Similar to BERT [24], ViT adds a learnable token, i.e., the class token. All \(N+1\) tokens are fed to the transformer encoder for multi-headed self-attention computation. The patch embedding can be implemented with a \(S\times S\times C\times D\) convolutional layer of padding = _\"valid\"_, channel number = \(C\) and stride = \(S\),  
\[\{\mathbf{t}_{1},\mathbf{t}_{2}\dots\mathbf{t}_{n}\}_{n=1}^{N}=Conv(\mathbf{ I}) \tag{1}\]  
where \(\mathbf{t}_{n}\in\mathcal{R}^{D}\) denotes the \(n\)-th patch token, and the dimension \(D\) of each token vector is determined by the number of output channels of the convolutional layer. To embed the position information, a learnable \(D\)-dimensional position encoding vector \(\mathbf{s}_{n}\) is added to each patch token, i.e., \(\mathbf{t}_{n}\leftarrow\mathbf{t}_{n}+\mathbf{s}_{n}\). An initial token matrix \(\mathbf{T}\in\mathcal{R}^{(N+1)\times D}\) is finally obtained by concatenating the class token and the \(N\) patch tokens. \[\mathbf{t}_{n}=\mathbf{t}_{n}+\mathbf{s}_{n}\in\mathcal{R}^{D} \tag{2}\] \[\mathbf{T}=Concat(\mathbf{t}_{class};\mathbf{t}_{1},\mathbf{t}_{2} \dots\mathbf{t}_{n})\in\mathcal{R}^{(N+1)\times D} \tag{3}\]  
_Transformer encoder_: The transformer encoder is composed of multiple encoder blocks based on multi-headed attention. In each encoder block, there are two basic computational units, Multi-Headed Self-Attention (MHSA) and MLP, each preceded by a LayerNorm  
Figure 1: Standard ViT architecture (with \(N=9\) patches as an example). (**a**) Data Flow of ViT. (**b**) Transformer Encoder Block. (LN) [25; 26] and followed by a residual connection. Based on single attention function, MHSA use multi-head for multiple parallel attention calculations. \[[QKV_{1},QKV_{2}...QKV_{h}]=FC(\textbf{T}) \tag{4}\]  
\(QKV\) represents the query \(Q\in\mathcal{R}^{(N+1)\times d}\), keyword \(K\in\mathcal{R}^{(N+1)\times d}\), and value \(V\in\mathcal{R}^{(N+1)\times d}\). Finally, MHSA consider \(h\) attention \"heads\", where \(h\) represents the number of heads, and \(D=h\times d\). \[MHSA(T)=Concat(head_{1},head_{2}...head_{h}) \tag{5}\] \[head_{i}=Attention(Q_{i},K_{i},V_{i})=softmax(\frac{Q_{i}K_{i}^{T }}{\sqrt{d}}V_{i}) \tag{6}\]  
Let **T** be the input token matrix of an encoder block, i.e., the output of the previous block. The data flow is shown in Figure 1b.