# Identifying Malignant Breast Ultrasound Images Using ViT-Patch
## 3 Experiments
### Learned Features for ViT vs. ViT-Patch  
We compare the ViT and ViT-Patch in terms of model training and feature learning. For both architectures, the patch size is set to \(S=32\), the size that performs best on the primary task (see Table 1).  
Figure 6 compares the loss-epoch learning curves and ACC (on the test set)-epoch curves of the standard ViT and ViT-Patch for the primary task. Although the Vit-Patch is trained with the joint loss in (4), for comparison, the learning curve is calculated only for \(l_{class}\), the primary-task loss component on the class token. As shown in Figure 6a, the  
Figure 5: Lesion localization using ViT-Patch models. Each row corresponds to one sample, and the columns from left to right are: original images, labeled mask maps, and mosaic maps predicted by ViT-Patch with patch size \(S=16\), 32, 64, and 128, respectively.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Model** & **DICE (\%)** \\ \hline ViT-Patch (16) & 62.0 \\ ViT-Patch (32) & 61.5 \\ ViT-Patch (64) & 63.1 \\ ViT-Patch (128) & 68.7 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Quantitative evaluation for the secondary task of ViT-Patch models.  
loss curve of the standard ViT decreases faster than that of the ViT-Patch as the number of epochs increases, and its converged loss on the training set is lower than that of the ViT-Patch. However, the ViT-Patch outperforms the standard ViT on the test set at almost all training epochs (see Figure 6b). The contrast on the training and test sets suggests that the standard ViT is prone to overfitting on small training sets under a single class-token supervision, and the proposed ViT-Patch can improve on the generalization of the model by introducing additional patch-token supervision.  
To further show the improvement of the ViT-Patch on feature learning, we visualized the token vectors learned by the standard ViT and ViT-Patch. The patch size is still set to be 32, which means that each level of transformer encoder has 145 token vectors (1 class token and 144 patch tokens) that can be visualized as a \(145\times 768\) token map. Column 2 of Figure 7 compares the token maps of the penultimate transformer block of the ViT and ViT-Patch, where the penultimate block is chosen because the patch tokens of the last block of the standard ViT are not included in the backpropagation loop. As shown in the figure, the row vectors in the ViT's token map are very similar, indicating that the multi-headed attention mechanism in the single-task mode based on the class token leads to homogenization across patch tokens and significant redundancy in the learned deeper features. In contrast, by introducing a patch-based second task, the ViT-Patch can avoid the homogenization of patch tokens.  
Figure 6: Learning curves and ACC curves for ViT vs. ViT-Patch (32). (**a**) Learning curves. (**b**) ACC (on test set) curves.  
Figure 7: Comparison of deep token features learned by ViT and ViT-Patch. Each row corresponds to one sample. The columns from left to right are: original image, token maps, variance–channel curves, and eigenvalue curves.  
To quantitatively evaluate the similarity between tokens, the variances of 145 tokens on each feature channel is calculated (see the variances-channel curves with the average variances over 768 channels in the third column of Figure 7). Consistent with the visual evaluation of the token maps, the variances obtained by the ViT-Patch are significantly larger than those obtained by the ViT. Further, we computed the eigenvalues of the token maps (matrices), which are arranged from smallest to largest in the last column of Figure 7, and it can be seen that the eigenvalue distribution of the ViT-Patch is better than that of the ViT, which is very concentrated, indicating a clear linear correlation between its learned feature vectors.