## 4 Experiment
### Experimental Results
As shown in Figure 3, the proposed method effectively reduces the number of misclassified samples and improves the correct classification compared with \"LayerNorm FT\". Specifically, the accuracy of the proposed method is increased in 63 out of 100 classes in the Aircraft dataset. Notably, for classes such as 707-320 and 767-300, our method achieves 15% higher accuracy compared with \"LayerNorm FT\". We visualized the attention map from fine-tuned models with and without consistency loss to qualitatively demonstrate the effects of consistency loss. The model trained solely with DINO loss assigns high attention scores to both the object and background patches, as shown in Figure 4. On the other hand, objects could be more precisely segmented from the background by applying the consistency loss function. Figure 3: Difference in confusion matrices obtained from linear probed models of “LayerNorm FT” and “Lastblock FT + \(L_{con}\)” in Table 1 on Aircraft. Each element is computed by subtracting the corresponding element of the “Lastblock FT + \(L_{con}\)” confusion matrix from the “LayerNorm FT” confusion matrix. Table 2 contains the fine-grained classification performance of each pretrained model after end-to-end supervised fine-tuning instead of a linear evaluation. At this time, we also consider the model pretrained on ImageNet in a supervised manner, denoted as \"Supervised\". The SSL pretrained models performed exceptionally better than \"Supervised\" in the Aircraft, Car, and Flower datasets. In addition, the performance increases when the last block is pretrained during transfer learning to the target dataset (see the third row of Table 2). Specifically, the proposed method shows the best performance among the SSL pretrained models on every FGVC dataset (fourth row of Table 2). On Aircraft, the proposed method improves the performance by 3.96% over \"Supervised\". In the cases of the Food, CUB, and Dog datasets, the supervised pretrained model performed the best. In summary, the proposed method does not surpass \"Supervised\" on the Food, CUB, and Dog datasets, but it performs best on all datasets compared with SSL pretrained models. As a result, we confirmed that SSL with the proposed consistency loss helps learn fine-grained representations. **ImageNet experiments. ** The previous experimental results demonstrated the effectiveness of the proposed consistency loss in the transfer learning framework. In this section, we further investigate the effect of consistency loss when applied to a coarse-grained dataset. To this end, we trained ViT-S/16 on the ImageNet dataset from scratch for 300 epochs. Similar to the previous experiments, we set \(\lambda\) and the output size of the RoI align layer to 0.5 and \(3\times 3\), respectively. First, we compared the test accuracy on ImageNet with that of DINO. We pretrained the model on ImageNet with consistency loss and then examined the top 1 accuracy of \(k\)-NN classification and linear probing on the validation set of ImageNet. We observed that the proposed consistency loss decreased performance when trained on a large-scale coarse-grained dataset, such as ImageNet. DINO shows 72.8 and 76.1 of test accuracy for \(k\)-NN and linear probing, respectively. However, when consistency loss is applied, the accuracy slightly decreases to 71.4 and 74.2 for \(k\)-NN and linear probing, respectively. This negative effect is consistent with observations from previous studies [43; 44]. Similar to these studies, \begin{table}
\begin{tabular}{l c c c c c c} \hline
**Pretrained Model** & **Aircraft [16]** & **Car [15]** & **Flower [39]** & **Food [40]** & **CUB [14]** & **Dog [41]** \\ \hline Supervised & 81.34 & 90.15 & 97.53 & **90.97** & **85.36** & **87.74** \\ Full FT & 84.70 & 91.94 & 98.10 & 90.67 & 82.07 & 82.33 \\ Lastblock FT & 85.05 & 92.14 & 98.21 & 90.78 & 81.45 & 82.34 \\ Lastblock FT + \(L_{\text{con}}\) & **85.30** & **93.10** & **98.54** & 90.86 & 82.29 & 83.11 \\ \hline \end{tabular}
\end{table} 
Table 2: Test accuracy on each dataset under fine-tuning evaluation.