## 4 Experiments
### Ablation Study  
Ablation experiments are conducted for the trade-off parameter \(\alpha\) and temperature \(\tau\), respectively. All the experiments are implemented using ResNet-18 as the backbone. We try different values for the trade-off parameter \(\alpha\) including \(\{0.1,0.3,0.5,0.7,0.9\}\) and the temperature \(\tau\) including \(\{1,2,3,4\}\) on the regular dataset CIFAR100 and the fine-grained dataset CUB-200-2011, respectively. The results are summarized in Figure 4, where \(\alpha=0\) equals the baseline model using cross-entropy loss. It can be seen from Figure 3(a) that CSD achieves the best performance on CIFAR100 when distillation loss and cross-entropy loss are balanced, i.e., \(\alpha=0.5\). The performance of the model becomes better as \(\tau\) increases, but is insensitive at values greater than 2. The CSD exceeds the baseline across all settings, showing its good robustness. On the fine-grained task CUB-200-2011, as shown in Figure 3(b), the trend of the model performance is different. The performance improves as \(\alpha\) is increased at any \(\tau\). The results on both datasets show that the effects of \(\alpha\) and \(\tau\) are independent of each other. Regular tasks are suitable for larger \(\tau\) values and a balanced distillation loss, while fine-grained tasks favor smaller \(\alpha\) values and larger distillation losses. To further clarify the impact of different methods on the features, we measure the performance of the model based on three other aspects in Table 4:  
(1) _Prediction distribution._ The Top-5 error rate is used here to quantitatively evaluate the prediction distribution of our model. It can be seen from Table 4 that CSD outperforms the existing methods on four datasets, especially on the CUB-200-2011, by 4% over SOTA and 12% over Cross-entropy, showing the great potential of CSD to improve the prediction distribution of the classification models. The reason is related to the inter-class relations introduced by distillation, which allow features from similar classes to hold closer positional relations in the feature space. Therefore, it can prevent the model from becoming over-optimized, enhance the generalization, and make the predictions more reasonable. This can also explain why the results of BAKE and CS-KD, which also use distillation, are also good. (2) _Feature distribution._ Here, we provide quantitative results on the metric R@1, which reflects the tightness of feature clusters [61]. A larger value of R@1 implies less inter-class crossover on the feature embedding or more distant feature clusters of different classes. As shown in Table 4, the R@1 values can be significantly improved when training ResNet-18 with CSD. The visualization of the features is shown in Figure 5. It can be seen that our method makes the features of different classes more dispersed and the features of the same class more aggregated. An important reason for this is that CSD drives same-label features to move toward their class centers, which, while not increasing the class-to-class distance, reduces the overlap between feature clusters. Fine-grained images typically have higher inter-class overlap; thus, CSD can provide greater performance improvements across several fine-grained tasks, as the results show. The experimental  
Figure 4: Ablation analysis for the trade-off parameter \(\alpha\) and temperature \(\tau\). The Top-1 error rates of ResNet-18 on CIFAR100 and CUB-200-2011 are reported under the CSD settings, respectively. results shed light on the intricate dynamics of clustering observed within this study. It was noted that clusters encompassing a larger number of classes tended to demonstrate enhanced cohesiveness, effectively distinguishing themselves from neighboring clusters. This distinction is critical for the clarity and accuracy of the clustering process. Additionally, an observable dispersion within these clusters reveals that, despite the collective identity, each individual sample deviates uniquely from the cluster's centroid.