## 4 Experiments
### Quantitative Results
Given the variability in data distributions, label smoothing yields notable benefits for only a subset of datasets. In contrast, our contrastive self-distillation (CSD) method dynamically updates class predictions, ensuring its applicability and effectiveness across all datasets. This adaptability allows CSD to maintain an optimal performance, irrespective of the underlying data distribution, thereby providing a more universally applicable solution. (5) _Center loss_[19]. Our perspective posits that inter-sample self-distillation diminishes intra-class variability, prompting a comparison between our contrastive self-distillation (CSD) approach and center loss. The superiority of our method primarily stems from the incorporation of constraints on inter-class dynamics alongside a targeted intra-class sample bias. This approach mitigates the risk of overfitting by preventing excessive clustering of intra-class features. Additionally, the emphasis on inter-class relationships fosters a more robust learning of class distributions, further enhancing model performance. In addition, our CSD has particularly significant performance improvements on three fine-grained datasets: about 5-6% relative to SOTA with the Resnet-18 as the backbone, and about 4-8% relative to SOTA with the DenseNet-121 as the backbone. The reason for this is that our method emphasizes the control of intra-class distance and inter-class distance. Since fine-grained datasets have a higher inter-class similarity and are more sensitive to the change in intra-class distances than common datasets, they are more likely to benefit from it. (6) _Training costs._ The detailed analysis and results presented in Table 3 demonstrate how the image size significantly affects the computational requirements, specifically the training time cost and memory consumption, for training a ResNet-18 model. Training the model with larger images (224 \(\times\) 224 pixels) requires substantially more computational resources--132.53 s per epoch for training time and 21.84 GB of memory--compared to smaller images (32 \(\times\) 32 pixels), which only need 72.87 s per epoch and 12.01 GB of memory. Furthermore, our method utilizes a single backbone for computational processes, unlike previous methods that typically rely on two independent backbones. This approach presents a significant advancement in computational efficiency, ensuring that the resource consumption for training and running the model does not escalate beyond the comparative baseline of direct training.This can be corroborated by comparing the CSB data with the backbone network data in Table 3. In essence, the method achieves a balance between maintaining a high performance in terms of accuracy and efficiency while minimizing the increase in computational demand. This balance is crucial for deep learning applications, particularly in scenarios where computational resources are limited or when optimizing for faster, more efficient model training and inference processes. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Image Size** & **Method** & \
\begin{tabular}{c} **Training Time Cost** \\ **(s/epoch)** \\ \end{tabular} &
\begin{tabular}{c} **Memory** \\ **Consumption (GB)** \\ \end{tabular} \\ \hline \multirow{2}{*}{\(224\times 224\)} & backbone & 128.74 & 20.95 \\  & CSD (ours) & 132.53 & 21.84 \\ \hline \multirow{2}{*}{\(32\times 32\)} & backbone & 71.53 & 11.56 \\  & CSD (ours) & 72.87 & 12.01 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Training time cost and memory consumption of ResNet-18 on various image sizes.