## 3 Methods
### How Inter-Sample Self-Distillation Affects Feature Learning
#### 3.2.1 Training with Cross-Entropy  
Some recent studies [38; 39; 40] have observed that hard targets resulting from artificially annotated ground truth labels turn out to be a key factor hindering further accuracy improvements in classification models. For a common network trained using hard targets, we usually minimize the expected value of the cross-entropy \(H(\mathbf{y},\mathbf{p})=\sum_{k=1}^{K}-\mathbf{y}_{k}ln(\mathbf{p}_{k})\) between \(\mathbf{y}\) and \(\mathbf{p}\), where \(y_{k}\) is 1 for the correct class (here \(\mathbf{y}_{c}=1\)) and 0 for the rest. Then, the updated direction of the feature \(\mathbf{x}\) can be obtained from its gradient. The gradient of the general cross-entropy loss acting on \(\mathbf{x}\) is \(\frac{\partial H(\mathbf{y},\mathbf{p})}{\partial\mathbf{x}}=\sum_{k=1}^{K}(\mathbf{p}_{k}- \mathbf{y}_{k})\mathbf{w}_{k}\), where \(\mathbf{w}_{k}\) indicates the \(k\)-th row vector of the weight matrix \(\mathbf{w}\), corresponding to the \(k\)-th class. For a more intuitive presentation, we define the gradient component associated with the \(k\)-th class as follows:  
\[\delta_{k}=(\mathbf{p}_{k}-\mathbf{y}_{k})\mathbf{w}_{k}.\]  
Then, the gradient can be expressed as follows:  
\[\frac{\partial H(\mathbf{y},\mathbf{p})}{\partial\mathbf{x}}=\sum_{k=1}^{K}\delta_{k} \triangleq\delta. \tag{1}\]  
Here, \(\mathbf{p}_{k}\) assumes values within the interval \((0,1)\), implying that \(\delta_{c}\) aligns in the direction opposite to \(\mathbf{w}_{c}\), while \(\delta_{k}\) aligns with \(\mathbf{w}_{k}\) for instances where \(k\
eq c\). This alignment is influenced by the optimization process, which inherently seeks to maximize the angular separation between the weight vectors of disparate classes. Consequently, the angle  
Figure 2: Visualization of features and their updated directions. The thin arrows indicate the updated directions, whose lengths are relative (do not specify the magnitude of the updates). The thick arrows point from the origin to the coordinates of each class weight. Here, three classes in different colors totaling 180 features are randomly generated and optimized using the SGD algorithm within the framework of cross-entry (**first row**), label smoothing (**second row**), and CS-KD (**third row**), respectively. The positions and update directions of the features at different training epochs are shown in different columns. between \(\mathbf{w}_{c}\) and the aggregate vector \(\sum_{k=1}^{K}\mathbf{w}_{k}\) remains consistently broad. This leads to a smaller angular divergence between \(\mathbf{\delta}\) and \(-\mathbf{w}_{c}\), indicating that the update trajectory of \(\mathbf{x}\) closely parallels \(\mathbf{w}_{c}\). This dynamic introduces two primary challenges: (1) the propensity for features to be over-learned, which can compromise the model's generalizability, and (2) updates to features within the same class tend to occur in approximately parallel directions, rendering intra-class variance heavily dependent on the initial positioning of the features. These phenomena are visually represented in the first row of Figure 2, where certain features, initially poorly positioned, remain distant from their class counterparts throughout the training duration. Such occurrences are not uncommon, particularly in tasks characterized by significant initial intra-class variation such as fine-grained classification tasks [41; 42; 43], where the challenge of minimizing intra-class variance can notably impair the network's ability to generalize. Additionally, pronounced intra-class variation results in the dispersion of similarly labeled features, adversely affecting the learning of class-specific weight vectors \(\mathbf{w}\). Furthermore, this examination of the impact exerted by cross-entropy loss on the spatial distribution of features also furnishes an insight into how pre-training on datasets like ImageNet [44] can enhance model generalization.