## 3 Methods
### How Inter-Sample Self-Distillation Affects Feature Learning
#### 3.2.2 Training with Inter-Sample Self-Distillation  
Knowledge distillation introduces the concept of temperature to soften the predicted output \(\hat{\mathbf{p}}\). Its \(k\)-th element is \(\hat{\mathbf{p}}_{k}=\frac{e^{\xi_{k}/\tau}}{\sum_{k=1}^{K}e^{\xi_{k}/\tau}}\), where \(\tau\) denotes the hyperparameter of temperature. Let \(\mathbf{q}\) represent the soft target generated by the teacher network for \(\mathbf{x}\). The joint minimization of the cross entropy and Kullback-Leibler divergence (KL divergence) between the predictions and the soft targets can be given as follows: \(\mathcal{L}_{kd}=(1-\alpha)H(\mathbf{y},\mathbf{p})+\alpha\tau^{2}D_{KL}(\mathbf{q}\|\hat{ \mathbf{p}})\), where \(\alpha\) is the weight of the two terms. Then, according to Equation (1), the gradient of feature \(\mathbf{x}\) in knowledge distillation can be computed as follows: \(\frac{\partial\mathcal{L}_{kd}}{\partial\mathbf{x}}=(1-\alpha)\sum_{k=1}^{K} \delta_{k}+\alpha\tau\sum_{k=1}^{K}(\sum_{i=1}^{K}(q_{i})\hat{\mathbf{p}}_{k}-q_{k} )\mathbf{w}_{k}\), where \(\mathbf{q}_{k}\) and \(\mathbf{q}_{i}\) denote the \(k\)-th and \(i\)-th elements of \(\mathbf{q}\), respectively. Generally, \(\mathbf{q}\) is a softmax output, so \(\sum_{i=1}^{K}(\mathbf{q}_{i})=1\). Here, when we define \(\hat{\mathbf{\delta}}_{k}=(\hat{\mathbf{p}}_{k}-\mathbf{q}_{k})\mathbf{w}_{k}\) and \(\hat{\mathbf{\delta}}=\sum_{k=1}^{K}\hat{\mathbf{\delta}}_{k}\), we have the following:  
\[ \begin{split}\frac{\partial\mathcal{L}_{kd}}{\partial\mathbf{x}}& =(1-\alpha)\sum_{k=1}^{K}\delta_{k}+\alpha\tau\sum_{k=1}^{K}\hat{ \mathbf{\delta}}_{k}\\ &=(1-\alpha)\delta+\alpha\tau\hat{\mathbf{\delta}}\end{split}  \tag{2}\]  
The effects of the gradient component \(\mathbf{\delta}\) on the feature distribution have been discussed above. Here, we explore the role of the gradient component \(\hat{\mathbf{\delta}}\) originating from \(\mathbf{q}\) in feature learning. For the \(k\)-th component of \(\hat{\mathbf{\delta}}\), \(\hat{\mathbf{\delta}}_{k}=0\) if and only if \(\hat{\mathbf{p}}_{k}=\mathbf{q}_{k}\). If \(\hat{\mathbf{p}}_{k}>\mathbf{q}_{k}\), the angle between the feature \(\mathbf{x}\) and the weight \(\mathbf{w}_{k}\) will be updated to be larger. If \(\hat{\mathbf{p}}_{k}<\mathbf{q}_{k}\), it will be updated to be smaller. This motivates \(\mathbf{x}\) and \(\mathbf{w}\) to maintain a certain positional relation in the feature space, which can penalize over-optimization. We define the logical vector satisfying \(\hat{\mathbf{p}}=\mathbf{q}\) as \(\hat{\mathbf{z}}\), i.e., \(\frac{e^{\xi_{k}/\tau}}{\sum_{k=1}^{K}e^{\xi_{k}/\tau}}=\mathbf{q}_{k}\). Then, the feature satisfying this condition must be a solution of the non-homogeneous linear equation \(\mathbf{w}\mathbf{x}^{T}=\hat{\mathbf{z}}\). The equation has solutions only if \(\mathbf{w}\) is full rank (it has a unique solution if \(\mathbf{w}\) is a square matrix). The exploration into the dynamics of feature learning through supervision with hard targets reveals that such an approach causes the angles between the weights of different classes to widen, symbolized by \(\mathbf{w}\), evolving towards a full rank matrix [45]. This progression enables the features to align more closely with their respective solutions within the feature space, thereby elucidating the contribution of knowledge distillation towards feature learning. A natural progression of this inquiry leads us to consider the scenario where samples within the same class serve as distillation targets for one another, a process herein referred to as inter-sample self-distillation. Building upon this premise, it becomes evident that within a given class, features act as mutual solutions, converging towards one another throughout the training process. This mechanism directly addresses the limitations identified in Section 3.2.1, where it was noted that cross-entropy alone fails to adequately constrain the intra-class variance, thereby allowing samples, particularly those with suboptimal initial positions, to stray from their class centroids.