# Logitwise Distillation Network: Improving Knowledge Distillation via Introducing Sample Confidence
## 3 Method
### The Proposed FD Loss  
As shown in Figure 2 and Algorithm 1, based on the teacher features \(f_{t}\) and teacher logits \(z^{t}\) of samples from that class, we obtain the weighted class mean \(c_{k}^{t^{\prime}}\). Through the FS module, the decision to use \(c_{k}^{t^{\prime}}\) instead of \(f_{t}\) for subsequent calculations is determined based on the ranking of the ground truth in logits.  
If the ground truth has a higher ranking in the logits, we calculate the cosine similarity between the feature direction \(f_{l}^{s}\) and the corresponding weighted class mean \(c_{k}^{t^{\prime}}\) as Equation (10):  
\[L_{d}=\frac{1}{C}\sum_{k=1}^{C}\frac{1}{|I_{k}|}\sum_{i\in I_{k}}\Bigl{(}1- \cos\left(f_{l}^{s},c_{k}^{t^{\prime}}\right)\Bigr{)} \tag{10}\]  
Figure 2: The calculation process of \(L_{fd}\). Teacher logits are ranked to compute sample-wise confidence weights. These weights dynamically adjust contributions to class means. Feature selection toggles between class means and raw features based on confidence. PNSA inverts supervision for mispredicted samples.  
If the ground truth has a lower ranking in the logits, we calculate the cosine similarity between the student feature \(f_{i}^{s}\) and the corresponding teacher feature \(f_{i}^{t}\) as Equation (11):  
\[{L_{d}}^{\prime}=\frac{1}{C}\sum_{k=1}^{C}\frac{1}{|I_{k}|}\sum_{i\in I_{k}} \bigl{(}1-\cos\left(f_{i}^{s},f_{i}^{t}\right)\bigr{)} \tag{11}\]  
Despite \(L_{d}\) extracting the supervisory signals present in the teacher's feature map, pre-training the teacher model does not ensure that the supervision is always accurate. When the teacher model makes incorrect predictions, the student model is prone to making the same errors. In order to tackle this problem, we introduce the PNSA module and obtain \(L_{fd}\) based on different rankings in the logits, as shown in Equations (12) and (13):  
\[L_{fd}=\begin{cases}\frac{1}{C}\sum_{k=1}^{C}\frac{1}{|I_{k}|}\sum_{i\in I_{k} }1-\cos\left(f_{i}^{s},{c_{k}^{t}}^{\prime}\right)&q^{t}=y\\ \frac{1}{C}\sum_{k=1}^{C}\frac{1}{|I_{k}|}\sum_{i\in I_{k}}-\eta(1-\cos\left( f_{i}^{s},{c_{k}^{t}}^{\prime}\right))&q^{t}\
eq y\end{cases} \tag{12}\]  
\[{L_{fd}}^{\prime}=\frac{1}{C}\sum_{k=1}^{C}\frac{1}{|I_{k}|}\sum_{i\in I_{k}} -\eta(1-\cos\left(f_{i}^{s},f_{i}^{t}\right)) \tag{13}\]  
where \(q^{t}\) represents the results predicted by the teacher model, \(y\) represents the true labels, and \(\eta\) is the weights.  
\(L_{fd}\) is compatible with existing KD methods and can be combined with \(L_{ce}\) and \(L_{kd}\) for student training. The total loss L can be expressed as Equation (14):  
\[L=L_{ce}+\beta L_{kd}+\gamma L_{fd} \tag{14}\]  
where \(\beta\) and \(\gamma\) represent the weights of \(L_{kd}\) and \(L_{fd}\), respectively. \(L_{kd}\) depends on the distillation method.  
```
1:teacher logits \(z^{t}\); student embedding features \(f_{s}\); ground truth \(y\); teacher embedding features \(f_{t}\); weighted class means \({c_{k}^{t}}^{\prime}\).
2:FD loss \(L_{fd}\)
3:\(pred\gets z^{t}.\text{data.topk}(1,\text{dim}=1)\)
4:\(pred2\gets z^{t}.\text{data.topk}(5,\text{dim}=1)\)
5:\(L_{fd}\gets 0.0\)
6:for\(s,t,i,l,n\), center in zip (\(f_{s},f_{t},y,pred,pred2,{c_{k}^{t}}^{\prime}\)) do
7:\(n\gets n.\text{tolist}()[:5]\)
8:\(e_{c}\leftarrow\text{center}/\text{center}.\text{norm}(p=2)\)
9:\(t_{c}\gets t/t.\text{norm}(p=2)\)
10:if\(l==i\)then
11:\(L_{fd}+=1-\text{torch.dot}(s,e_{c})/s.\text{norm}(p=2)\)
12:else
13:\(L_{fd}+=2.5*(-1+\text{torch.dot}(s,e_{c})/s.\text{norm}(p=2))\)
14:else
15:\(L_{fd}+=1.5*(-1+\text{torch.dot}(s,t_{c})/s.\text{norm}(p=2))\)
16:endif
17:endfor
18:return\(L_{fd}\)
```  
**Algorithm 1** FD loss computation in a PyTorch-like style, using PyTorch version 2.0.0.