# Revisiting Label Smoothing Regularization with Knowledge Distillation
## 5 Experiments
### Cifar-100
#### 5.1.2 Training Detail  
This subsection demonstrates the training curves of the proposed methods on the CIFAR-100 dataset with the WRN-40-4 network [33]. The implementation details are introduced at the beginning of Section 4.1. The accuracy curves on the training and test dataset are visualized in Figure 4. As Figure 4b shows, the LSR failed to improve the baseline test set accuracy on the WRN-40-4 network, whereas LSR-OS succeeded. With the modified teacher, LSR-OS-TC can improve the performance further.  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Model** & **Size** & **BL** & **LSR**[19] & **Tf-KD**[27] & **LSR-OS** & **LSR-OS-TC** \\ \hline ResNet56 [1] & 0.9M & 27.56 & **27.12** & 27.25 & 27.34 & 27.33 (0.23) \\ ResNet110 & 1.7M & 25.79 & 25.48 & 25.45 & 25.60 & **25.43** (0.36) \\ Pre110 [32] & 1.7M & 25.63 & 25.49 & **25.46** & 25.42 & 25.52 (0.11) \\ ResNet164 & 1.7M & 23.76 & 23.44 & 22.76 & 22.89 & **22.51** (1.25) \\ Pre164 & 1.7M & 22.37 & 22.33 & **22.01** & 22.08 & 22.24 (0.13) \\ WRN-40-4 [33] & 9.0M & 20.73 & 20.96 & 20.56 & 20.71 & **20.52** (0.21) \\ WRN-28-10 & 36.5M & 19.25 & 20.08 & 19.23 & 19.32 & **18.96** (0.29) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Test set error rate comparison on the CIFAR-100 dataset. The “BL” in the table’s column head is short for “baseline”. The results with no improvement compared to the baselines are underlined, and the bold results are the best ones for every network. The relative error rate drop of LSR-OS-TC compared to the baseline is shown in the brackets.  
Figure 3: The typical images from the CIRFAR-100 dataset. (**Top**): original image. (**Bottom**): augmented image. The labels from left to right: spider, possum, motorcycle, lamp, skyscraper, pickup truck, bear, and bottle.  
Figure 4(a),b illustrates the cross-entropy and KL-divergence loss of the training set, and Figure 4(c) shows the cross-entropy loss on the test set. Note that for LSR, we utilized the reformulated LSR-OS version with \(\tau\) equal to one. The curves in Figures 4 and 5 reveal a wealth of information.  
With a stronger regularization effect, KD methods have a higher cross-entropy loss on both the training and test datasets. Figures 5(a) and 4(a) show that, although all the regularization methods make the classification cross-entropy loss higher, their training accuracy is increased. This observation demonstrates that the regularization methods even generalize better on the training dataset.  
Figure 4: The accuracy curves for training on the CIRFAR100 dataset with the WRN-40-4 network.  
Figure 4(a) shows the cross-entropy loss of multiple methods. The teacher correction method can effectively reduce the corresponding cross-entropy loss. Figure 4(b) gives the KL-divergence loss. We computed the KL-divergence between the output and a uniform distribution as LSR without using it in backpropagation for the baseline. Comparing to the baseline, the lower KL-divergence loss of LSR presents the regularization effect. The loss is further reduced on LSR-OS by choosing a softmax temperature larger than one (Figure 2(f)). LSR-OS-TC further reduces the loss by learning from a clearer teacher than the uniform distribution (Figure 2(g)).