# Revisiting Label Smoothing Regularization with Knowledge Distillation
###### Abstract  
Label Smoothing Regularization (LSR) is a widely used tool to generalize classification models by replacing the one-hot ground truth with smoothed labels. Recent research on LSR has increasingly focused on the correlation between the LSR and Knowledge Distillation (KD), which transfers the knowledge from a teacher model to a lightweight student model by penalizing their output's Kullback-Leibler-divergence. Based on this observation, a Teacher-free Knowledge Distillation (Tf-KD) method was proposed in previous work. Instead of a real teacher model, a handcrafted distribution similar to LSR was used to guide the student learning. Tf-KD is a promising substitute for LSR except for its hard-to-tune and model-dependent hyperparameters. This paper develops a new teacher-free framework LSR-OS-TC, which decomposes the Tf-KD method into two components: model Output Smoothing (OS) and Teacher Correction (TC). Firstly, the LSR-OS extends the LSR method to the KD regime and applies a softer temperature to the model output softmax layer. Output smoothing is critical for stabilizing the KD hyperparameters among different models. Secondly, in the TC part, a larger proportion is assigned to the uniform distribution teacher's right class to provide a more informative teacher. The two-component method was evaluated exhaustively on the image (dataset CIFAR-100, CIFAR-10, and CINIC-10) and audio (dataset GTZAN) classification tasks. The results showed that LSR-OS can improve LSR performance independently with no extra computational cost, especially on several deep neural networks where LSR is ineffective. The further training boost by the TC component showed the effectiveness of our two-component strategy. Overall, LSR-OS-TC is a practical substitution of LSR that can be tuned on one model and directly applied to other models compared to the original Tf-KD method. label smoothing; knowledge distillation; teacher correction; image classification; music genre classification +
Footnote †: journal: Applied Sciences  
distributions are different from each other. By minimizing KL-divergence loss, a student model can mimic the inter-class relationship of the teacher prediction. Note that the softmax temperature \(\tau_{1}\) for the KL-loss in KD is usually larger than one, which is the common choice of traditional cross-entropy loss (\(\tau_{0}=1\)). In addition to its many application in model compression, KD is also used to boost network training with multiple models with an identical architecture [14; 15] or a single-model self-distillation [16; 17; 18]. On the other hand, LSR was first proposed by [PERSON] et al. [19] to regularize the Inception network on the ImageNet dataset. The traditional classification model calculates the cross-entropy loss between the model output and the one-hot ground truth vector, whereas LSR assigns a small ratio for the logits of the incorrect class and reduces the ratio of the ground truth class from one to a reasonably smaller value (Figure 1c). Figure 1: Graphical illustration of different methods. (**a**) Knowledge distillation for model compression; (**b**) traditional classification framework with cross-entropy loss; (**c**) label smoothing regularization; (**d**) label smoothing regularization in the KD form; (**e**) teacher-free knowledge distillation; (**f**) our proposed LSR-OS; (**g**) LSR-OS with teacher correction. Since [PERSON] et al. [19], LSR has been widely used in image classification [20; 21; 22], speech recognition [23], and machine translation [24]. Recently, plenty of research stressed the correlation of label smoothing regularization and knowledge distillation [25; 26; 27; 28; 29]. Reference [26] showed that LSR is equivalent to penalizing the KL-divergence between a uniform distribution and the model output distribution. Since KD penalizes the KL-divergence of the teacher and student distribution, this uniform distribution in LSR provides a virtual teacher model for KD [27] (Figure 1d).