Adaptive Label Smoothing with Self-Knowledge

1 INTRODUCTION . In common practice , a neural network is trained to maximize the expected likelihood of observed targets , and the gradient with respect to the objective updates the learnable model parameters . With hard targets ( one-hot encoded ) , the maximum objective can be approached when a model assigns a high probability mass to the corresponding target label over the output space . That is , due to the normalizing activation functions ( i.e . softmax ) , a model is trained in order for logits to have a marked difference between the target logit and the other classes logits ( Müller et al. , 2019 ) . Despite its wide application and use , the maximum likelihood estimation with hard targets has been found to incur an overconfident problem : the predictive score of a model does not reflect the actual accuracy of the prediction . Consequently , this leads to degradation in model calibration ( Pereyra et al. , 2017 ) , as well as in model performance ( Müller et al. , 2019 ) . Additionally , this problem stands out more clearly with a limited number of samples , as a model is more prone to overfitting . To remedy such phenomenon , Szegedy et al . ( 2016 ) proposed label smoothing , in which one-hot encoded targets are replaced with smoothed targets . Label smoothing has boosted performance in computer vision ( Szegedy et al. , 2016 ) , and has been highly preferred in other domains , such as Natural Language Processing ( Vaswani et al. , 2017 ; Lewis et al. , 2020 ) . However , there are several aspects to be discussed in label smoothing . First , it comes with certain downsides , namely the static smoothing parameter . The smoothing regularizer fails to account for the change in probability mass over the course of training . Despite the fact that a model can benefit from adaptive control of the smoothing extent depending on the signs of overfitting and overconfidence , the smoothing parameter remains fixed throughout training in all instances . Another aspect of label smoothing for consideration is its connection to knowledge distillation ( Hinton et al. , 2015 ) . There have been attempts to bridge label smoothing with knowledge distillation , and the findings suggest the latter is an adaptive form of the former ( Tang et al. , 2021 ; Yuan et al. , 2020 ) . However , the regularization effect on overconfidence by self-knowledge distillation is still poorly understood and explored . To tackle the issues mentioned above , this work presents adaptive label smoothing with selfknowledge as a prior label distribution . Our regularizer allows a model to self-regulate the extent of smoothing based on the entropic level of model probability distribution , varying the amount per sample and per time step . Furthermore , our theoretical analysis suggests that self-knowledge distillation and the adaptive smoothing parameter have a strong regularization effect by rescaling gradients on logit space . Our work validates the efficacy of the proposed regularization method on machine translation tasks and achieves superior results in model performance and model calibration compared to other baselines . 2 PRELIMINARIES & RELATED WORK . 2.1 LABEL SMOOTHING . Label smoothing ( Szegedy et al. , 2016 ) was first introduced to prevent a model from making a peaked probability distribution . Since its introduction , it has been in wide application as a means of regularization ( Vaswani et al. , 2017 ; Lewis et al. , 2020 ) . In label smoothing , one-hot encoded ground-truth label ( y ) and a pre-defined prior label distribution ( q ) are mixed with the weight , the smoothing parameter ( α ) , forming a smoothed ground-truth label . A model with label smoothing is learned to maximize the likelihood of predicting the smoothed label distribution . Specifically , Lls ( x ( n ) , y ( n ) ) = − |C|∑ i=1 ( 1− α ) y ( n ) i logPθ ( yi|x ( n ) ) + αqi logPθ ( yi|x ( n ) ) ( 1 ) |C| denotes the number of classes , ( n ) the index of a sample in a batch , and Pθ the probability distribution mapped by a model . α is commonly set to 0.1 , and remains fixed throughout training ( Vaswani et al. , 2017 ; Lewis et al. , 2020 ) . A popular choice of q is an uniform distribution ( q ∼ U ( |C| ) ) , while unigram distribution is another option for dealing with an imbalanced label distribution ( Vaswani et al. , 2017 ; Szegedy et al. , 2016 ; Müller et al. , 2019 ; Pereyra et al. , 2017 ) . The pre-defined prior label distribution remains unchanged , hence the latter cross-entropy term in Equation 1 is equivalent to minimizing the KL divergence between the model prediction and the predefined label distribution . In line with the idea , Pereyra et al . ( 2017 ) proposed confidence penalty ( ConfPenalty ) that adds negative entropy term to the loss function , thereby minimizing the KL divergence between the uniform distribution and model probability distribution . Ghoshal et al . ( 2021 ) proposed low-rank adaptive label smoothing ( LORAS ) that jointly learns a noise distribution for softening targets and model parameters . Li et al . ( 2020 ) ; Krothapalli & Abbott ( 2020 ) introduced smoothing schemes that are data-dependent . 2.2 KNOWLEDGE DISTILLATION . Knowledge distillation ( Hinton et al. , 2015 ) aims to transfer the dark knowledge of ( commonly ) a larger and better performing teacher model to a student model ( Buciluundefined et al. , 2006 ) . The idea is to mix the ground-truth label with the model probability distribution of a teacher model , the result acting an adaptive version of label smoothing ( Tang et al. , 2021 ) . Lkd ( x ( n ) , y ( n ) ) = − |C|∑ i=1 ( 1− α ) y ( n ) i logPθ ( yi|x ( n ) ) + αPφ ( yi|x ( n ) ) logPθ ( yi|x ( n ) ) ( 2 ) φ and θ denote the parameters of a teacher model and a student model respectively . Similar to label smoothing , φ remains unchanged in training ; thus a student model is learned to minimize the KL divergence between its probability distribution and that of the teacher model . When Pφ follows a uniform distribution with temperature set to 1 , the loss function of knowledge distillation is identical to that of uniform label smoothing . Training a large teacher model can be computationally expensive ; for this reason , there have been attempts to replace the teacher model with the student model itself , called self-knowledge distillation ( Zhang et al. , 2019 ; Yuan et al. , 2020 ; Kim et al. , 2021 ; Zhang & Sabuncu , 2020 ) . TF-KD ( Yuan et al. , 2020 ) trains a student with a pre-trained teacher that is identical to the student in terms of structure . SKD-PRT ( Kim et al. , 2021 ) utilizes the previous epoch checkpoint as a teacher with linear increase in α. Zhang & Sabuncu ( 2020 ) incorporate beta distribution sampling ( BETA ) and self-knowledge distillation ( SD ) , and introduce instance-specific prior label distribution . Yun et al . ( 2020 ) utilize self-knowledge distillation to minimize the predictive distribution of samples with the same class , encouraging consistent probability distribution within the same class . 3 METHODOLOGY . The core components of label smoothing are two-fold : smoothing parameter ( α ) and prior label distribution . The components determine how much to smooth the target label using which distribution , a process that requires careful choice of selection . In this section , we illustrate how to make the smoothing parameter adaptive . We also demonstrate how our adaptive smoothing parameter and self-knowledge distillation as a prior distribution act as a form of regularization with theoretical analysis on the gradients . 3.1 ADAPTIVE α An intuitive and ideal way of softening the hard target is to bring dynamic nature into choosing α ; a sample with low entropic level in model prediction receives a high smoothing parameter to further smooth the target label . In another scenario , in which high entropy of model prediction ( flat distribution ) is seen , the smoothing factor is decreased . With the intuition , our method computes the smoothing parameter on the fly during the forward propagation in training , relying on the entropic level of model probability distribution per sample , and per time step in case of sequential classification.1 H ( Pθ ( y|x ( n ) ) ) = − |C|∑ i=1 Pθ ( yi|x ( n ) ) logPθ ( yi|x ( n ) ) ( 3 ) The entropy quantifies the level of probability mass distributed across the label space ; therefore , low entropy is an indication of overfitting and overconfidence ( Pereyra et al. , 2017 ; Meister et al. , 2020 ) . Since entropy does not have a fixed range between 0 and 1 , one simple scheme is to normalize the entropy with maximum entropy ( log |C| ) . Hence , the normalization is capable of handling variable 1For notational simplicity , time step is not included in the equation hereafter . size of class set among different datasets . α ( n ) = 1− H ( Pθ ( y|x ( n ) ) ) log |C| ( 4 ) With this mechanism , a sample with high entropy is trained with low α , and a sample with low entropy receives high α . The computation for α is excluded from the computation graph for the gradient calculation , hence , the gradient does not flow through adaptive α ( n ) . There are two essential benefits of adopting the adaptive smoothing parameter . As the smoothing extent is determined by its own probability mass over the output space , the hyperparameter search on α is removed . Furthermore , it is strongly connected to the gradient rescaling effect on selfknowledge distillation , which will be dealt in Section 3.3 in detail . 3.2 SELF-KNOWLEDGE AS A PRIOR . Similar to ( Kim et al. , 2021 ; Liu et al. , 2021 ) , our regularizer loads a past student model checkpoint as teacher network parameters in the course of training . The intuition is to utilize past selfknowledge which is a generalized supervision , thereby hindering the model from overfitting to observations in the training set . φ = argmax θ g ( f ( X ′ ; θi ) , Y ′ ) , where θi ∈ Θ ( 5 ) Θ is a set of past model checkpoints in training , and function f is a specific task , which in our work is machine translation . X ′ and Y ′ are sets of input and ground-truth samples from a validation dataset2 , and the function g could be any proper evaluation metric for model generalization ( i.e . accuracy ) .3 Our work utilizes the n-gram matching score , BLEU ( Papineni et al. , 2002 ) being the function for finding the prior label distribution ( q ∼ Pφ ) . Combining the adaptive smoothing parameter and self-knowledge as a prior distribution , our loss function is as follows : L ( x ( n ) , y ( n ) ) = − |C|∑ i=1 ( 1− α ( n ) ) y ( n ) i logPθ ( yi|x ( n ) ) + α ( n ) Pφ ( yi|x ( n ) ) logPθ ( yi|x ( n ) ) ( 6 )

Summary: The paper proposes a simple yet effective way of smoothing the labels for each data point. The smoothing parameter alpha is dynamically adapted for each data point based on the (normalized) entropy of the predicted label-distribution for this data point. The distribution that is used for label-smoothing originates from the same but at earlier epoch of the training, i.e., a teacher-student learning framework like in knowledge distillation is employed, where the teacher is a model learned in an earlier epoch. From among these earlier model-candidates, the one is chosen that obtains the best evaluation metric g (Eq. 5). Function g does not have to be the training loss, but can be the (possibly different) evaluation metric. The validity of this approach is theoretically supported by a gradient analysis, and experimentally corroborated by improved evaluation metrics, improved calibration, and an ablation study.