Adaptive Label Smoothing with Self-Knowledge

1 INTRODUCTION . In common practice , a neural network is trained to maximize the expected likelihood of observed targets , and the gradient with respect to the objective updates the learnable model parameters . With hard targets ( one-hot encoded ) , the maximum objective can be approached when a model assigns a high probability mass to the corresponding target label over the output space . That is , due to the normalizing activation functions ( i.e . softmax ) , a model is trained in order for logits to have a marked difference between the target logit and the other classes logits ( Müller et al. , 2019 ) . Despite its wide application and use , the maximum likelihood estimation with hard targets has been found to incur an overconfident problem : the predictive score of a model does not reflect the actual accuracy of the prediction . Consequently , this leads to degradation in model calibration ( Pereyra et al. , 2017 ) , as well as in model performance ( Müller et al. , 2019 ) . Additionally , this problem stands out more clearly with a limited number of samples , as a model is more prone to overfitting . To remedy such phenomenon , Szegedy et al . ( 2016 ) proposed label smoothing , in which one-hot encoded targets are replaced with smoothed targets . Label smoothing has boosted performance in computer vision ( Szegedy et al. , 2016 ) , and has been highly preferred in other domains , such as Natural Language Processing ( Vaswani et al. , 2017 ; Lewis et al. , 2020 ) . However , there are several aspects to be discussed in label smoothing . First , it comes with certain downsides , namely the static smoothing parameter . The smoothing regularizer fails to account for the change in probability mass over the course of training . Despite the fact that a model can benefit from adaptive control of the smoothing extent depending on the signs of overfitting and overconfidence , the smoothing parameter remains fixed throughout training in all instances . Another aspect of label smoothing for consideration is its connection to knowledge distillation ( Hinton et al. , 2015 ) . There have been attempts to bridge label smoothing with knowledge distillation , and the findings suggest the latter is an adaptive form of the former ( Tang et al. , 2021 ; Yuan et al. , 2020 ) . However , the regularization effect on overconfidence by self-knowledge distillation is still poorly understood and explored . To tackle the issues mentioned above , this work presents adaptive label smoothing with selfknowledge as a prior label distribution . Our regularizer allows a model to self-regulate the extent of smoothing based on the entropic level of model probability distribution , varying the amount per sample and per time step . Furthermore , our theoretical analysis suggests that self-knowledge distillation and the adaptive smoothing parameter have a strong regularization effect by rescaling gradients on logit space . Our work validates the efficacy of the proposed regularization method on machine translation tasks and achieves superior results in model performance and model calibration compared to other baselines . 2 PRELIMINARIES & RELATED WORK . 2.1 LABEL SMOOTHING . Label smoothing ( Szegedy et al. , 2016 ) was first introduced to prevent a model from making a peaked probability distribution . Since its introduction , it has been in wide application as a means of regularization ( Vaswani et al. , 2017 ; Lewis et al. , 2020 ) . In label smoothing , one-hot encoded ground-truth label ( y ) and a pre-defined prior label distribution ( q ) are mixed with the weight , the smoothing parameter ( α ) , forming a smoothed ground-truth label . A model with label smoothing is learned to maximize the likelihood of predicting the smoothed label distribution . Specifically , Lls ( x ( n ) , y ( n ) ) = − |C|∑ i=1 ( 1− α ) y ( n ) i logPθ ( yi|x ( n ) ) + αqi logPθ ( yi|x ( n ) ) ( 1 ) |C| denotes the number of classes , ( n ) the index of a sample in a batch , and Pθ the probability distribution mapped by a model . α is commonly set to 0.1 , and remains fixed throughout training ( Vaswani et al. , 2017 ; Lewis et al. , 2020 ) . A popular choice of q is an uniform distribution ( q ∼ U ( |C| ) ) , while unigram distribution is another option for dealing with an imbalanced label distribution ( Vaswani et al. , 2017 ; Szegedy et al. , 2016 ; Müller et al. , 2019 ; Pereyra et al. , 2017 ) . The pre-defined prior label distribution remains unchanged , hence the latter cross-entropy term in Equation 1 is equivalent to minimizing the KL divergence between the model prediction and the predefined label distribution . In line with the idea , Pereyra et al . ( 2017 ) proposed confidence penalty ( ConfPenalty ) that adds negative entropy term to the loss function , thereby minimizing the KL divergence between the uniform distribution and model probability distribution . Ghoshal et al . ( 2021 ) proposed low-rank adaptive label smoothing ( LORAS ) that jointly learns a noise distribution for softening targets and model parameters . Li et al . ( 2020 ) ; Krothapalli & Abbott ( 2020 ) introduced smoothing schemes that are data-dependent . 2.2 KNOWLEDGE DISTILLATION . Knowledge distillation ( Hinton et al. , 2015 ) aims to transfer the dark knowledge of ( commonly ) a larger and better performing teacher model to a student model ( Buciluundefined et al. , 2006 ) . The idea is to mix the ground-truth label with the model probability distribution of a teacher model , the result acting an adaptive version of label smoothing ( Tang et al. , 2021 ) . Lkd ( x ( n ) , y ( n ) ) = − |C|∑ i=1 ( 1− α ) y ( n ) i logPθ ( yi|x ( n ) ) + αPφ ( yi|x ( n ) ) logPθ ( yi|x ( n ) ) ( 2 ) φ and θ denote the parameters of a teacher model and a student model respectively . Similar to label smoothing , φ remains unchanged in training ; thus a student model is learned to minimize the KL divergence between its probability distribution and that of the teacher model . When Pφ follows a uniform distribution with temperature set to 1 , the loss function of knowledge distillation is identical to that of uniform label smoothing . Training a large teacher model can be computationally expensive ; for this reason , there have been attempts to replace the teacher model with the student model itself , called self-knowledge distillation ( Zhang et al. , 2019 ; Yuan et al. , 2020 ; Kim et al. , 2021 ; Zhang & Sabuncu , 2020 ) . TF-KD ( Yuan et al. , 2020 ) trains a student with a pre-trained teacher that is identical to the student in terms of structure . SKD-PRT ( Kim et al. , 2021 ) utilizes the previous epoch checkpoint as a teacher with linear increase in α. Zhang & Sabuncu ( 2020 ) incorporate beta distribution sampling ( BETA ) and self-knowledge distillation ( SD ) , and introduce instance-specific prior label distribution . Yun et al . ( 2020 ) utilize self-knowledge distillation to minimize the predictive distribution of samples with the same class , encouraging consistent probability distribution within the same class . 3 METHODOLOGY . The core components of label smoothing are two-fold : smoothing parameter ( α ) and prior label distribution . The components determine how much to smooth the target label using which distribution , a process that requires careful choice of selection . In this section , we illustrate how to make the smoothing parameter adaptive . We also demonstrate how our adaptive smoothing parameter and self-knowledge distillation as a prior distribution act as a form of regularization with theoretical analysis on the gradients . 3.1 ADAPTIVE α An intuitive and ideal way of softening the hard target is to bring dynamic nature into choosing α ; a sample with low entropic level in model prediction receives a high smoothing parameter to further smooth the target label . In another scenario , in which high entropy of model prediction ( flat distribution ) is seen , the smoothing factor is decreased . With the intuition , our method computes the smoothing parameter on the fly during the forward propagation in training , relying on the entropic level of model probability distribution per sample , and per time step in case of sequential classification.1 H ( Pθ ( y|x ( n ) ) ) = − |C|∑ i=1 Pθ ( yi|x ( n ) ) logPθ ( yi|x ( n ) ) ( 3 ) The entropy quantifies the level of probability mass distributed across the label space ; therefore , low entropy is an indication of overfitting and overconfidence ( Pereyra et al. , 2017 ; Meister et al. , 2020 ) . Since entropy does not have a fixed range between 0 and 1 , one simple scheme is to normalize the entropy with maximum entropy ( log |C| ) . Hence , the normalization is capable of handling variable 1For notational simplicity , time step is not included in the equation hereafter . size of class set among different datasets . α ( n ) = 1− H ( Pθ ( y|x ( n ) ) ) log |C| ( 4 ) With this mechanism , a sample with high entropy is trained with low α , and a sample with low entropy receives high α . The computation for α is excluded from the computation graph for the gradient calculation , hence , the gradient does not flow through adaptive α ( n ) . There are two essential benefits of adopting the adaptive smoothing parameter . As the smoothing extent is determined by its own probability mass over the output space , the hyperparameter search on α is removed . Furthermore , it is strongly connected to the gradient rescaling effect on selfknowledge distillation , which will be dealt in Section 3.3 in detail . 3.2 SELF-KNOWLEDGE AS A PRIOR . Similar to ( Kim et al. , 2021 ; Liu et al. , 2021 ) , our regularizer loads a past student model checkpoint as teacher network parameters in the course of training . The intuition is to utilize past selfknowledge which is a generalized supervision , thereby hindering the model from overfitting to observations in the training set . φ = argmax θ g ( f ( X ′ ; θi ) , Y ′ ) , where θi ∈ Θ ( 5 ) Θ is a set of past model checkpoints in training , and function f is a specific task , which in our work is machine translation . X ′ and Y ′ are sets of input and ground-truth samples from a validation dataset2 , and the function g could be any proper evaluation metric for model generalization ( i.e . accuracy ) .3 Our work utilizes the n-gram matching score , BLEU ( Papineni et al. , 2002 ) being the function for finding the prior label distribution ( q ∼ Pφ ) . Combining the adaptive smoothing parameter and self-knowledge as a prior distribution , our loss function is as follows : L ( x ( n ) , y ( n ) ) = − |C|∑ i=1 ( 1− α ( n ) ) y ( n ) i logPθ ( yi|x ( n ) ) + α ( n ) Pφ ( yi|x ( n ) ) logPθ ( yi|x ( n ) ) ( 6 )

Summary: Label smoothing is a popular approach to regularize modern neural networks. However, the amount of smoothing is the same across all samples in the dataset, which can be sub-optimal. In this paper, the authors of the paper proposed a novel adaptive label smoothing method so that each sample gets a different amount of smoothing. The key insight of the paper involves using the entropy of predictions from past timestamps as a way to quantify the amount of label smoothing applied to samples. The authors of the paper demonstrate empirically the effectiveness of the proposed method.