Better Supervisory Signals by Observing Learning Paths

1 INTRODUCTION . In multi-class classification problems , we usually supervise our model with “ one-hot ” labels : label vectors y which have yi = 1 for one i , and 0 for all other dimensions . Over time , however , it has gradually become clear that this “ default ” setup is not always the best choice in practice , as measured by the generalization performance of the resulting model.1 One alternative is to summarize a distribution of human annotations , as Peterson et al . ( 2019 ) did on CIFAR-10 . An alternative approach is label smoothing ( e.g . Szegedy et al. , 2016 ) , mixing between a one-hot label and the uniform distribution . Knowledge distillation ( KD ) , first training a teacher network on the training set and then a student network on the teacher ’ s output probabilities , was originally proposed for model compression ( Hinton et al. , 2015 ) but can also be thought of as refining the supervision signal : it provides “ soft ” teacher outputs rather than hard labels to the student . Knowledge distillation is promising because it requires no additional annotation effort , but , unlike label smoothing , can still provide sample-specific refinement . Perhaps surprisingly , knowledge distillation can improve student performance even when the teacher is of exactly the same form as the student ; this is known as self-distillation ( Furlanello et al. , 2018 ; Zhang et al. , 2019 ) . There have been many recent attempts to explain the success of knowledge distillation and specifically self-distillation ( e.g . Menon et al. , 2021 ; Allen-Zhu & Li , 2020 ; Tang et al. , 2020 ) , from both optimization and supervision perspectives . We focus on the latter area , where it is usually claimed that the teacher provides useful “ dark knowledge ” to the student through its labels . Inspired by this line of work , we further explore why and how this improved supervisory signal emerges during the teacher ’ s one-hot training . Specifically , we first clarify that given any input sample x , good supervision signals should be close ( in L2 distance ) to the ground truth categorical distribution , i.e. , p∗ ( y | x ) . We then show that a neural network ( NN ) can automatically refine “ bad labels ” , those where p∗ ( y | x ) is far from the one-hot vector from the training set.2 During one-hot training , the model prediction on such a sample first moves towards p∗ ( y | x ) , and then slowly 1We are concerned here only with traditional i.i.d . generalization from the training set to held-out test sets . 2This might be because x is ambiguous ( perhaps p∗ ( y | x ) is flat , or we simply got a sample from a less- likely class ) , or because the one-hot label has been corrupted through label noise or otherwise is “ wrong. ” converges to its supervisory label , following a “ zig-zag ” pattern . A well-trained teacher , one that does not overfit to particular training labels , can thus provide supervisory signals closer to p∗ ( y | x ) . We find this phenomenon is quite common in gradient descent training , and provide some analytical justification ( Section 3.3 ) . Based on our explanations , we point out that this signal can be better achieved if we take a moving average of the teacher ’ s prediction , an algorithm we term Filter-KD . This approach provides better supervision and hence better final performance , especially when there are more bad labels . 2 SUPERVISION INFLUENCES GENERALIZATION . We will begin by establishing the classification setting , and making clearer how the choice of supervisory signal affects the learned model . 2.1 CHOICES OF SUPERVISION SIGNAL . In K-way classification , our goal is to learn a mapping f : X → ∆K that can minimize the risk R ( f ) , E ( x , y ) ∼P [ L ( y , f ( x ) ] = ∫ p ( x ) p ( y|x ) L ( y , f ( x ) ) dxdy , ( 1 ) where L ( y , f ( x ) ) is the loss function measuring the gap between model ’ s prediction and the true label ( e.g . cross-entropy or square loss ) . The true label y is an integer ranging from 1 to K , and the input signal x is usually high dimensional , e.g. , an image or a sequence of word embeddings . The joint distribution of ( x , y ) is P , whose density3 can be written as p ( x ) p ( y|x ) . In practice , as P is unknown , we instead ( approximately ) minimize the empirical risk Remp ( f , D ) , N∑ n=1 K∑ k=1 1 N 1 ( yn=k ) L ( k , f ( xn ) ) = N∑ n=1 1 N eTynL ( f ( xn ) ) , ( 2 ) where 1 ( yn = k ) is an indicator function which equals 1 if yn = k or 0 otherwise , and eyn ∈ { 0 , 1 } K is its one-hot vector form . L ( f ( xn ) ) = ( L ( 1 , f ( xn ) ) , . . . , L ( K , f ( xn ) ) ) ∈ RK is the loss for each possible label . In Remp , the N training pairs D , { ( xn , yn ) } Nn=1 are sampled i.i.d . from P. Comparing ( 2 ) to ( 1 ) , we find that p ( x ) is approximated by an uniform distribution over the samples , which is reasonable . However , using an indicator function ( i.e. , one-hot distribution ) to approximate p ( y | x ) bears more consideration . For example , if a data point x is quite vague and its true p ( y|x ) is flat or multimodal , we might hope to see x multiple times with different label y during training . But actually , most datasets have only one copy of each x , so we only ever see one corresponding ey . Although Remp is an unbiased estimator for R , if we used a better ( e.g . lower-variance ) estimate of p ( y | x ) , we could get a better estimate for R and thus , hopefully , better generalization . Specifically , suppose we were provided a “ target ” distribution ptar ( y | x ) ( written in vector form as ptar ( x ) ) for each training point x , as D′ = { ( xn , ptar ( xn ) ) } Nn=1 . Then we could use Rtar ( f , D′ ) , N∑ n=1 K∑ k=1 1 N ptar ( yn = k | xn ) L ( k , f ( xn ) ) = N∑ n=1 1 N ptar ( xn ) TL ( f ( xn ) ) . ( 3 ) Standard training with Remp is a special case of Rtar , using ptar ( xn ) = eyn . The CIFAR-10H dataset ( Peterson et al. , 2019 ) is one attempt at a different ptar , using multiple human annotators to estimate ptar . Label smoothing ( e.g . Szegedy et al. , 2016 ) sets ptar to a convex combination of ey and the constant vector 1K 1 . In knowledge distillation ( KD ; Hinton et al. , 2015 ) , a teacher is first trained on D , then a student learns from D′ with ptar based on the teacher ’ s outputs . All three approaches yield improvements over standard training with Remp . 3Because we are working with classification problems , we use densities with respect to a product of some arbitrary measure on x ( probably Lebesgue ) with counting measure on y , and assume that these densities exist for notational convenience . None of our arguments will depend on the choice of base measure . 2.2 MEASURING THE QUALITY OF SUPERVISION . Choosing a different ptar , then , can lead to a better final model . Can we characterize which ptar will do well ? We propose the following , as a general trend . Hypothesis 1 . Suppose we train a model supervised by ptar , that is , we minimize Rtar ( f , D′ ) . Then , smaller average L2 distance between ptar and the ground truth p ∗ on these samples , i.e . small Ex [ ‖ptar ( x ) − p∗ ( x ) ‖2 ] , will in general lead to better generalization performance . This hypothesis is suggested by Proposition 3 of Menon et al . ( 2021 ) , which shows that for any predictor f and loss bounded as L ( y , ŷ ) ≤ ` , E x [ ( Rtar ( f , D′ ) −R ( f ) ) 2 ] ≤ 1 N Var x [ pTtarL ( f ( x ) ) ] +K ` 2 ( E x [ ‖ptar ( x ) − p∗ ( x ) ‖2 ] ) 2 . ( 4 ) When N is large , the second term will dominate the right-hand side , implying smaller average ‖ptar − p∗‖ will lead to Rtar being a better approximation of the true risk R. Minimizing a better Rtar should then lead to a better model , suggesting that the quality of the supervision signal can be roughly measured by its L2 distance to the ground truth p∗ . To further support this hypothesis , we conduct experiments on a synthetic Gaussian problem ( Figure 1 ( a ) ; details in Appendix B ) , where we can easily calculate p∗ ( y | x ) for each sample . We first generate several different ptar by adding noise 4 to the ground truth p∗ , then train simple 3-layer NNs under that supervision . We also show five baselines : one-hot training ( OHT ) , label smoothing ( LS ) , KD , early-stopped KD ( ESKD ) , and ground truth ( GT ) supervision ( using p∗ ) . KD refers to a teacher trained to convergence , while ESKD uses a teacher stopped early based on validation accuracy . We early-stop the student ’ s training in all settings . From Figure 1 ( b-c ) , it is clear that smaller ‖ptar − p∗‖2 leads to better generalization performance , as measured either by accuracy ( ACC ) or expected calibration error ( ECE ) 5 on a held-out test set . Appendix A has more detailed results . 3 INSIGHTS FROM THE LEARNING PATH . In the toy example of Section 2 , we see that ESKD outperforms other baselines in accuracy by a substantial margin ( and all baselines are roughly tied in ECE ) . We expect that supervision with smaller ‖ptar − p∗‖2 leads to better generalization performance , but it is not clear how better ptar emerges from when the teacher in ESKD is trained using one-hot labels . This section will answer this , by observing the learning paths of training samples . 4Whenever we mention adding noise to p∗ , we mean we add independent noise to each dimension , and then re-normalize it to be a distribution . Large noise can thus flip the “ correct ” label . 5ECE measures the calibration of a model ( Guo et al. , 2017 ) . Briefly , lower ECE means the model ’ s confidence in its predictions is more accurate . See Appendix A for details .

Summary: This paper introduces a method for doing knowledge distillation with noisy labels. The contribution consists on representing distilled labels as a weighted moving average of the predicted labels from a teacher as it trains (using noisy one-hot encoded labels). Finally, a student is trained using the distilled labels.