Better Supervisory Signals by Observing Learning Paths

1 INTRODUCTION . In multi-class classification problems , we usually supervise our model with “ one-hot ” labels : label vectors y which have yi = 1 for one i , and 0 for all other dimensions . Over time , however , it has gradually become clear that this “ default ” setup is not always the best choice in practice , as measured by the generalization performance of the resulting model.1 One alternative is to summarize a distribution of human annotations , as Peterson et al . ( 2019 ) did on CIFAR-10 . An alternative approach is label smoothing ( e.g . Szegedy et al. , 2016 ) , mixing between a one-hot label and the uniform distribution . Knowledge distillation ( KD ) , first training a teacher network on the training set and then a student network on the teacher ’ s output probabilities , was originally proposed for model compression ( Hinton et al. , 2015 ) but can also be thought of as refining the supervision signal : it provides “ soft ” teacher outputs rather than hard labels to the student . Knowledge distillation is promising because it requires no additional annotation effort , but , unlike label smoothing , can still provide sample-specific refinement . Perhaps surprisingly , knowledge distillation can improve student performance even when the teacher is of exactly the same form as the student ; this is known as self-distillation ( Furlanello et al. , 2018 ; Zhang et al. , 2019 ) . There have been many recent attempts to explain the success of knowledge distillation and specifically self-distillation ( e.g . Menon et al. , 2021 ; Allen-Zhu & Li , 2020 ; Tang et al. , 2020 ) , from both optimization and supervision perspectives . We focus on the latter area , where it is usually claimed that the teacher provides useful “ dark knowledge ” to the student through its labels . Inspired by this line of work , we further explore why and how this improved supervisory signal emerges during the teacher ’ s one-hot training . Specifically , we first clarify that given any input sample x , good supervision signals should be close ( in L2 distance ) to the ground truth categorical distribution , i.e. , p∗ ( y | x ) . We then show that a neural network ( NN ) can automatically refine “ bad labels ” , those where p∗ ( y | x ) is far from the one-hot vector from the training set.2 During one-hot training , the model prediction on such a sample first moves towards p∗ ( y | x ) , and then slowly 1We are concerned here only with traditional i.i.d . generalization from the training set to held-out test sets . 2This might be because x is ambiguous ( perhaps p∗ ( y | x ) is flat , or we simply got a sample from a less- likely class ) , or because the one-hot label has been corrupted through label noise or otherwise is “ wrong. ” converges to its supervisory label , following a “ zig-zag ” pattern . A well-trained teacher , one that does not overfit to particular training labels , can thus provide supervisory signals closer to p∗ ( y | x ) . We find this phenomenon is quite common in gradient descent training , and provide some analytical justification ( Section 3.3 ) . Based on our explanations , we point out that this signal can be better achieved if we take a moving average of the teacher ’ s prediction , an algorithm we term Filter-KD . This approach provides better supervision and hence better final performance , especially when there are more bad labels . 2 SUPERVISION INFLUENCES GENERALIZATION . We will begin by establishing the classification setting , and making clearer how the choice of supervisory signal affects the learned model . 2.1 CHOICES OF SUPERVISION SIGNAL . In K-way classification , our goal is to learn a mapping f : X → ∆K that can minimize the risk R ( f ) , E ( x , y ) ∼P [ L ( y , f ( x ) ] = ∫ p ( x ) p ( y|x ) L ( y , f ( x ) ) dxdy , ( 1 ) where L ( y , f ( x ) ) is the loss function measuring the gap between model ’ s prediction and the true label ( e.g . cross-entropy or square loss ) . The true label y is an integer ranging from 1 to K , and the input signal x is usually high dimensional , e.g. , an image or a sequence of word embeddings . The joint distribution of ( x , y ) is P , whose density3 can be written as p ( x ) p ( y|x ) . In practice , as P is unknown , we instead ( approximately ) minimize the empirical risk Remp ( f , D ) , N∑ n=1 K∑ k=1 1 N 1 ( yn=k ) L ( k , f ( xn ) ) = N∑ n=1 1 N eTynL ( f ( xn ) ) , ( 2 ) where 1 ( yn = k ) is an indicator function which equals 1 if yn = k or 0 otherwise , and eyn ∈ { 0 , 1 } K is its one-hot vector form . L ( f ( xn ) ) = ( L ( 1 , f ( xn ) ) , . . . , L ( K , f ( xn ) ) ) ∈ RK is the loss for each possible label . In Remp , the N training pairs D , { ( xn , yn ) } Nn=1 are sampled i.i.d . from P. Comparing ( 2 ) to ( 1 ) , we find that p ( x ) is approximated by an uniform distribution over the samples , which is reasonable . However , using an indicator function ( i.e. , one-hot distribution ) to approximate p ( y | x ) bears more consideration . For example , if a data point x is quite vague and its true p ( y|x ) is flat or multimodal , we might hope to see x multiple times with different label y during training . But actually , most datasets have only one copy of each x , so we only ever see one corresponding ey . Although Remp is an unbiased estimator for R , if we used a better ( e.g . lower-variance ) estimate of p ( y | x ) , we could get a better estimate for R and thus , hopefully , better generalization . Specifically , suppose we were provided a “ target ” distribution ptar ( y | x ) ( written in vector form as ptar ( x ) ) for each training point x , as D′ = { ( xn , ptar ( xn ) ) } Nn=1 . Then we could use Rtar ( f , D′ ) , N∑ n=1 K∑ k=1 1 N ptar ( yn = k | xn ) L ( k , f ( xn ) ) = N∑ n=1 1 N ptar ( xn ) TL ( f ( xn ) ) . ( 3 ) Standard training with Remp is a special case of Rtar , using ptar ( xn ) = eyn . The CIFAR-10H dataset ( Peterson et al. , 2019 ) is one attempt at a different ptar , using multiple human annotators to estimate ptar . Label smoothing ( e.g . Szegedy et al. , 2016 ) sets ptar to a convex combination of ey and the constant vector 1K 1 . In knowledge distillation ( KD ; Hinton et al. , 2015 ) , a teacher is first trained on D , then a student learns from D′ with ptar based on the teacher ’ s outputs . All three approaches yield improvements over standard training with Remp . 3Because we are working with classification problems , we use densities with respect to a product of some arbitrary measure on x ( probably Lebesgue ) with counting measure on y , and assume that these densities exist for notational convenience . None of our arguments will depend on the choice of base measure . 2.2 MEASURING THE QUALITY OF SUPERVISION . Choosing a different ptar , then , can lead to a better final model . Can we characterize which ptar will do well ? We propose the following , as a general trend . Hypothesis 1 . Suppose we train a model supervised by ptar , that is , we minimize Rtar ( f , D′ ) . Then , smaller average L2 distance between ptar and the ground truth p ∗ on these samples , i.e . small Ex [ ‖ptar ( x ) − p∗ ( x ) ‖2 ] , will in general lead to better generalization performance . This hypothesis is suggested by Proposition 3 of Menon et al . ( 2021 ) , which shows that for any predictor f and loss bounded as L ( y , ŷ ) ≤ ` , E x [ ( Rtar ( f , D′ ) −R ( f ) ) 2 ] ≤ 1 N Var x [ pTtarL ( f ( x ) ) ] +K ` 2 ( E x [ ‖ptar ( x ) − p∗ ( x ) ‖2 ] ) 2 . ( 4 ) When N is large , the second term will dominate the right-hand side , implying smaller average ‖ptar − p∗‖ will lead to Rtar being a better approximation of the true risk R. Minimizing a better Rtar should then lead to a better model , suggesting that the quality of the supervision signal can be roughly measured by its L2 distance to the ground truth p∗ . To further support this hypothesis , we conduct experiments on a synthetic Gaussian problem ( Figure 1 ( a ) ; details in Appendix B ) , where we can easily calculate p∗ ( y | x ) for each sample . We first generate several different ptar by adding noise 4 to the ground truth p∗ , then train simple 3-layer NNs under that supervision . We also show five baselines : one-hot training ( OHT ) , label smoothing ( LS ) , KD , early-stopped KD ( ESKD ) , and ground truth ( GT ) supervision ( using p∗ ) . KD refers to a teacher trained to convergence , while ESKD uses a teacher stopped early based on validation accuracy . We early-stop the student ’ s training in all settings . From Figure 1 ( b-c ) , it is clear that smaller ‖ptar − p∗‖2 leads to better generalization performance , as measured either by accuracy ( ACC ) or expected calibration error ( ECE ) 5 on a held-out test set . Appendix A has more detailed results . 3 INSIGHTS FROM THE LEARNING PATH . In the toy example of Section 2 , we see that ESKD outperforms other baselines in accuracy by a substantial margin ( and all baselines are roughly tied in ECE ) . We expect that supervision with smaller ‖ptar − p∗‖2 leads to better generalization performance , but it is not clear how better ptar emerges from when the teacher in ESKD is trained using one-hot labels . This section will answer this , by observing the learning paths of training samples . 4Whenever we mention adding noise to p∗ , we mean we add independent noise to each dimension , and then re-normalize it to be a distribution . Large noise can thus flip the “ correct ” label . 5ECE measures the calibration of a model ( Guo et al. , 2017 ) . Briefly , lower ECE means the model ’ s confidence in its predictions is more accurate . See Appendix A for details .

Summary: This paper proposes an explanation for the success of distillation. It first experiments with synthetic Gaussian data. On synthetic data, it shows that distillation works better from an early-stopped model. It probes why, and finds that, when the one-hot label is far from the true conditional distribution, the early-stopped model’s distribution tends to be closer to the true distribution than either the one-hot label or the converged model. The early-stopped model is better because, over the course of training, the model tends to produce a distribution close to the true one before finally overfitting the one-hot label. Some examples show similar patterns for real-world networks/datasets, provided the predictions over training are smoothed. Motivated by these results, the paper proposes to use an averaged distribution of labels from different steps of training rather than the distribution at the end of training, and shows that this improves performance by a little bit.