Year: 2021

Abstract:
Better supervision might lead to better performance. In this paper, we first clarify what makes supervision good for a classification problem, and then explain two existing label refining methods: label smoothing and knowledge distillation, in terms of our proposed criterion. To further answer why and how better supervision emerges, we observe the learning path, i.e., the trajectory of the network’s predicted distribution, for each training sample during training. We surprisingly find that the model can spontaneously refine “bad” labels with through a “zig-zag” learning path, which occurs on both toy and real datasets. Observing the learning path not only provides a new perspective for understanding knowledge distillation, overfitting, and learning dynamics, but also reveals that the supervisory signal of a teacher network can be very unstable near the best points in training on real tasks. Inspired by this, we propose a new algorithm, Filter-KD, which further enhances classification performance of knowledge distillation, as verified by experiments in various settings.

heading: 1 INTRODUCTION
text: In multi-class classification problems, we usually supervise our model with “one-hot” labels: label vectors y which have yi = 1 for one i, and 0 for all other dimensions. Over time, however, it has gradually become clear that this “default” setup is not always the best choice in practice, as measured by the generalization performance of the resulting model.1 One alternative is to summarize a distribution of human annotations, as Peterson et al. (2019) did on CIFAR-10. An alternative approach is label smoothing (e.g. Szegedy et al., 2016), mixing between a one-hot label and the uniform distribution. Knowledge distillation (KD), first training a teacher network on the training set and then a student network on the teacher’s output probabilities, was originally proposed for model compression (Hinton et al., 2015) but can also be thought of as refining the supervision signal: it provides “soft” teacher outputs rather than hard labels to the student.
Knowledge distillation is promising because it requires no additional annotation effort, but, unlike label smoothing, can still provide sample-specific refinement. Perhaps surprisingly, knowledge distillation can improve student performance even when the teacher is of exactly the same form as the student; this is known as self-distillation (Furlanello et al., 2018; Zhang et al., 2019). There have been many recent attempts to explain the success of knowledge distillation and specifically self-distillation (e.g. Menon et al., 2021; Allen-Zhu & Li, 2020; Tang et al., 2020), from both optimization and supervision perspectives. We focus on the latter area, where it is usually claimed that the teacher provides useful “dark knowledge” to the student through its labels.
Inspired by this line of work, we further explore why and how this improved supervisory signal emerges during the teacher’s one-hot training. Specifically, we first clarify that given any input sample x, good supervision signals should be close (in L2 distance) to the ground truth categorical distribution, i.e., p∗(y | x). We then show that a neural network (NN) can automatically refine “bad labels”, those where p∗(y | x) is far from the one-hot vector from the training set.2 During one-hot training, the model prediction on such a sample first moves towards p∗(y | x), and then slowly
1We are concerned here only with traditional i.i.d. generalization from the training set to held-out test sets. 2This might be because x is ambiguous (perhaps p∗(y | x) is flat, or we simply got a sample from a less-
likely class), or because the one-hot label has been corrupted through label noise or otherwise is “wrong.”
converges to its supervisory label, following a “zig-zag” pattern. A well-trained teacher, one that does not overfit to particular training labels, can thus provide supervisory signals closer to p∗(y | x). We find this phenomenon is quite common in gradient descent training, and provide some analytical justification (Section 3.3). Based on our explanations, we point out that this signal can be better achieved if we take a moving average of the teacher’s prediction, an algorithm we term Filter-KD. This approach provides better supervision and hence better final performance, especially when there are more bad labels.

heading: 2 SUPERVISION INFLUENCES GENERALIZATION
text: We will begin by establishing the classification setting, and making clearer how the choice of supervisory signal affects the learned model.

heading: 2.1 CHOICES OF SUPERVISION SIGNAL
text: In K-way classification, our goal is to learn a mapping f : X → ∆K that can minimize the risk
R(f) , E (x,y)∼P [L(y, f(x)] =
∫ p(x) p(y|x)L(y, f(x)) dxdy, (1)
where L(y, f(x)) is the loss function measuring the gap between model’s prediction and the true label (e.g. cross-entropy or square loss). The true label y is an integer ranging from 1 to K, and the input signal x is usually high dimensional, e.g., an image or a sequence of word embeddings. The joint distribution of (x, y) is P, whose density3 can be written as p(x)p(y|x). In practice, as P is unknown, we instead (approximately) minimize the empirical risk
Remp(f,D) , N∑ n=1 K∑ k=1 1 N 1(yn=k)L (k, f(xn)) = N∑ n=1 1 N eTynL(f(xn)), (2)
where 1(yn = k) is an indicator function which equals 1 if yn = k or 0 otherwise, and eyn ∈ {0, 1}K is its one-hot vector form. L(f(xn)) = (L(1, f(xn)), . . . , L(K, f(xn))) ∈ RK is the loss for each possible label. In Remp, the N training pairs D , {(xn, yn)}Nn=1 are sampled i.i.d. from P. Comparing (2) to (1), we find that p(x) is approximated by an uniform distribution over the samples, which is reasonable. However, using an indicator function (i.e., one-hot distribution) to approximate p(y | x) bears more consideration. For example, if a data point x is quite vague and its true p(y|x) is flat or multimodal, we might hope to see x multiple times with different label y during training. But actually, most datasets have only one copy of each x, so we only ever see one corresponding ey . Although Remp is an unbiased estimator for R, if we used a better (e.g. lower-variance) estimate of p(y | x), we could get a better estimate for R and thus, hopefully, better generalization. Specifically, suppose we were provided a “target” distribution ptar(y | x) (written in vector form as ptar(x)) for each training point x, as D′ = {(xn,ptar(xn))}Nn=1. Then we could use
Rtar(f,D′) , N∑ n=1 K∑ k=1 1 N ptar(yn = k | xn)L(k, f(xn)) = N∑ n=1 1 N ptar(xn) TL(f(xn)). (3)
Standard training with Remp is a special case of Rtar, using ptar(xn) = eyn . The CIFAR-10H dataset (Peterson et al., 2019) is one attempt at a different ptar, using multiple human annotators to estimate ptar. Label smoothing (e.g. Szegedy et al., 2016) sets ptar to a convex combination of ey and the constant vector 1K 1. In knowledge distillation (KD; Hinton et al., 2015), a teacher is first trained on D, then a student learns from D′ with ptar based on the teacher’s outputs. All three approaches yield improvements over standard training with Remp.
3Because we are working with classification problems, we use densities with respect to a product of some arbitrary measure on x (probably Lebesgue) with counting measure on y, and assume that these densities exist for notational convenience. None of our arguments will depend on the choice of base measure.

heading: 2.2 MEASURING THE QUALITY OF SUPERVISION
text: Choosing a different ptar, then, can lead to a better final model. Can we characterize which ptar will do well? We propose the following, as a general trend.
Hypothesis 1. Suppose we train a model supervised by ptar, that is, we minimize Rtar(f,D′). Then, smaller average L2 distance between ptar and the ground truth p
∗ on these samples, i.e. small Ex [‖ptar(x)− p∗(x)‖2], will in general lead to better generalization performance.
This hypothesis is suggested by Proposition 3 of Menon et al. (2021), which shows that for any predictor f and loss bounded as L(y, ŷ) ≤ `,
E x
[ (Rtar(f,D′)−R(f))2 ] ≤ 1 N Var x [ pTtarL(f(x)) ] +K`2 ( E x [‖ptar(x)− p∗(x)‖2] )2 . (4)
When N is large, the second term will dominate the right-hand side, implying smaller average ‖ptar − p∗‖ will lead to Rtar being a better approximation of the true risk R. Minimizing a better Rtar should then lead to a better model, suggesting that the quality of the supervision signal can be roughly measured by its L2 distance to the ground truth p∗.
To further support this hypothesis, we conduct experiments on a synthetic Gaussian problem (Figure 1 (a); details in Appendix B), where we can easily calculate p∗(y | x) for each sample. We first generate several different ptar by adding noise
4 to the ground truth p∗, then train simple 3-layer NNs under that supervision. We also show five baselines: one-hot training (OHT), label smoothing (LS), KD, early-stopped KD (ESKD), and ground truth (GT) supervision (using p∗). KD refers to a teacher trained to convergence, while ESKD uses a teacher stopped early based on validation accuracy. We early-stop the student’s training in all settings. From Figure 1 (b-c), it is clear that smaller ‖ptar − p∗‖2 leads to better generalization performance, as measured either by accuracy (ACC) or expected calibration error (ECE)5 on a held-out test set. Appendix A has more detailed results.

heading: 3 INSIGHTS FROM THE LEARNING PATH
text: In the toy example of Section 2, we see that ESKD outperforms other baselines in accuracy by a substantial margin (and all baselines are roughly tied in ECE). We expect that supervision with smaller ‖ptar − p∗‖2 leads to better generalization performance, but it is not clear how better ptar emerges from when the teacher in ESKD is trained using one-hot labels. This section will answer this, by observing the learning paths of training samples.
4Whenever we mention adding noise to p∗, we mean we add independent noise to each dimension, and then re-normalize it to be a distribution. Large noise can thus flip the “correct” label.
5ECE measures the calibration of a model (Guo et al., 2017). Briefly, lower ECE means the model’s confidence in its predictions is more accurate. See Appendix A for details.

heading: 3.1 PAY MORE ATTENTION TO HARDER SAMPLES
text: For a finer-grained understanding of early stopping the teacher, we would like to better understand how the teacher’s predictions evolve in training. Assuming Hypothesis 1, the main factor is Ex‖q(x) − p∗(x)‖2, where q is the teacher’s output probability distribution. We expect, though, that this term will vary for different x, in part because some samples are simply more difficult to learn; as a proxy for this, we define base difficulty as ‖ey − p∗(x)‖2. This measure is high if:
• x is ambiguous: p∗ has several large components, so there is no one-hot label near p∗. • x is not very ambiguous (there is a one-hot label near p∗), but the sample was “unlucky” and
drew y from a low-probability class.
Figure 2 shows these two quantities at three points in training: initialization, the point where ESKD’s teacher stops, and convergence. At initialization, most points6 have large ‖q(x) − p∗(x)‖2. By the point of early stopping, most q(x) values are roughly near p∗. At convergence, however, q(x) ≈ ey , as the classifier has nearly memorized the training inputs, leading to a diagonal line in the plot.
It is clear the biggest contributors to Ex‖q(x) − p∗(x)‖2 when training a model to convergence are the points with high base difficulty. Per Hypothesis 1, these are the points we should most focus on.

heading: 3.2 LEARNING PATH OF DIFFERENT SAMPLES
text: To better understand how q changes for each sample, we track the model’s outputs of all training samples at each training step. We plot the vectors of three probabilities as points on the simplex (details in Appendix A), and visualize four samples with different base difficulty in Figure 3.
The easy samples very quickly move to the correct location near ey (as indicated by the light color until reaching the corner). The medium sample takes a less direct route, drifting off slightly towards p∗, but still directly approaches ey . The hard sample, however, does something very different: it first approaches p∗, but then veers off towards ey , giving a “zig-zag” path. In both the medium and hard cases, there seems to be some “unknown force” dragging the learning path towards p∗; in both cases, the early stopping point is not quite perfect, but is noticeably closer to p∗ than the final converged point near ey . In other words, during one-hot training, the NNs can spontaneously refine the “bad labels.” Under Hypothesis 1, this partly explains the superior performance of ESKD to KD with a converged teacher.7
We only show four points, because they are quite representative of all samples, under different toydataset settings. We also provide a “zig-zagness score” to further illustrate the correlation between base difficulty and the learning path patterns (refer to Appendix D for more details).
6The curve structure is expected: points with p∗ ≈ ( 1 3 , 1 3 , 1 3 ) are near the middle of the base difficulty range, and their initialization is also fairly ambiguous. 7KD’s practical success is also related to the temperature and optimization effects, among others; better supervision is not the whole story. We discuss the effect of various hyperparameters in Appendix C.

heading: 3.3 EXPLANATION OF PATTERNS IN THE LEARNING PATH
text: Where does this “unknown force” pushing q towards p∗ come from? The following decomposition, proved in Appendix G, will help elucidate its source:
Proposition 1. Let zt(x) , f(wt, x) denote the NN output logits with parameters wt, and qt(x) = Softmax(zt(x)) the corresponding probabilities. Let wt+1 , wt − η∇w ( ptar(xu) TL(qt(xu)) )
be the result of applying one step of SGD to wt on the data point (xu,ptar(xu)) with learning rate η. Then, the change in predictions for a particular observed sample, xo, is
qt+1(xo)− qt(xo) = ηAt(xo) Kt(xo, xu) ( ptar(xu)− qt(xu) ) +O(η2),
where At(xo) = ∇zqt(xo) and Kt(xo, xu) = (∇wz(xo)|wt) (∇wz(xu)|wt)T are K ×K matrices.
The matrix Kt is known as the empirical neural tangent kernel, which we can think of roughly as a notion of “similarity” between xo and xu based on the network’s representation at time t. In general, Kt changes during training in potentially complex ways, but in very wide networks with appropriate random initialization, it is nearly invariant throughout training and nearly independent of the initialization (Jacot et al., 2018; Arora et al., 2019). For more details, see Appendix G.
Figure 4 shows the learning path of a hard sample during one-hot training, say xo, where the label eyo is far from p
∗(xo). In each epoch, w will receive N − 1 updates based on the loss of xu 6= xo (the small blue path), and one update based on xo (the big red path). At any time t, “dissimilar” samples will have small Kt(xo, xu) (as measured, e.g., by its trace), and hence only slightly affect the predicted label for xo. Similar samples will have large Kt(xo, xu), and hence affect its updates much more; because p∗ is hopefully similar for similar x values, it is reasonable to expect that the mean of ey for data points with similar x will be close to p∗(xo). Thus the net effect of updates for xu 6= xo should be to drag q(xo) towards p∗(xo). This is the “unknown force” we observed earlier.
The other force affecting q(xo) during training is, of course, the update based on xo, driving q(xo) towards eyo . How do these two forces balance over the course training? Early on, ‖eyu − qt(xu)‖2 is relatively large for nearly any xu, because near initialization qt(xu) will be relatively flat. Hence the size of the updates for “similar” xu and the update for xo should be comparable, meaning that, if there are at least a few “similar” training points, qt(xo) will move towards p∗(xo). Throughout training, some of these similar samples will become well-classified, so that ‖eyu − qt(xu)‖2 becomes small, and their updates will no longer exert much force on q(xo). Thus, the xo updates begin to dominate, causing the zig-zag pattern as the learning path turns towards eyo .
For easy samples, where p∗ and eyo are in the same direction, these forces agree and lead to fast convergence. On samples like the “medium” point in Figure 3, the two forces broadly agree early on, but the learning path deviates slightly towards p∗ en route to eyo .

heading: 4 LEARNING PATHS ON REAL TASKS
text: Our analysis in Sections 2 and 3 demonstrate that the model can spontaneously refine the bad labels during one-hot training: the zig-zag learning path first moves towards the unknown p∗. But is this also true on real data, for more complex network architectures? We will now show that they do, although seeing the patterns requires a little more subtlety. We then propose an algorithm called Filter-KD to take further advantage of these observations for improved generalization.
To verify this explanation on real tasks – we use CIFAR10 and CIFAR100 as examples – the first conundrum is that we can no longer calculate the base difficulty ‖ey−p∗(x)‖2 of a sample, because p∗ is unknown.8 Furthermore, as the labels of CIFAR dataset are relatively clean, we cannot reliably find many samples with bad labels to see whether the pattern is representative. Recall that there are two cases for a label to be bad: ambiguous sample with flat p∗, and unambiguous samples with “wrong” labels. It might be more persuasive to see that the model can spontaneously “correct” those wrong labels by early stopping at the inflection of the zig-zagging path. Thus, we choose a random 1,000 of the 50,000 training samples, and flip their labels to a random y′ 6= y. With these changes, we visualize the learning path of data points while training a ResNet-18 (He et al., 2016) on CIFAR10 for 200 epochs as an example. The first panel of Figure 5 shows the learning path of an easy sample with good label. We can see that qt converges quickly towards the left corner, the one-hot distribution for the correct class, because the color of the scattered points in that figure are quite light. At the early stopping epoch, qt has already converged to ey . However, for a sample with a bad label, it is very difficult to observe any patterns from the raw path (blue points): there are points almost everywhere in this plot. This is likely caused by the more complex network and dataset, as well as high learning rate in early training. To find the hidden pattern in this high-variance path, we treat qt as a time series signal with many high-frequency components. Thus we can collect its low-frequency components via a low-pass filter and then draw it, i.e., taking a moving average of the qt (red points). This makes it clear that, overall, the pattern zig-zags, first moving towards the unknown true label before eventually turning to memorize the wrong label.
This smoothing method not only helps us observe the zig-zag pattern, but also improves the predictions. Figure 6 tracks how often the most likely prediction of the network, argmaxqt, recovers the true original label, rather than the provided random label, for the corrupted data points. At the beginning of the model’s training, the initialized model randomly guesses labels, getting about 10% of the 1,000 flipped labels correct. During training, the model spontaneously corrects some predictions, as the learning path first moves towards p∗. Eventually, though, the model memorizes all its training labels. Training with the predictions from the 400th epoch corresponds to standard KD (no
corrected points). Early stopping as in ESKD would choose a point with around 700 corrected labels. The smoothed path, though, performs best, with over 800 corrections.
8We do not believe the human annotations of the aforementioned CIFAR10H (Peterson et al., 2019) are a perfect estimate of p∗; see Appendix E.

heading: 5 SMOOTHING AS A METHOD FOR KNOWLEDGE DISTILLATION
text: Figure 6 gives a clear motivation for a new knowledge distillation method, which we call Filter-KD (Algorithm 1): train the student from the smoothed predictions of the teacher network. Specifically, we maintain a look-up table qsmooth ∈ RN×K to store a moving average of qt for each training sample. Note that in one epoch, each qsmooth(xn) will be updated only once. We check the early stopping criterion with the help of a validation set. Afterwards, the teaching supervision qsmooth is ready, and we can train a student network under its supervision. This corresponds to using a moving average of the teacher model “in function space,” i.e. averaging the outputs of the function over time.
Compared to ESKD, Filter-KD can avoid the extremely high variance of qt during training. Note that unlike Figure 5 which plots qt after every iteration of training, most practical implementations only consider early-stopping at the end of each epoch. This is equivalent to down-sampling the noisy learning path (as in Figure 5 (c)), further exacerbating the variance of qt. Thus ESKD will likely select a bad ptar for many data points. Filter-KD, by contrast, has much more stable predictions.
We will show the effectiveness of this algorithm shortly, but first we discuss its limitations. Compared to ESKD, the running time of Filter-KD might be slightly increased (but not too much), because we only do a few extra multiplications and summations for each batch. More importantly, compared to the teaching model in ESKD, the Filter-KD requires a teaching table qsmooth, which will require more memory when the dataset is large. One alternative avoiding the need for this table would be to instead take an average “in parameter space,” like e.g. the momentum parameters updating as in Ruppert (1988); Szegedy et al. (2015); He et al. (2020). We empirically find that, although this helps the model converge faster, it does not lead to a better teacher network; see Appendix H. Thus, although Filter-KD has clear drawbacks, we hope that our explanations here may lead to better practical algorithms for large datasets in the future.
Input: Dataset {(xn, yn)}Nn=1, where In = {1, 2, ..., N} is the index for each pair # Train the teacher Initialize a network model, initialize an N ×K matrix called qsmooth Go through the entire dataset, calculate qsmooth[n] = Softmax(model(xn)) for epoch = 1, 2, ..., T do
for n ∈ {1, 2, . . . , N} in random order do p̂ = Softmax(model(xn)) qsmooth[n] = (1− α) · qsmooth[n] + α · p̂ loss = CrossEntropy(p̂, yn) Backpropagate and update the parameters end for Check the early stopping criterion; stop training if satisfied
end for # Train the student For each input xn, set ptar = qsmooth[n]; train the network under the supervision of ptar
Algorithm 1: Filter-KD. α controls the cut-off frequency of low-pass filter (0.05 here).

heading: 5.1 QUANTITATIVE RESULTS OF FILTER-KD
text: We now compare the performance of Filter-KD and other baselines on a real dataset. We focus on self-distillation and a fixed temperature τ = 1 (except Table 2), as we want the only difference among these methods to be ptar. Thus we can conclude the improvement we achieved purely comes from the refined supervision. Appendix C for discussion and results related to the choice of temperature; τ = 1 is reasonably good for our current setting.
The first task we consider is noisy-label classification, where we train and validate the model on a dataset with some labels randomly flipped. After training, all the models are evaluated on a clean held-out test set. The experiments are conducted on CIFAR (Figure 7) and TinyImageNet (Table 1), under different noise ratios σ: σ = 0.1 means 10% of the labels are flipped. In Figure 7, an interesting trend can be observed: the enhancement brought by Filter-KD is not significant when σ is too small or too large. That is reasonable because for small σ, the few samples have bad labels,
Noise σ 0% 5% 10% 20% OHT 56.95 53.02 52.02 30.52
ESKD 58.61 53.53 52.99 31.55 FilterKD 59.32 56.43 55.51 40.81
Table 1: Results on TinyImageNet dataset.
thus the possible enhancement might not be large. When σ is too high, the labels of similar points become less reliable, and the learning path will no longer head as reliably towards p∗. Thus, for very high noise ratios, the performance of Filter-KD decays back towards that of OHT.
Filter-KD also outperforms other methods in both accuracy and calibration on the clean dataset, as illustrated in Table 3. This remains true in the more common KD case when the teacher network is bigger than the student; see results in Table 2. Note that in Table 2, we no longer keep τ = 1, as the baselines are more sensitive to τ when teacher is large (see Appendix C for more discussion); instead we optimize τ for each setting. Here “Effi” is short for EfficientNet-B1 (Tan & Le, 2019), “Res” is short for ResNet18, “Mobi” is short for MobileNetV2 (Sandler et al., 2018), “VGG” is short for VGG-11 (Simonyan & Zisserman, 2015).

heading: 6 RELATED WORK
text: Human label refinement. Peterson et al. (2019) use the distribution of annotations from multiple annotators to replace the one-hot label in training, and find that doing so enhances both the generalization performance of trained models and their robustness under adversarial attacks. However, this approach is relatively expensive, as it requires more annotation effort than typical dataset creation techniques; several experienced annotators are required to get a good estimate of p∗. It is also unclear in general how well the distribution of decisions by human annotators corresponds with the true variability under the data distribution. Appendix E has experiments based on this dataset.
Label smoothing. Another popular method is label smoothing (Szegedy et al., 2016), discussed in the previous sections. We believe that suitable smoothing can decrease ‖ptar − p∗‖2 to some extent, but the lower bound of this gap should be large, because label smoothing treats each class and each sample in the same way. Much prior research (e.g. Müller et al., 2019) shows that KD usually outperforms label smoothing as well.
KD and ESKD. Knowledge distillation, by contrast, can provide a different ptar for each training sample. KD was first proposed by Hinton et al. (2015) for model compression, under the assumption that it would help to have a more complex teacher extract more information from the dataset. Later, Furlanello et al. (2018); Cho & Hariharan (2019); Zhang et al. (2019); Yuan et al. (2020) found that distillation can bring enhancement even when teacher and student have identical structure. These surprising findings sparked studies of self-distillation; our work applies to both self-distillation and larger-teacher cases.
Another active direction in research about KD is finding explanations for its success, which is often still considered somewhat mysterious. Tang et al. (2020) decompose the “dark knowledge” provided by the teacher into three parts: uniform, intra-class and inter-class. This explanation also overlaps with ours: if we observe the samples with different difficulty in the same class, we are discussing intra-class knowledge; if we observe samples with two semantically similar classes, we then have inter-class knowledge. Among previous work, the most relevant study is that of Menon et al. (2021), who proposes a statistical explanation of KD. Our Hypothesis 1 is suggested by the main claim of that paper, which focuses on the question of what is a good ptar. Our work builds off of theirs by helping explain why this good ptar emerges, and how to find a better one. Besides, by looking deeper into the learning path of samples with different difficulty, our work might shed more light on KD, anti-noisy learning, supervised learning, overfitting, etc.
Noisy label task. Learning useful information from noisy labels is a long-standing task (Angluin & Laird, 1988; Natarajan et al., 2013). There are various ways to cope with it; for instance, Menon et al. (2019) use gradient clipping, Patrini et al. (2017) use loss correction, Huang et al. (2020) change the supervision during training, and Zhang et al. (2020) take advantage of extra information to achieve good performance. It is possible to combine KD-based methods with traditional ones to further enhance performance; our perspective of learning paths also provides further insight into why KD can help in this setting.

heading: 7 DISCUSSION
text: In this paper, we first claim that better supervision signal, i.e., ptar that is closer to p ∗, leads to better generalization performance; this is supported by results of Menon et al. (2021) and further empirical suggestions given here. Based on this hypothesis, we explain why LS and KD outperform one-hot training using experiments on a toy Gaussian dataset. To further understand how such better supervision emerges, we directly observe the behavior of samples with different difficulty by projecting them on a 2-D plane. The observed zig-zag pattern is well-explained by considering the tradeoff between two forces, one pushing the prediction to be near that of similar inputs, the other pushing the prediction towards its training label; we give an intuitive account based on Proposition 1 of why this leads to the observed zig-zag pattern.
To apply these findings on real tasks, in which the data and network are more complex, we conduct low-pass filter on the learning path, and propose Filter-KD to further enhance ptar. Experimental results on various settings (datasets, network structures, and label noise) not only show the advantage of the proposed method as a method for a teacher in knowledge distillation methods, but also help verify our analysis of how generalization and supervision work in training neural network classifiers.

heading: A MISCELLANEOUS BACKGROUND
text: Calculation of ECE: expected calibration error is a measurement about how well the predicted confidence represent the true correctness likelihood (Guo et al., 2017). For example, if our model gives 100 predictions, each with confidence, say, q(y = k|x) ∈ [0.7, 0.8]). Then we might expect there are 70 ∼ 80 correct predictions among those 100 ones. To calculate this, we first uniformly divide [0, 1] into M bins, with each bin represented by Im ∈ ( m−1 M , m M ] and Bm is all the x samples whose confidence falls into Im. Then, ECE is calculated as:
ECE = M∑ m=1 |Bm| n |acc(Bm)− conf(Bm)| , (5)
where acc(Bm) = 1|Bm| ∑ i∈Bm 1(ŷi = ki), conf(Bm) = 1 |Bm| ∑ i∈Bm q(y = ki|xi), ki is the true label and ŷi = argmax[q(y|xi)] is the model’s prediction. All the ECE mentioned in this paper is calculated by setting M = 10.
Barycentric coordinate system: when visualizing the learning path, it is problematic to directly choose two dimensions and draw them in the Cartesian coordinate system, as illustrated in the first panel in Figure 8. In geometry, a suitable way to project a 3-simplex vector onto a 2-D plane is converting it to a point in a barycentric coordinate system. Specifically, we have three basis vectors:
v0 = [0, 0]T, v1 = [1, 0] T, and v2 = [ 1 2 , √ 3 2 ]T , which are the corner and two edges of an equilateral triangle respectively. Then the 2D-coordinate is calculated as (x, y) = [v0; v1; v2] · q. So every points in the left corner of a Cartesian system plane can be converted to the triangle in a barycentric system, as illustrated in the last panel in Figure 8.

heading: B TOY GAUSSIAN DATASET
text: In Sections 2 and 3, we apply a toy Gaussian dataset to verify two facts derived from Hypothesis 1, so as the learning path of specific samples. Here we provide more details about this dataset.
Generate the dataset: here, we have N samples, each sample a 3-tuple (x, y,p∗). To get one sample, we first select the label y = k following an uniform distribution over all K classes. After that, we sample the input signal x|y=k ∼ N (µk, σ2I), where σ is the noisy level for all the samples.
µk is the mean vector for all the samples in class k. Each µk is a 30-dim vector, in which each dimension is randomly selected from {−δµ, 0, δµ}. Such a process is similar to selecting 30 different features for each class. Finally, we calculate the true Bayesian probability of this sample, i.e., p∗(y|x). Calculate the ground truth probability: we use the fact that p∗(y|x) ∝ p(x|y)p(y). As y follows an uniform distribution, we have p∗(y|x) = p(x|y=k)∑
j 6=k p(x|y=j) . Following p(x|y = k) ∼ N (µk, σ2I),
we find p∗(y|x) should have a Softmax form, i.e., p = e sk∑
j 6=k e sj . Specifically, we have:
p∗(y = k|x) = e sk∑
j 6=k e sj
; si = − 1
2σ2 ‖x− µi‖22. (6)
Setup of experiments in Figure 1: in this experiment, we generate a toy Gaussian dataset with K = 3, σ = 2 and N = 105. To reduce the variance of test error, we make a train/valid/test split with ratio [0.05 0.05, 0.9]. We train an MLP with 3 hidden layers, each with 128 hidden units and ReLU activations. We first conduct experiments on some baseline settings, i.e., learning from one-hot supervision (OHT for short), from smoothed label supervision (LS for short), from a converged teacher’s prediction (KD for short), and from an early-stopped teacher’s prediction (ESKD for short). In OHT case, an NN is trained under the supervision of ey . If we train this NN until the convergence of training accuracy, we obtain the ptar for the KD case. If we select the snapshot of that NN based on the best validation accuracy, we obtain the ptar for the ESKD case. If we directly set ptar = 0.9ey+0.1u, where u is an uniform distribution overK classes, we obtain the supervision for LS case. With different supervisions, we train a new network with identical structure until the validation accuracy no longer increase. Furthermore, to see a trend between generalization ability and ‖ptar − p∗‖2, we run the experiment 200 times under different noisy supervisions.

heading: C TEMPERATURE IN DIFFERENT KD METHODS
text: Temperature is an important hyper-parameter in different kinds of KD methods, which might influence the performance a lot. In this appendix, we will discuss why we prefer τ = 1, and why this is a reasonably good empirical choice in our settings.
The role played by τ :
In general, the loss function in KD has the form: L = β · ( 1
τ2
) · H(qτ ,pτtar) + (1− β) · H(q, ey), (7)
where β ∈ [0, 1] is another hyper-parameter to trade-off the importance between one-hot label and teacher’s predictions. Furthermore, for this loss, the gradient of L to logits z is:
∂L ∂zi = β ·
( 1
τ
) · (qτi − pτtari) + (1− β) · (qi − eyi). (8)
Why we use τ = 1:
The most important reason for us to choose τ = 1 (as well as β = 1), even with the risk of providing sub-optimal performance, is that we want to observe how much enhancement is brought by refining the label. In our mind, KD’s success comes from the following two aspects:
(a) better label (i.e., ptar is better than ey); (b) better learning dynamics (soften q to qτ make the training easier).
The first one provides the student with more useful knowledge, and the second one helps the student to extract it. The focus of our paper is the improvement in labels. When the temperature is not 1, both (a) and (b) are influenced, as we are using qτ (i.e., the smoothed student output) to match pτtar (i.e., the smoothed target) during training and using q to inference during testing. If we fix τ = 1, the only difference between Filter-KD, ESKD, KD, LS, and OHT training is ptar. Under such a condition, we believe the fact that Filter-KD/ESKD outperforms OHT (strictly better in each run) is enough to conclude KD methods can provide better labels. Furthermore, as Filter-KD is proposed to mitigate the high-variance issue in ESKD, and we can directly observe the zig-zag learning path of samples with bad labels, we believe it is reasonable to make the conclusion in the paper.
Furthermore, using τ 6= 1 doesn’t substantially change our analysis. As illustrated in Equation (8), the gradient shows that qτ moves towards pτtar in each update. Hence our analysis in proposition 1 still holds, which means the trade-off between the two forces still exists. But considering τ 6= 1 makes the explanation awkward because the prediction of the model is q, not qτ : readers might feel confused why we visualize q and ptar in the learning path but in fact, the gradient comes from q τ and pτtar. If we directly visualize q τ and pτtar, how could we read the value of q from the figure? So, to make the paper easier to follow, we prefer τ = 1.
τ = 1 is reasonably good in our settings:
In fact, the optimal τ depends on many settings, e.g., teacher’s size, optimizer, learning rate (scheduler), and etc: there is still no consensus on what is the best choice of τ for all settings. So for the experiments in the main context, we fix τ and fine-tune other hyper-parameters to achieve good results. However, to verify this choice of τ is not terrible, we performed a coarse grid search on both CIFAR and toy examples. As illustrated in Table 4 and Figure 11, choosing τ ∈ [1, 2] seems to be good choice: the performance doesn’t significantly decay until τ is too large.
0 5 10 15 20 0.70
0.75
0.80
0.85
0.90
Te st
a cc
ur ac
y
1 2 3 4 5
0.898 0.899 0.900 0.901 0.902 Te st a cc ur ac y
0 5 10 15 20 Temperature
0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 Te st a cc ur ac y
1 2 3 4 5 Temperature
0.898 0.899 0.900 0.901 0.902 0.903 Te st a cc ur ac y
Figure 12: First row: large teacher to small student; second row: self-distill. Left column: τ in a larger range; right column: τ in a small range.
Some readers might also curious about why our results don’t match the common notion that the optimal τ should be 4, which is first provided in Hinton et al. (2015) and followed by much later work (Gao et al., 2018; Zagoruyko & Komodakis, 2017; Cho & Hariharan, 2019, e.g.). Some works also use a grid search to conclude that a temperature as high as 20 should be the best choice (Yuan et al., 2020; Kim et al., 2019). We find this mismatch comes from the relative difference between the network size of teacher and student. In short, when distilling from a large teacher to a small student (as most of the cases in aforementioned works), high τ is preferred. When conducting self-distillation, τ need not be that high. For example, Zhang
et al. (2019) and Roth et al. (2021) claim τ = 1 is the best choice, while Zhang & Sabuncu (2020) claim τ ≈ 2 is the best. To further verify this, we compare the temperature trend between two cases on the toy dataset. In Figure 12, the first row is distilling a 10-layer, 256-width MLP to a 3-layer, 32-width MLP; the second row is self-distillation between 3-layer, 32-width MLPs.
At the same time, we also run a grid search the smoothing factor α of our Filter-KD in Table 5.

heading: D HOW REPRESENTATIVE IS THE ZIG-ZAG PATTERN?
text: In the main paper, we only visualize the zig-zag learning path of a few samples in Figures 3 and 5. The readers might then wonder whether this pattern is representative across the whole dataset. To verify this, we define a quantitative metric called zig-zag score. Specifically, we first calculate the integral of each dimension of the prediction:
Qi∈{1,...,K} , T∑ t=1 qti(xn). (9)
We then use the highest Qi among i ∈ {1, ...,K} \ {y}, where y is the label’s class, as the zig-zag score. In other words, we focus on the behavior of q on those dimensions that are not the training label. If this score is large, we might expect the neighbouring samples exert high influence on the path, and vice versa. Note that as we have ∑ iQi = constant for any sample (as q
t is a K-simplex), this zig-zag score might correlated with C-score of Jiang et al. (2021). However, their focuses are different: C-score is similar to −Qi=y and focuses more on how fast the prediction converge to the label; zig-zag score, on the other hand, is more about how much the path deviates towards a different class, possibly the label close to p∗.
With this score, we test the correlation between base difficulty and zig-zag score on toy datasets under different settings. From Figure 14, it is safe to conclude that samples with higher base difficulty will have a more zig-zagging path during training. We also compare the expected zig-zag score of the 1000 samples with flipped label in CIFAR10 (recall the experiment in Figure 6). In Figure 13, it is clear that the zig-zagness of these wrong label samples (which we are quite have high base difficulty) is significantly higher than the average.
Finally, we also randomly select some data samples in each class of CIFAR10 and visualize their learning paths. Figure 15 shows random samples for the noisy label experiment. It is clear that the learning path of easy samples with correct labels converge fast while that of samples with wrong labels is zig-zag. In Figure 16, which shows the learning path with high zig-zag score when the
training set is clean, we can still observe some samples with zig-zag path. These samples might be ambiguous, with a quite flat p∗. However, as we do not know the true p∗ of these samples, it is impossible to provide a result like Figure 13.

heading: E CIFAR10H
text: As previously mentioned, Peterson et al. (2019) propose a CIFAR10H dataset to refine the one-hot label by human annotators. However, we decided not to use this baseline in the main body of the paper, for the following reasons:
• CIFAR10H only annotates the test set of CIFAR10, making it hard to compare with other standard baselines; • We cannot control the difficulty of the task like we can for the toy dataset; • We believe the labels provided by human annotators is not a perfect estimate of p∗.
The first two reasons are straightforward. For the third: recall that from our definition, p∗ , p∗(y|x) is related to the generating process of the image. Although Peterson et al. (2019) show that training using this label can bring enhancement and hence make the prediction more robust, we still believe that human annotators do not correspond to the true labeling process in this case; labels from CIFAR10H are far better than one-hot, but not perfect. To verify this, we conduct the following two experiments.
First, we add noise to the labels of CIFAR10H and use them to train a student network. We then evaluate these models on the original CIFAR10 training set; Peterson et al. (2019) also use this strategy to evaluate. Results are provided in the first two panels of Figure 17, similar to Figure 9 for the toy dataset. For the first panel, when the added noise are high enough to flip some labels, we see a clear negative correlation between ‖ptar − pcifarh‖2 and test accuracy. However, when we add small amount of noise to pcifarh, as in the second panel in the figure, the test accuracy can increase substantially. This means pcifarh might be a little conservative when estimating p
∗: adding noise and hence making ptar smoother can enhance the generalization ability. In the toy dataset, where p
∗ is precisely the generating probability, learning from p∗ always performs the best.
Furthermore, we also compare the supervisions provided by Fitler-KD and pcifarh. As illustrated in the last panel in Figure 17, Filter-KD can achieve notably higher accuracy than training on CIFAR10H labels, which we believe is more evidence for our concern.

heading: F DISTANCE GAP UNDER DIFFERENT SUPERVISIONS
text: Figure 2 provides the distance gap between q and corresponding p∗ for each training sample in different training stages, under the supervision of one-hot label ey . From the results in this figure, we notice that the behavior of hard samples contributes more to the success of ESKD. Here we further visualize how these gaps changes when the model is trained under different types of supervision, i.e., ground truth, LS and ESKD.
In the first row, which is the result of label smoothing, we see the ‖ptar − p∗‖2 (i.e., red dots) has a ‘V’-shape. The vertex location depends on the smoothing parameter we choose. However, as discussed previously, label smoothing has only one parameter to control ptar, which is like using a linear model to fit a high-order function. So, although a proper smoothing value can bring better supervision than one-hot label, its upper bound might be limited. Regarding the training dynamics, we can see a similar trend as the results shown in the one-hot case, i.e., all the samples first move down and then converge to ptar. From the middle panel, we might expect label smoothing to outperform the one-hot case, because the scatters are closer to the x-axis, which represents the ground truth p∗.
The second and the third rows demonstrate the KD and ESKD case. Results in the KD case are quite similar to the one-hot case in Figure 2, because the converged ptar is close to ey . However, we can still expect KD to outperform one-hot training, because the ptar is closer to p
∗ than ey is. The last panel in this row also demonstrates that ‖q − p∗‖2 might be smaller than ‖ptar − p∗‖2, which can be considered as an explanation for why iterated self-distillation, e.g., Born Again Networks (Furlanello et al., 2018), can improve the performance. For the ESKD case, we see the overfitting almost disappear: the distribution of the blue points do not change too much after the early stopping criterion is satisfied.
In the last row, which illustrates the training under the supervision of p∗, we find all the blue points move toward the x-axis, i.e., their ptar = p
∗, and finally converge to it. There is also no overfitting in this case. From the last two panels, we see the disperse of blue points is significantly smaller than all other settings, which means the network’s prediction is quite close to p∗. Hence the performance of this case is the best.

heading: G MORE ABOUT THE DECOMPOSITION AND NTK MODEL
text: Proof of Proposition 1: to observe how qt(xo) changes when the network’s parameters update after observing xu, we can directly write down ∆t(xo) = qt+1(xo)− qt(xo) and simplify it. Recall z(x) = f(w, x) is the vector of output logits, and q = Softmax(z) is the output probability); the derivation is explained line-by-line afterwards.
∆t(xo) = ∇wqt(xo)|wt · ( wt+1 −wt ) +O ( ‖wt+1 −wt‖2 ) (10)
= ( ∇zqt(xo)|wt · ∇wz(xo)|wt ) · ( wt+1 −wt ) +O(η2‖∇wL‖2) (11)
= ( ∇zqt(xo)|wt · ∇wz(xo)|wt ) · (−η∇wL(xu)|wt)T +O(η2) (12)
= ( ∇zqt(xo)|wt · ∇wz(xo)|wt ) · (−η∇zL(xu)|wt · ∇wz(xu)|wt)T +O(η2) (13)
= −η∇zqt(xo)|wt · [ ∇wz(xo)|wt · (∇wz(xu)|wt)T ] · (∇zL(xu)|wt)T +O(η2) (14)
= η · At(xo) · Kt(xo, xu) · ( ptar(xu)− qt(xu) ) +O(η2). (15)
For the right-hand term, we used ‖∇wL‖2 ≤ ‖∇zL‖2 · ‖∇wz‖2; note that ‖∇zL‖ = ‖ptar − qt‖ is bounded because they are probabilities. The term ‖∇wz‖ is also bounded if we use a bounded loss, such as gradient-clipped softmax, and will in general stay bounded as long as the optimization doesn’t diverge. Thus the last term isO(η2), which can be ignored when the learning rate η is small. The first line is a Taylor expansion, where ∆t(xo) is a K×1 vector,∇wqt(xo)|wt is a K×dmatrix, and w is a d × 1 vector (d is the number of parameters). The second line follows the chain rule, where∇zqt(xo)|wt is a K ×K matrix and∇wz(xo)|wt is a K × d matrix. The third line applies the definition of SGD, where∇wL(xu)|wt is a 1× d vector. The fourth line also follows the chain rule, where∇zL(xu)|wt is a 1×K vector and∇wz(xu)|wt is a K × d matrix. The fifth and the sixth lines are rearrangements of different terms, where bothAt andKt are K×K matrices. In this expression,
At(xo) =  q1(1− q1) −q1q2 · · · −q1qK −q2q1 q2(1− q2) · · · −q2qK
... ...
. . . ...
−qKq1 −qKq2 · · · qK(1− qK)  , (16) which is a symmetric positive semi-definite (PSD) matrix9 with trace tr(At(xo)) = 1 − ∑K i=1 q 2 i .
As we have ∑ i qi = 1, it is easy to check the trace of this matrix is larger at the beginning of training (when q tend to be flat) than that at the end of training (q tend to be peaky), as illustrated by most panels in Figure 19. Given that the trace of a matrix is the sum of its eigenvalues andAt(xo) is PSD, smaller tr(At(xo)) = 1 − ∑K i=1 q 2 i means this matrix will tend to shrink the vector it multiplies more. Hence the step size of ∆t(xo) tend to decrease when qt becomes more peaky.
9The matrixA can be observed to be the covariance matrix of a categorical distribution with item probabilities q, and hence PSD.
The second term in that expression, Kt(xo, xu), is the inner product of gradients at xo and xu. Intuitively, if their gradients have similar directions, this matrix is large, and vice versa. This matrix is known as the empirical neural tangent kernel, and it can change through the course of training as the network’s notion of “similarity” evolves. For appropriately initialized very wide networks trained with very small learning rates, however, Kt remains almost constant during the course of training, and is almost independent of the initialization of the network parameters; the kernel it converges to is known as the neural tangent kernel (Jacot et al., 2018; Arora et al., 2019).
Learning Dynamics of q(xo): in proposition 1, we decompose ∆t(xo) into three parts and analyze what the learning path of a training sample might be like in Section 3.3. Here we will provide more detailed illustration of the two groups of force imposed on q(xo).
Specifically, ∆t(xo) will be influenced by two variables, i.e., updating sample xu and time t. We discuss their effects separately. For xu, only the last two terms, i.e.,Kt(xo, xu) and (ptar(xu)− qt(xu)) depends on it.
In Section 3.3, we claim that if xo and xu are similar, the norm of K0(xo, xu) might be large, and vice versa. Here we empirically verify this using a toy Gaussian dataset, as illustrated in Figure 19. The figure shows how the similarity between xo and xu correlates with tr(K0(xo, xu)). Each panel represents one xo, which is claimed in the title of the subfigures. The x-axis represents the rank of the cosine similarity between observed xo and all N training samples (including itself). The y-axis is the trace of K0(xo, xu). The color of each scatter point is the true label of xu. From the figure, we can observe a clear decreasing trend, which means smaller similarity leads to larger tr(K0(xo, xu)). Additionally, the term (ptar(xu)− qt(xu)) provides a direction that qt(xo) should move towards. We claim in Section 3.3 that at any point in the input space, the labels of these input samples might follow the ground truth distribution, i.e., p∗(y|x). Hence most of the neighbouring x might pull q(xo) towards its ground truth p∗(y|xo). We will discuss the norm of this term when discussing the influence of t. In short, at any time, the neighboring xu will impose stronger force on xo and the direction of the force roughly points to the ground truth p∗(y|xo). Another factor to consider is t, which influences ∆t(xo) through the term At(xo) and term (ptar(xu)− qt(xu)). However, these two terms do not have an analytical expression w.r.t t, as qt is quite complicated. So we can only provide some intuitions through the behavior of qt and then experimentally verify them. In Figure 19, we show how tr(At(xo)) depends on qt: flat qt leads to larger trace. A similar trend also exists in the norm of (ptar(xu)− qt(xu)). As the initialized q tend to be flat, updates of any samples will influence network’s parameters a lot. When the training progress, those easy samples converge fast, so their ‖ptar(xu)− qt(xu)‖2 and tr(At) become small. However, as the qt for the hard sample is still far away from its ey , the large ‖ptar(xu) − qt(xu)‖2 and tr(At) will finally drag qt back towards the one-hot distribution, as illustrated in Figure 3.

heading: H LOW PASS FILTER ON NETWORK PARAMETERS
text: In Section 4, we point out the high variance issue of the traditional KD methods after observing the learning path of those hard samples. We then propose a Filter-KD algorithm to smooth the output of each training sample during training. Such a low pass filtering method is quite similar to the momentum mechanism used in self-supervised learning, e.g., from the classic method of Ruppert (1988) to MOCO (He et al., 2020), which conduct low pass filtering on each parameter of the network. As mentioned before, the proposed Filter-KD algorithm might require more memory when the dataset becomes larger, because qsmooth would record every training sample’s prediction.
Conducting low pass filtering on network parameters might be a good way to solve this. To verify whether this method works, we train a ResNet18 on CIFAR100, using one-hot supervision. At the beginning of training, we copy the training model and call it tracking model. At the end of each update (i.e., each batch), the parameters of training model is updated as usual while the tracking model’s parameters are updated with momentum. Specifically, for the training model, wt+1train = wttrain + η∇L, for the tracking model, w t+1 track = (1 − α)wttrack + αw t+1 train . We train the model for 150 epochs, and show the learning curve of test accuracy in the left panel of Figure 20. We see the performance of the tracking model converges faster than the training model, which is reasonable because filtering parameters is regularizing the training process. However, the converging accuracy of this two models are the same. At the same time, we find this tracking model is not as good as qsmooth when teaching a new model, as illustrated in the right panel of Figure 20.
As this paper mainly focuses on the behavior of the model’s prediction, the discussion and experiments of filtering in parameter space is limited. However, as many papers demonstrates the effectiveness of this momentum mechanism, we think it is important to explore the relationship further.

Review summaries:
This paper proposes an explanation for the success of distillation. It first experiments with synthetic Gaussian data. On synthetic data, it shows that distillation works better from an early-stopped model. It probes why, and finds that, when the one-hot label is far from the true conditional distribution, the early-stopped model’s distribution tends to be closer to the true distribution than either the one-hot label or the converged model. The early-stopped model is better because, over the course of training, the model tends to produce a distribution close to the true one before finally overfitting the one-hot label. Some examples show similar patterns for real-world networks/datasets, provided the predictions over training are smoothed. Motivated by these results, the paper proposes to use an averaged distribution of labels from different steps of training rather than the distribution at the end of training, and shows that this improves performance by a little bit.

This paper introduces a method for doing knowledge distillation with noisy labels. The contribution consists on representing distilled labels as a weighted moving average of the predicted labels from a teacher as it trains (using noisy one-hot encoded labels). Finally, a student is trained using the distilled labels.