Better Supervisory Signals by Observing Learning Paths
Authors: Yi Ren
Shangmin Guo
Danica J. Sutherland
Venue: ICLR 2022 Poster (2022)
Keywords: Classification
Supervision
Knowledge Distillation

Abstract: Better-supervised models might have better performance. In this paper, we first clarify what makes for good supervision for a classification problem, and then explain two existing label refining methods, label smoothing and knowledge distillation, in terms of our proposed criterion. To further answer why and how better supervision emerges, we observe the learning path, i.e., the trajectory of the model's predictions during training, for each training sample. We find that the model can spontaneously refine "bad" labels through a "zig-zag" learning path, which occurs on both toy and real datasets. Observing the learning path not only provides a new perspective for understanding knowledge distillation, overfitting, and learning dynamics, but also reveals that the supervisory signal of a teacher network can be very unstable near the best points in training on real tasks. Inspired by this, we propose a new knowledge distillation scheme, Filter-KD, which improves downstream classification performance in various settings.

Published as a conference paper at ICLR 2022
BETTER SUPERVISORY SIGNALS BY OBSERVING LEARNING PATHS
Yi Ren
Shangmin Guo University of Edinburgh guoged .ckuk
Danica J. Sutherland UBC and Amii dsuth@cs cs .bc. ca
UBC
renyi jasisjjsshu@seem com
ABSTRACT
Better-supervised models might have better performance. In this paper, we first clarify what makes for good supervision for classification problem, and then explain two existing label refining methods, label smoothing and knowledge dis- tillation, in terms of our proposed criterion. To further answer why and how bet- ter supervision emerges, we observe the learning path, i.e., the trajectory of the model's predictions during training, for each training sample. We find that the model can spontaneously refine "bad" labels through a "zig-zag" learning path, which occurs on both toy and real datasets. Observing the learning path not only provides a new perspective for understanding knowledge distillation, overfitting, and learning dynamics, but also reveals that the supervisory signal of a teacher network can be very unstable near the best points in training on real tasks. In- spired by this, we propose a new knowledge distillation scheme, Filter-KD, which improves downstream classification performance in various settings.
INTRODUCTION
In multi-class classification problems, we usually supervise our model with "one-hot" labels: label vectors y which have y; 1 for one i, and 0 for all other dimensions. Over time, however, it has gradually become clear that this "default" setup not always the best choice in practice, in that other schemes yield better performance on held-out test sets, One such alternative is to summarize distribution of human annotations, as Peterson et al. (2019) did for CIFARIO. An alternative approach is label smoothing (e.g. Szegedy al., 2016), mixing between one-hot label and the uniform distribution. Knowledge distillation (KD), first training a teacher network on the training set and then student network on the teacher's 's output probabilities, was originally proposed for model compression (Hinton et al., 2015) but can also be thought of as refining the supervision signal: it provides "soft" teacher outputs rather than hard labels to the student.
Knowledge distillation is promising because it requires no additional annotation effort, but unlike label smoothing can still provide sample-specific refinement. Perhaps surprisingly. knowledge distillation can improve student performance even when the teacher is of exactly the same form as the student and trained on the same data; this is known as self distillation (Furlanello et al., 2018; Zhang et al., 2019). There have been many recent attempts to explain knowledge distillation and specifically self-distillation (e-g. Menon al., 2021; Allen-Zhu Li, 2020; Tang et al., 2020), from both optimization and supervision perspectives. We focus on the latter area, where it is usually claimed that the teacher provides useful "dark knowledge" to the student through its labels.
Inspired by this line of work, we further explore why and how this improved supervisory signal emerges during the teacher's is one-hot training. Specifically, we first clarify that given any input sample good supervision signals should be close (in Lg distance) to the ground truth categorical distribution, i.e., p" (y x). We then show that a neural network (NN) can automatically refine "bad labels", those where p" (y) x) is far from the training set's 's one-hot vector.' During one-hot training, the model prediction on such a sample first moves towards p" (y x), and then slowly converges to its supervisory label, following "zig-zag" pattern. A well-trained teacher, one that does not overfit
'This might he because is ambiguous (perhaps py x) is flat, simply got sample from less- likely class), or because the one-hot label has been corrupted through label noise or otherwise is "wrong."

---

Published as a conference paper at ICLR 2022
to particular training labels, can thus provide supervisory signals closer to p" x). We justify analytically (Section 3.3) that this pattern common in gradient descent training. Our explanations cause to recognize that this signal can be better identified by taking moving average of the teacher's prediction, an algorithm we term Filter-KD) KD. This approach yields better supervision and hence better downstream performance, especially when there are many bad labels.
After completing this work, we became aware of an earlier paper (Liu et al., 2020) studying almost the same problem in a similar way. We discuss differences between the papers throughout.
SUPERVISION INFLUENCES GENERALIZATION
We begin by clarifying how the choice of supervisory signal affects the learned model.
2.1 CHOICES OF SUPERVISION SIGNAL
In K K-way classification, our goal is to learn mapping f X + A that can minimize the risk
L(y, [())] = f p(x) p(y|x) M(y, ff (x)) dx dy,
R(F)
A
E (x.y)~
Here the label y an integer ranging from I K, and gives predictions in the probability simplex A (a nonnegative vector of length K summing to one); L(y, f(x)) is the loss function, e.g. cross- entropy or square loss. The input signal x usually high dimensional, e.g., an image or a sequence of word embeddings. The joint distribution of (x, y) is P. whose density" can be written as p(x)p(y x)). In practice, as P is unknown, we instead (approximately) minimize the empirical risk
K Σ kk)=k) L[k,
Remp (J.D) A
Σ
J ((x))
Σ e L(f(x)))
where (v,, = k)) an indicator function which equals yh or 0 otherwise, and ey. {0, 1}" is its one-hot vector form. L(f(x,)) (I(1, (x,)) L(K, f(x,))) RRK the loss for each possible label. Remps the N training pairs D - ((x,: Wn)) are sampled i.i.d. from P.
Comparing (2) to (1), we can see p(x) is approximated by an uniform distribution over the samples, which is reasonable. However, using an indicator function (i.e., one-hot distribution) to approximate p(y x) bears more consideration. For example, if a data point x is quite vague and its true p(y)x ()))) flat or multimodal, we might hope to see x multiple times with different label during training. But actually, most datasets have only one copy of each x, Although Remp is an unbiased estimator for Rt if we used a better (e.g. lower-variance) estimate of ply x), we could get better estimate for R and thus, hopefully, better generalization.
only ever see corresponding ey-
Specifically, suppose we were provided a "target" distribution Par(y x) (written vector form as Punc (x)) for each training point I, as D' = {(xn> Pear (x (x) ) Then we could use
Rem (f, D')
N Pez (u le xx. L(k, f(xx)) = Σ Peer (xm) ((f(x,)).
(3)
Standard training with Remp a special case of Rarr using Pue (x,) e.... The CIFARIOH dataset (Peterson et al., 2019) is one attempt a different Pear using multiple human annotators to estimate Par Label smoothing (e.g. Szegedy et al., 2016) sets Pia to a convex combination of e, and the constant vector * 1. In knowledge distillation (KD; Hinton et al., 2015), a teacher is first trained on D, then student learns from 2'' with Pa.. based on the teacher's outputs. All three approaches yield improvements over standard training with Remp-
Because we are working with classification problems, we use densities with respect to a product of some arbitrary measure on x (probably I Lebesgue) with counting measure y, and assume that these densities exist for notational convenience. None of our arguments will depend on the choice of base measure.
2

---

Published as a conference paper at ICLR 2022
OIT
Nowe
ofp and
(a) ISNE of toy dataset.
(b) ACC Vs |
p 112
(c) ECE V.S. Par p lla
Figure 1: Experiments on a toy dataset when learning from different Per In (b-c), the horizontal axis represents PPur p2112. and the vertical axis is the generalization performance. OHT means one-hot training (on em LS means label smoothing, GT means ground truth training with p* KD is knowledge distillation, and ESKD is early-stopped KD. The Spearman correlation coefficient for results in (b) -0.930 with p-value 1.9 10-53; 53, for (c) is 0.895 with p-value 2.7 x 10-13.
2.2 MEASURING THE QUALITY OF SUPERVISION
Choosing different Per- then, can lead better final model. Can characterize which Peur will do well? We propose the following, as general trend.
Hypothesis 1. Suppose we train a model supervised by Per that is, we minimize R D')). Then, smaller average L2 distance between Pur and the ground truth p" on these samples, i.e. small Ex [||P (x) p" (x) (2), will in general lead to betler generalization performance.
This hypothesis is suggested by Proposition 3 of Menon et al. (2021), which shows (tracking con- stants omitted in their proof) that for any predictor f and loss bounded L(y. Ú) S f,
1 Var [L]L(((x))] + + Pr. p" px(x)ll2) N
E D
(4)
When N is large, the second term will dominate the right-hand side, implying smaller average Puur plll will lead to Rar being a better approximation of the true risk R; minimizing it should then lead to a better learned model. This suggests that the quality of the supervision signal can be roughly measured by its L2 distance to the ground truth p'. Appendix B slightly generalizes the result of Menon et al. with bounds based on total variation (L1) and KI. divergences; we focus on the T2 version here for simplicity.
To further support this hypothesis, we conduct experiments on a synthetic Gaussian problem (Fig- ure (a); details in Appendix C), where we can easily calculate p" x) for each sample. We first generate several different Par by adding noise² to the ground truth p", then train simple 3-layer NNs under that supervision. We also show five baselines: one-hot training (OHT), label smoothing (LS), KD, carly-stopped KD (ESKD), and ground truth (GT) supervision (using p*). KD refers to teacher trained convergence, while ESKD uses a teacher stopped early based on validation accu- racy. We early-stop the student's 's training in all settings. From Figure/ 1 (b-c), it is clear that smaller || p" l| leads better generalization performance, as measured either by accuracy (ACC) or expected calibration error (ECE)* on held-out test set. Appendix C has more detailed results.
INSIGHTS FROM THE LEARNING PATH
In the toy example of Section 2, we see that ESKD outperforms other baselines in accuracy by substantial margin (and all baselines are roughly tied in ECE). We expect that supervision with smaller | l21 leads better generalization performance, but not clear how better Ptr emerges from when the teacher in ESKD is trained using one-hot labels. This section will answer this, by observing the learning paths of training samples.
WWhnnever mention adding noise to p", mean add independent noise each dimension, and then re-normalize it distribution. Large noise can thus flip the "correct" label. *ECE measures the calibration of model (Guo et al., 2017). Briefly, lower ECE means the model's 's confi- dence in its predictions is more accurate. See Appendix for details.
3

---

Published as a conference paper at ICLR 2022
tx30lo
He000
[[010]
35
04
2.0
100
1208
04
24 08
130 03
20
0.8
200
11c,00-mp p 0olla
10,000 P bkills
Figure 2: Normalized (divided by v2) distance between output distribution q and p" during the one-hot training in different stages (left to right: initial, early stop, convergence). In these figures, Ex |/(x)) - p (x)ll2 of Hypothesis the mean height of all points in the figure. We provide more results about the NNs trained under different supervisions using this fashion in Appendix F.
3.1 PAY MORE ATTENTION TO HARDER SAMPLES
For a finer bher-grained understanding of early stopping the teacher, we would like to better under- stand how the teacher's predictions evolve in training. Assuming Hypothesis 1, the main factor is [x]]q(x) p (x)] where is the teacher's output probability distribution. We expect, though, that this term will vary for different x, part because some samples are simply more difficult to learn. As proxy for this, we define base difficulty as lle, p" (x) [|2, which is large if:
x is ambiguous: p" has several large components, so there is no one-hot label near p". x is not very ambiguous (there is one-hot label near p*), but the sample was "unlucky" and drew from a www-pobbbblity class.
Figure 2 shows these two quantities three points in training: initialization, the point where ESKD's teacher stops, and convergence. At initialization, most points have large [qq(x p" (x)]l2- By the point of early stopping, most (x) values roughly near p" At convergence, however, q(x) x ey, as the classifier has nearly memorized the training inputs, leading to diagonal line in the plot.
It is clear the biggest contributors to [x||q(x) p" px(x)l2 when training model convergence the points with high base difficulty. Per Hypothesis 1, these are the points we should most focus on.
3.2 LEARNING PATH OF DIFFERENT SAMPLES
To better understand how changes for each sample, we track the model's outputs on all training samples at each training step. Figure shows four samples with different base difficulty, with the vectors of three probabilities plotted as points the simplex (details in Appendix A).
The two easy samples very quickly move to the correct location near e, (as indicated by the light color until reaching the corner). The medium sample takes less direct route, drifting off slightly to- wards p", but still directly approaches ey. The hard sample, however, does something very different: it first approaches p", but then veers off towards ey, giving "zig-zag" path. In both the medium and hard cases, there seems to be some "unknown force" dragging the learning path towards p" in both cases, the early stopping point is not quite perfect, but is noticeably closer to p" than the final converged point near ey- In other words, during one-hot training, the NNs can spontaneously refine the "bad labels." " Under Hypothesis this partly explains the superior performance of ESKD to KD with a converged teacher."
These four points are quite representative of all samples under different toy-dataset settings. In Appendix E, we also define a "zig sig-zaggess score" to numerically summarize the learning path shape, and show is closely correlated to base difficulty.
'The curve structure is expected: points with p' (1) 1. )) range, and all points are initialized with fairly ambiguous predictions q.
the middle of the base difficulty
'KD's practical success is also related to the temperature and optimization effects, among others; hetter supervision not the whole story. We discuss the effect of various hyperparameters Appendix D.
4

---

Published as a conference paper at ICLR 2022
crigin Path GRT Label
ge Label
Start
Earty Stosped
base_dii diff 0.0045
base_dit diff 0.1270
base_dif diff 0.6101
base_di diff 0.9425
Figure 3:
Learning path of samples with different base difficulty. Corners correspond to one-hot vectors. Colors represent training time: transparent at initialization, dark blue at the end of training.
3.3 EXPLANATION OF PATTERNS THE LEARNING PATH
The following decomposition will help explain the "unknown force" pushing towards p".
Proposition 1. Let (x) f(w', x) denote the network output logits with parameters w', and (x) Softmax (z' (x)) the probabilities. Let w a w' V (Pior(X.) (x)) LL('' (x.))) be the result of applying Then the change in network predictions for particular sample xx,
step of SGD w using the data point (Xu; Piar (xu)) with learning rate n.
q (+1 -q" (xxx) (7 Kc (xxxxx xu) (Ptar(x)) q" (xxx) O
where A' Axxx)) Vzq' (xo) and (C' X.) (Vuz(xo)lw wrr) ( (Vwz(x))lw)) are K x K matrices.
The matrix K is the empirical neural langent kernel, which we can think of roughly as a notion of "similarity" between x, and xu based on the network's representation time t, which can change during training in potentially complex ways. In very wide networks, though, X is nearly invariant throughout training and nearly independent of the initialization (Jacot et al., 2018; Arora et al., 2019). The gradient norm can be controlled by gradient clipping, or bounded with standard SGD analyses when optimization doesn't 'tt diverge. Appendix G has more details and the proof.
Figure shows the learning path a hard sample during one-hot training, say x., where the label ey, is far from p" (x.). In each epoch, will receive N updates based the loss of x, A x, (the small blue path), and one update based on x, (the big red path). At any time I, "dissimilar" samples will have small ('(Xo. x,) (as measured, c.g., by its trace), and hence only slightly affect the predicted label for x Similar samples will have large IC'(x., x,). and hence affect its updates reasonable to expect that the
much more; because p" hopefully similar for similar x values, mean of ey for data points with similar will be close to p" (x.). Thus the net effect of updates for x, F x, should be drag (xx)) towards p" (x,). This is the "unknown force" we observed earlier.
The other force affecting (xx)) during training is, of course, the update based on x., driving q(x,) towards e How do these two forces balance over the course training? Early on, lle q'((x)l is relatively large for nearly any x,, because near initialization q' px(.) will be relatively flat. Hence the size of the updates for "similar" x,, and the update for x,, should be comparable, meaning that, if there least a few "similar" training points, (x(x,) will move towards p" (x.). Throughout train- ing, some of these similar samples will become well-classified, so that |le, q'(x.))l be- comes small, and their updates will no longer exert much force on (x.)) Thus, the x, up- dates begin to dominate, causing the zig-zag pattern as the learning path turns towards e For easy samples, where p" and e, are in the same direction, these forees agree and lead fast convergence. On samples like the "medium" point in Figure 3, the two forces broadly agree early on, but the learning path deviates slightly towards p* route e
epoch start
epoch
Xo update start
Xo update end
epoch 1:
Other Xu update
epoch 3:
at
epoch 10:
epoch 90:
Figure Updates of (xx.) over training.
5

---

Published as a conference paper at ICLR 2022
Other:
Sensstred Peth
Other
Osgin Pyn
Other
Ongin
Sput turly Susped
Suatt Caryy Stopped
Dary Shopped
(corr)
Top2
(corr)
Top2 (corr) plane ship
Top2 plane
horse
dog
ship
Figure 5: Learning path on CIFARIO. In the first two panels we record q' for each batch, while in the last panel we record it for each epoch. See Appendix E for the learning paths of more samples.
Liu et al. (2020) prove that a similar pattern occurs while training a particular linear model on data with some mislabeled points, and specifically that stopping training at an appropriate early point will give better predictions than continuing training.
4 LEARNING PATHS ON REAL TASKS
Our analysis in Sections 2 and 3 demonstrate that the model can spontaneously refine the bad labels during one-hot training: the zig-zag learning path first moves towards the unknown p". But is this also true real data, for more complex network architectures? We will now show that they do, although seeing the patterns requires a little more subtlety.
We visualize the learning path of data points while training ResNet18 (He et al., 2016) CIFARIO for 200 epochs as an example. The first panel of Figure 5 shows the learning path of an easy sample.' We can see that q' converges quickly towards the left corner, the one-hot distribution for the correct class, because the color of the scattered points in that figure are quite light. At the early stopping epoch, q' has already converged to ey- However, for hard sample, it is very difficult to observe any patterns from the raw path (blue points): there are points almost everywhere in this plot. This likely caused by the more complex network and dataset, as well as high learning rate in early training. To find the hidden pattern in this high-variance path, we treat q² as time series signal with many giguffreyeency components. Thus we can collect its low-frequency components via a low- pass filter and then draw that, i.e., taking exponential moving average (EMA) on q' (red points). This makes it clear that, overall, the pattern zig-zags, first moving towards the unknown true label before eventually turning to memorize the wrong label.
This filtering method not only helps us observe the zig-zag pattern, but can also refine the labels. We use the following two experiments to verify this. First, we train the model on CIFARION 1OH using one-hot labels. As CIFAR provides Paum. which can be considered as an approximation of p". track the mean distance between q and Prum during training. In the first panel of Figure 6, which averages the distance of all 10k training samples, the difference between the blue qua curve and red 4m curve is quite small. However, we can still observe that the red curve is lower than the blue one before overfitting: filtering can indeed refine the labels. This trend is more significant when considering the most difficulty samples, as in the second panel: the gap between curves larger.
To further verify the label refinement ability of filtering we randomly choose 1,000 of the 50,000 CIFARIO 10 training samples, and flip their labels to different y' + y. The last panel in Figure tracks how often the most likely prediction of the network, argmax q', recovers the true original label, rather than the provided random label, for the corrupted data points. At the beginning of the model's training, the initialized model randomly guesses labels, getting about 10% of the 1,000 flipped labels correct. During training, the model spontaneously corrects some predictions, as the learning path first moves towards p". Eventually, though, the model memorizes all its training labels. Training with the predictions from the 400th epoch corresponds to standard KD (no corrected points). Early stopping as in ESKD would choose a point with around 70% of labels corrected. The filtered path, though, performs best, with over 80% corrected.
Witthout knowing p", we instead use zig-7ag score see Appendix measure the difficulty.

---

Published as a conference paper at ICLR 2022
All 10K samples in CIFARIOH
Most ambiguous 1k samples CIFAR1OH
1K wrong labeled samples CIFARTO
Normalized Klla Prod
0.8
1.0
Our
Our Piem
1.0
0.8
The to Phum
the to
600
800
700
Random KINDHT
Original
0.6-
Ou Pluxe
0.6
See to Pran
500
O%:
dex C,
400
ESKD FiterDD
0.4
300
0.2
1
100
200
0.0
0.0
100 125 150 175 200 Epochs
25
50
75 100 125 150 175 200
50 100 150 200 250 300 350 400 EDochs
Epochs
Figure 6: Filtering can refine the labels in both clean and noisy label case.
CIFARIO
CIFAR160
CITARL0G
0.8
CHT
0.8
0.7
Test Restarccncy
0.6
Filer
Tes 3ccurscy
0.6
Fitter
0.7
0.5
0.5
0.8
0.4
0.5
0.3
0.3
ESKD
0.4
Fitter
0.1
0.5
0.4
Molse ratio
Figure 7: Test accuracy under different noise ratio σ. Solid lines are the means while shade region are the standard errors for 3 runs with different random seeds (shaded range the standard error). Last panel compare the influence of different temperatures. Each thin rectangle plot represents a different o = {0, 0,1,011.... ..,.,........ 0.8). in which we plot the results with different T = (0.5, 1, 2, 4, 10}.
FILTERING AS A METHOD FOR KNOWLEDGE DISTILLATION
Figure gives a clear motivation for a new knowledge distillation method, which we call Filter-Kr KD (Algorithm I): train the student from the smoothed predictions of the teacher network. Specifically, we maintain a look-up table qamooth E RN to store a moving average of q, for each training sample. Note that in one epoch, each 4smooth (x..) will be updated only once. We check the early stopping criterion with the help of a validation set. Afterwards, the teaching supervision 4smooth is ready, and we can train student network under its supervision. This corresponds using a moving average of the teacher model "in function space," i.e. averaging the outputs of the function over time.
Compared to ESKD, Filter-KD can avoid the extremely high variance of q' during training. Unlike Figure 5, which plots q" after every iteration of training, most practical implementations only con- sider early-stopping at the end of each epoch. This is equivalent to wwwnlagmiing the noisy learning path (as in the last panel of Figure 5), further exacerbating the variance of q'. Thus ESKD will likely select a bad Perr for many data points. Filter-KD, by contrast, has much more stable predictions.
We will show the effectiveness of this algorithm shortly, but first we discuss its limitations. Com- pared to ESKD, the running time Filter-KD might be slightly increased. Furthermore, compared to the teaching model in ESKD, the Filter-KD requires teaching table 4moooth which will require substantial memory when the dataset is large. One alternative avoiding the need for this table would be to instead take an average "in parameter space," like e.g. momentum parameter updating in Szegedy et al. (2015). We empirically find that, although this helps the model converge faster, it does not lead to better teacher network; Appendix I. Thus, although Filter-KD has clear drawbacks, we hope that our explanations here may lead to better practical algorithms in the future.
Similarly inspired by this spontaneous label refining mechanism (or early stopping regularization), Liu et al. (2020) and Huang et al. (2020) each propose algorithms aiming at the noisy label problem. We discuss the relationship between these three methods Appendix H.
5.1 QUANTITATIVE RESULTS OF FILTER-KI KD
We now compare the performance of Filter- KD and other baselines on a real dataset. We focus on self-distillati and fixed temperature т 1 (except Table 2 and last panel in Figure 7), as
7

---

Published as a conference paper at ICLR 2022
Eff->Res Res->Mob
Res-vVGG
Noise o OHT
0%
5%
10%
20%
Teacher
86.83
77.23
77.98
56.95
53.02
52.02
30.52
Student
78.09
72.58
71.04
ESKD
58.61
53.53
52.99
36.55
ESKD
81.16
73.13
72.64
FilterKD
59.32
56.43
55.51
40.81
FilterKD
83.03
75.79
74.49
Table 1: Results on TinylmageNet dataset.
Table 2: Teacher->student, on CIFAR100.
want the only difference among these methods be Pur- Thus conclude the improvement we achieved comes purely from the refined supervision. See Appendix for other temperatures.
Accuracy ESKD
ECE
OHT
KD
FilterKD
OHT
KD
ESKD
FilterKD
CIFARIO
95.34
95.39
95.42
95.63
0.026
0.027
0.027
0.007
C0
78.07
78.40
78.83
80.09
0.053
0.061
0.067
0.029
Table 3:
Quantitative comparison of generalization performance for ResNet18 self-distillation (mean value of 5 runs). Refer to Table 6 for detailed results.
Input: Dataset ((x.; ((x,)yn)] Wn)) where I, 21, N) is the index for each pair # Train the teacher Initialize a network model, initialize an V K matrix called Qanooth
Go through the entire dataset, calculate 4smooth [n] Softmax(model(x, for epoch = 1, T do
for n E {1, ,,, y in random order do
p Softtmax
4mmooth [x] (1 ax) 4mmooth [n] + +a á
Update parameters based on loss CrossEntropy((, yn),
end for
Check the early stopping criterion; stop training if satisfied
end for
# Train the student
For each input X set Pur 4smooth [n]: train the network under the supervision of Pur
Algorithm 1: Filter-Kee KD. controls the cut-off frequency of low-pas. filter (0.05 here).
The first task we consider is nabsy classification, where we train and validate the model on a dataset with some labels randomly flipped. After training, all the models are evaluated clean held-out test set. The experiments are conducted on CIFAR (Figure 7) and TinylmageNer (Table 1), 0.1 means 10% of the labels interesting trend can be observed: the enhancement brought by Filter-KD is not significant when too small or large. That reasonable because for small a. few samples have bad labels, thus the possible enhancement might not be large. When ar is too high, the labels of similar points become less reliable, and the learning path will no longer head as reliably towards p'. Thus, for very high noise ratios, the performance of Filter-KI KD decays back towards that of OHT.
under different noise ratios
flipped. In Figure an
Filter-KD also outperforms other methods in both accuracy and calibration on the clean dataset. as illustrated in Table 3. This remains true in the more common KD case when the teacher network is bigger than the student; see results in Table 2. Note that in Table 2, we no longer keep sensitive to when the teacher is large (see Appendix D for more discussion); instead we optimize т for each setting. Here "Eff" is short for EfficientNet-BI (Tan & Le, 2019), "Res" is short for ResNet18, "Mobi" is short for MobileNetV2 (Sandler al., 2018), "VGG" is short for VGG-11 (Simonyan & Zisserman, 2015).
1, as the
baselines are
In summary, the explanation in Section 4 and experimental results in this section demonstrate that better supervisory signal, which can be obtained by Filter-KD KD, can enhance the prediction perfor- mance in both clean and noisy-label case. (It is also possible that the algorithm improves perfor-
8

---

Published as a conference paper at ICLR 2022
mance for other reasons well, especially in the clean label case; the influence of temperature may be particularly relevant.)
6 RELATED WORK
Human label refinement Peterson et al. (2019) use distribution of labels obtained from multiple annotators to replace the one-hot label in training, and find that doing so enhances both the general- ization performance of trained models and their robustness under adversarial attacks. However, this approach is relatively expensive, as requires more annotation effort than typical dataset creation techniques; several experienced annotators are required to get a good estimate of p*.
Label smoothing Another popular method is label smoothing (Szegedy et al., 2016), discussed in the previous sections. We believe that suitable smoothing can decrease ||Par p'112 to some extent, but the lower bound of this gap should be large, because label smoothing treats each class and each sample in the same way. Much prior research (e.g. Müller et al., 2019) shows that KD usually outperforms label smoothing as well.
KD and ESKD Knowledge distillation, by contrast, can provide a differrntt Pur for each training sample. KD was first proposed by Hinton et al. (2015) for model compression. Later, Furlanello et al. (2018); Cho & Hariharan (2019); Zhang et al. (2019); Yuan et al. (2020) found that distillation can help even in "self-distillation." when the teacher and student have identical structure. Our work applies to both self-distillation and larger-teacher cases.
Another active direction in research about KD is finding explanations for its success, which is often still considered somewhat mysterious. Tang et al. (2020) decompose the "dark knowledge" provided by the teacher into three parts: uniform, intra-class and inter-class. This explanation also overlaps with ours: if we observe the samples with different difficulty in the same class, we are discussing intra-class knowledge; if observe samples with two semantically similar classes, we then have inter-class knowledge. Among previous work, the most relevant study is that of Menon et al. (2021). Our Hypothesis is suggested by their main claim, which focuses the question of what is a good Par Our work builds off of theirs by helping explain why this good Plar emerges, and how to find better one. Besides, by looking deeper into the learning path of samples with different difficulty, our work might shed more light on KD, anti-noisy learning, supervised learning, overfitting, etc.
Noisy label task Learning useful information from noisy labels a long-standing task (Angluin & Laird, 1988; Natarajan et al., 2013). There are various ways cope with it; for instance, Menon et al. (2019) use gradient clipping, Patrini et al. (2017) use loss correction, Huang et al. (2020) change the supervision during training, and Zhang et al. (2020) employ extra information. It is possible to combine KD-based methods with traditional ones to further enhance performance; our perspective of learning paths also provides further insight into why KD can help in this setting.
DISCUSSION
In this paper, we first claim that better supervision signal, i.c., Per that is closer to p", leads to better generalization performance; this is supported by results of Menon et al. (2021) and further empirical suggestions given here. Based on this hypothesis, we explain why LS and KD outperform one-hot training using experiments on toy Gaussian dataset. To further understand how such better supervision emerges, we directly observe the behavior of samples with different difficulty by projecting them on 2-D plane. The observed zig-zag pattern is well-explained by considering the tradeoff between two forces, one pushing the prediction to be near that of similar inputs, the other pushing the prediction towards its training label; we give an intuitive account based on Proposition of why this leads to the observed zig-zag pattern.
To apply these findings on real tasks, in which the data and network are more complex, conduct low-pass filter on the learning path, and propose Filter-KD KD to further enhance Pur- Experimental results on various settings (datasets, network structures, and label noise) not only show the advantage of the proposed method as a method for a teacher in knowledge distillation methods, but also help verify our analysis of how generalization and supervision work in training neural network classifiers.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
This research was enabled in part by support provided by the Canada CIFAR Al Chairs program, WestGrid, and Compute Canada.
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816, 2020.
Dana Angluin and Philip Laird. Learning from noisy examples. Machine Learning, 2(4):343-370, 1988.
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang- On exact computation with an infinitely wide neural net. In Advances Neural Information Processing Systems, 2019.
Jang Hyun Cho and Bharath Hariharan. On the efficacy knowledge distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4794 2019.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In Proceedings of International Conference on Machine Learning. pp. 1607-1616. PMLR, 2018.
Mengya Gao, Yujun Shen, Quanquan Li, Junjie Yan, Liang Wan, Dahua Lin, Chen Change Loy, and Xiaoou Tang, An embarrassingly simple approach for knowledge distillation. arXiv preprint arXiv:1812.01819. 2018.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330 PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253/ 2015.
Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: beyond empirical risk minimization. Advances in Neural Information Processing Systems, 33, 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kerne:: Convergence and gen- eralization in neural networks. In Advances in Neural Information Processing Systems, 2018.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regu- larities of labeled data in overparameterized models. Proceedings of International Conference on Machine Learning, 2021.
Taehyeon Kim, Jaehoon Oh, Nakyil Kim, Sangwook Cho, and Se-Young Yun. Understanding knowledge distillation. http:://eenweeeeeeeeleoonntltlls
2019.
Sheng Liu, Jonathan Niles-Weeds Narges Razavian, and Carlos Fernandez-Graada. Early-learning regularization prevents memorization noisy labels. Advances in Neural Information Processing Systems, 2020.
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Can gradient clipping mitigate label noise? In International Conference Learning Representations, 2019.
10

---

Published as a conference paper at ICLR 2022
Aditya Krishna Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In International Conference on Machine Learning, pp- 7632-7642. PMLR, 2021.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, 2019.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances Neural Information Processing Systems, 26:1 1294,4 2013.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1944-1952, 2017.
Joshua Peterson, Ruairidh M Battleday, Thomas Griffiths, and Olga Russakovsky. Human robust. In Proceedings of the IEEF/CVF International
uncertainty makes classification Conference on Computer Vision, pp. 9617-9626, 2019.
Karsten Roth, Timo Milbich, Björ Ommer, Joseph Paul Cohen, and Marzyeh Ghassemi. S2sd: Simultaneous samilarity-bsse self-distillation for deep metric learning. Proceedings of Interna- tional Conference on Machine Learning, 2021.
David Ruppert. Efficient estimations from slowly convergent Robbins-Monro process. Technical report, Cornell University Operations Research and Industrial Engineering, 1988.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chied Chen. Mo- bileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4510-4520, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. Proceedings of International Conference on Learning Representations, 2015.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the Inception architecture for computer vision. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2818-2822 2016.
Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-6114, PMLR, 2019.
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Under- standing and improving knowledge distillation. arXiv preprint arXiv:2002.035532, 2020.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3903-3911 2020.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the per- formance of convolutional neural networks via attention transfer. Proceedings of International Conference on Learning Representations, 2017.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In ICCV, 2019.
Zhilu Zhang and Mert R Sabuncu. Self-distillation as instance-specifie label smoothing. Advances in Neural Information Processing Systems, 2020.
Zizhao Zhang, Han Zhang. Sercan O Arik, Honglak Lee, and Tomas Pfister. Distilling effective supervision from severe label noise. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp.
2020.

---

Published as a conference paper at ICLR 2022
Code, including the experiments producing the figures and Fi implementation, is available at httpp::///tttub.....//// wwtp://wwooooo.///t///oo../..to Ret.aaaarreeebtttee_eeeeee httpa//aaeeeeeettttteesaeeeearra signal.
A MISCELLANEOUS BACKGROUND
Calculation of ECE Expected calibration error is a measurement about how well the predicted confidence represent the true correctness likelihood (Guo al., 2017). For example, if model gives 100 predictions, each with confidence, say, q = k|x) E [0.7, 0.8).. Then we might expect there are 70 80 correct predictions among those 100 ones. To calculate this, first uniformly and Bm is all the samples divide [0, 1] into M bins, with each bin represented by Im (mM M
whose confidence falls into I.. Then, ECE is calculated as:
M Σ m=1
Bml
ECE =
|ace(Bm) - conf(
(5)
where (c) = [Bml EteBm. Bm 1 k,), conf [B.ml E B. q = k,(x)), k; is the true label and ij: (oggaaal((y)x))))) is the model's prediction. All the ECE mentioned in this paper is calculated by setting M = 10.
Barycentric coordinate system When visualizing the learning path, it is problematic to directly choose two dimensions and draw them in the Cartesian coordinate system, as illustrated in the first panel in Figure 8. In geometry, suitable way to project a 3-simplex vector onto 2-D plane is converting
point barycentric coordinate system. Specifically, have three basis vectors:
Vo = [0, To_ v1 = [1, and
v3 which are the corner and two edges of an equilateral
triangle respectively. Then the 2D-coordinate is calculated as (xr, y) = | Vi; V2] So every points in the left corner of Cartesian system plane be converted to the triangle barycentric system, as illustrated in the last panel in Figure 8.
(1
212
2222
[2:0
1.11 (1/222)
1 (15'13
1
0
(3/3'3
(LA0)
(0.0)
(1.0)
(3,00)
Figure 8: How to project a probability vector on to the plane of Barycentric coordinate system.
B RISK ESTIMATE VARIANCE BOUND
The following result is generalization of Proposition of Menon al. (2021), whose proof replicate and extend here:
Proposition 2. Let L be any bounded loss, L(y. 0) S l co, and consider Rear of (3). For any predictor f X + AK we have that
1
E [Rarr - R(f)) S N x
Var [PTLL(((x))] +
12

---

Published as a conference paper at ICLR 2022
where can be any of the following seven quantities:
e B Pxur
p" (x)]l2)
e²
E B
p" (xx)]12)
28 [ (KL(P (x) ||pp (x))
26 IE (L (x) |l (x))
26a E (XT(p=(x) (x)
2 E KT. IT
Piar ( (x))
E [LL |1 (x)) + KL(pp Piar (x))
Proof. To begin,
E (f, - - Varl (fr + (I) [Ra (F,D) - D
D'
For the variance term, since R(f) constant and Rear an average of N i.i.d. terms, we get
1 N Varlp (x)xxx())))
Var Rum (f, R(f)] =
The other term, Rux an average of i.i.d. terms and R(f) Ex p" (x) (()(),), is
E (ua R(f))² (() - px)) LL(x)))(((xx))) 2
For the first bound, we apply the Cauchy-Schwarz inequality,
(Par (x) (x)) L(((xx) < P... p" -x(x))aall((((x))
since the elements of (xf())) are each at most f, the term [f)( is at most VKE.
For the other bounds, we instead apply Hölder's inequality, yielding
(Pa.. x) (x)) L(f(xx) S P - p" (x)))x S (x) (x)
The KL bounds follow by Pinsker's 's inequality and then Jensen's 's inequality. The last bound, for the Jeffreys divergence, combines the two KL. bounds.
C Toy GAUSSIAN DATASET
delta
delta mu-1
mu-0.5
catt
12 diancc
Figure Correlation between test accuracy and | pa 1|2 for different settings.
della mu-4
delta muz2
delta mu=1
delta mu=0.5
Figure 10: Correlation between test ECE and |
p l|2 for different settings.
In Sections and 3, we apply a toy Gaussian dataset to verify two facts derived from Hypothesis 1, the learning path of specific samples. Here provide more details about this dataset.
13

---

Published as a conference paper at ICLR 2022
Runl
Run2
Run3
==0.5
y=
Y=2
y=10
7=0.5
7=2
y=
r=0.5
y=2
OHT
94.87
95.15
94.48
ESKD
95.02
95.27
95.34
95.05
95.41
95,41
95.39
94.76
94.11
94.65
94.88
94.75
FilterKD
95.18
95.87
95.66
94.81
95.11
95.56
95.71
94.81
95.09
95.31
95.22
95.16
OHT
77.85
78.23
77.99
ESKD
78.28
78.56
79.28
79.09
78.50
78.20
79.3
78.81
78.12
78.31
78.84
78.36
FilterKD
78.43
79.57
79.72
79.65
79.23
79.61
79.45
79.01
79.75
80.32
80.41
79.99
Table 4: The influence of temperature in self-distillation on CIFAR10/100 dataset.
Generate the dataset Here, we have N samples, each sample 3-tuple (x, y/ p"). To get one sample, we first select the label y = following an uniform distribution over all K classes. After that, sample the input signal xly=k ( I), where is the noisy level for all the samples. Hg is the mean vector for all the samples class k. Each μ4 is a 30-dim vector, in which each dimension is randomly selected from {-6,,, 6, J. Such process is similar selecting 30 different features for each class. Finally, we calculate the true Bayesian probability of this sample, i.c., p" (y)x).
Calculate the ground truth probability an uniform distribution, we have p" (y/x) Σsar P(X|==j) Following p(xly k) N o² I),
We use the fact that p" (ylx) x p(xlu)p(v). As follows p(xly=k)
we find p" (y/xx should have Softmax form, i.c., p =
8x o Specifically, we have:
et
py = k|x)
Ejth
8
20 1il12
Setup of experiments in Figure 1 in this experiment, generate toy Gaussian dataset with K 3, a 2 and N = 10 To reduce the variance of test error, we make a train/valid/test split with ratio [0.05 0.05, 0.9]. We train MLP with hidden layers, each with 128 hidden units and ReLU activations. We first conduct experiments on some baseline settings, i.e., learning from one-hot supervision (OHT for short), from smoothed label supervision (LS for short), from a converged teacher's prediction (KD for short), and from an early-stopped teacher's prediction (ESKD for short). In OHT case, an NN is trained under the supervision of ey. If we train this NN until the convergence of training accuracy, we obtain the Puu for the KD case. If we select the snapshot of that NN based on the best validation accuracy, we obtain the Pur for the ESKD case. we directly set Paar 0.9e, + 0.1u, where u is an uniform distribution over K classes, we obtain the supervision for I.S case. With different supervisions, we train new network with identical structure until the validation accuracy no longer increase. Furthermore, to see trend between generalization ability and | 12112. we run the experiment 200 times under different noisy supervisions.
D TEMPERATURE IN DIFFERENT KD METHODS
Temperature an important hyper-parameter in different kinds of KD methods, which might influ- ence the performance lot. In this appendix, we will discuss why prefer 1.
The role played by In general, the loss function in KD has the form:
1 -
== 3-
H(q", PTT)) + (1 3) H(q.
where 3 [0, 1] is is another hyper-parameter to trade-off the importance between one-hot label and teacher's predictions. Furthermore, for this loss, the gradient of L to logits z is:
aL
B- (1)
(q Pinn (1 (3) (q,
ey.) -g.)))
(8)
θz:
14

---

Published as a conference paper at ICLR 2022
delta mu=2
delta __mm_mmm111
A002
LS
3000
6350
3070
(39)
0473
3400
33e0
0200
KD
Can
230
ESKD
Ain
1.00
100
14:30
15 26
53
10.32
50
Figure 11: The influence of label smoothing factor on LS (first row) and different T on KD (other rows) under two different settings of the toy dataset. It is clear that these hyper-parameters won't t influence the performance too much as long as they are in good region.
Why we use =1 The most important reason for us to choose T = 1 (as well as 3 = 1), even with the risk of providing sub-optimal performance. is that we want to observe how much enhancement is brought by refining the label. In our mind, KD's success comes from the following two aspects:
(a) better label (i.e.. Pur is better than ey):
(b) better learning dynamics (soften to q" make the training easier).
The first one provides the student with more useful knowledge, and the second one helps the student to extract it. The focus of our paper is the improvement in labels. When the temperature is not 1, both (a) and (b) are influenced, as we are using q" (i.e., the smoothed student output) to match Pir (i.e., the smoothed target) during training and using to inference during testing. If we fix = 1, the only difference between Filter-KD, ESKD, KD, LS, and OHT training is Pta- Under such a condition, we believe the fact that Filter- KD/ESKD outperforms OHT (strictly better in each run) is enough to conclude KD methods can provide better labels. Furthermore, as Filter-KD is proposed directly observe the zig-zag learning path
to mitigate the high-variane issue in ESKD, and of samples with bad labels, we believe it is reasonable to make the conclusion in the paper.
Furthermore, using * 1 doesn't substantially change our analysis. As illustrated in Equa- tion (8), the gradient shows that q" moves to- wards Pir in each update. Hence our analysis in Proposition still holds, which means the trade-off between the two forces still exists.
0.90
F cas
0,900
0.75
0.599
0.70
0.590
10
20
0,903
T = is reasonably good in our settings In fact, the optimal 7 depends on many set- tings, e.g., teacher's size, optimizer, learning rate (scheduler), and etc: there is still no con- sensus on what is the best choice of r for all settings. So for the experiments in the main context, we fix and fine-tune other hyper- parameters to achieve good results. However, to verify this choice of 7 is not terrible, formed a coarse grid search on both CIFAR and toy examples. As illustrated in Table 4 and Fig- ure 11, choosing т E [1,2] seems to be good choice: the performance doesn't significantly decay until 7 is too large.
cas
rertacurey
"0,90.
0,500
0.899
10
15
20
Temperature
Tempersture
Figure 12: First row: large teacher to small stu- dent; second row: self-distill. Left column: 7 in a larger range; right column: in small range.
per-
Some readers might also curious about why our results don't match the common notion that the optimal 7 should be 4, which is first provided in Hinton al. (2015) and followed by much later
15

---

Published as a conference paper at ICLR 2022
1.0
0.8
zigzagness
0.4
0.2
Wrong labels
Correct labels
Figure 13: The learning path of samples with correct and wrong labels.
work (e.g. Gao al., 2018; Zagoruyko & Komodakis, 2017; Cho Hariharan, 2019). Some works also use grid search to conclude that temperature as high as 20 should be the best choice (Yuan al., 2020; Kim et al., 2019). We find this mismatch comes from the relative difference between the network size of teacher and student. In short, when distilling from large teacher to small student (as most of the cases aforementioned works), high preferred. When conducting self- distillation, need not be that high. For example, Zhang et al. (2019) and Roth et al. (2021) claim the best. To further
T = 1 is the best choice, while Zhang & Sabuncu (2020) claim T verify this, compare the temperature trend between two cases the toy dataset. In Figure 12, the first row is distilling a 10-layer, 256-width MLP to a 3-layer, 32-width MLP; the second row is self-distillatio between 3-layer, 32-width MLPs.
At the same time, we also run a grid search the smoothing factor a our Filter-Ki KD Table
Smoothing a FilterKD
0.01
0.05
0.1
0.2
0.5
1 (ESKD)
79.50
80.00
79.83
79.59
78.48
78.39
Table 5: The influence of the smoothing factor in FilterKD CIFARI00 (mean of 3 runs). For comparison, OHT obtained 77.64.
E How REPRESENTATIVE IS THE ZIG-ZAG PATTERN?
In the main paper, we only visualize the zig-zag learning path of few samples in Figures and 5. The readers might then wonder whether this pattern is representative across the whole dataset. To verify this, we define quantitative metric called zig-zag score. Specifically, we first calculate the integral of each dimension of the prediction:
T
QiE(L.....)) K}
Σ (x... 1m1
We then use the highest Q; among {1, K) {v), where y is the label's class, the zig-zag score. In other words, we focus on the behavior of on those dimensions that are not the training label. If this score large, we might expect the neighbouring samples exert high influence on the path, and vice versa. Note that as we have E, Qi = constant for any sample (as q' is K-simplex), this ag-zaag score might correlated with C-score of Jiang et al. (2021). However, their focuses are different: C-score is similar to -Qi=y and focuses more on how fast the prediction converge to the label: zig-zag score, on the other hand, more about how much path deviates towards a different class, possibly the label close to p".
With this score, we test the correlation between base difficulty and zig-zag score on toy datasets under different settings. From Figure 14, it is safe to conclude that samples with higher base dif- ficulty will have a more zig-zage zig-zagging path during training. We also compare the expected zig-zag score of the 1000 samples with flipped label in CIFAR10 (recall the experiment in Figure 6). In Figure 13, it is clear that the zig-zagness of these wrong label samples (which we are sure have high base difficulty) is significantly higher than the average.
Finally, we also randomly select some data samples in each class CIFARIO and visualize their learning paths. Figure 15 shows random samples for the noisy label experiment. It is clear that the
16

---

Published as a conference paper at ICLR 2022
de mu-4
delta, m
detta m
Gelto
1.0
0.0
0.8
2otagness
0.6
0.6
0.6
0..4
0.4
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.5 base
1.0
0.5
0.0
1.0
0.5
base
base
baie
Figure 14: Correlation between base difficulty and zig-zag score in four different toy datasets.
learning path of easy samples with correct labels converge fast while that of samples with wrong labels is zig-zag. In Figure 16, which shows the learning path with high zig-zag zag score when the training set is clean, can still observe some samples with zig-zag path. These samples might be ambiguous, with a quite flat p" However, as we do not know the true p" of these samples, it is impossible provide result like Figure 13.
F DISTANCE GAP UNDER DIFFERENT SUPERVISIONS
Figure 2 provides the distance gap between q and corresponding p" for each training sample in different training stages, under the supervision of one-hot label ey, From the results in this figure, we notice that the behavior of hard samples contributes more to the success of ESKD. Here we further visualize how these gaps changes when the model is trained under different types of supervision, i... ground truth. LS and ESKD.
In the first row, which is the result of label smoothing, we see the || p"112 |l2 (i.e., red dots) has a "V"-shape. The vertex location depends on the smoothing parameter we choose. However, as discussed previously, label smoothing has only one parameter to control Par. which is like using linear model fit a high-order function. So, although a proper smoothing value can bring better su- pervision than one-hot label, its upper bound might be limited. Regarding the training dynamics, can see a similar trend as the results shown in the one-hot case, i.e., all the samples first move down might expect label smoothing to outperform
and then converge to Paar From the middle panel, the one-hos case, because the scatters are closer to the x-axis, which represents the ground truth p".
The second and the third rows demonstrate the KD and ESKD case. Results in the KD quite similar to the one-hot case in Figure 2, because the converged Pur is close to ey. However, we still expect KD to outperform one-hot training, because the Pear is closer to p" than e, is. The last panel in this row also demonstrates that ||q - p" p'Il2 might be smaller than || p" p'1[2. which an explanation for why iterated self-distillation, e.g., Born Again Networks
can be considered (Furlanello et al., 2018), can improve the performance. For the ESKD case, we see the overfitting almost disappear: the distribution of the blue points do not change too much after the early stopping criterion is satisfied.
In the last row, which illustrates the training under the supervision of p", we find all the blue points move toward the x-axis, i.e., their Pur p", and finally converge to it. There is also no overfitting in this case. From the last two panels, we see the disperse of blue points is significantly smaller than all other settings, which means the network's 's prediction is quite close to p'. Hence the performance of this case is the best.
17

---

Published as a conference paper at ICLR 2022
(a) Samples with clean labels.
(b) Samples with wrong labels.
Figure 15: Random selection of samples in CIFARIO with 1000 flipped labels.
18

---

Published as a conference paper at ICLR 2022
Figure 16: Random selection of samples with high zig-zag score in clean CIFAR1O.
19

---

Published as a conference paper at ICLR 2022
T=0 (Initial)
(Early stop)
T=200 (Converge)
122
1.2
32-
1.2
=p (x0ll2
68
0.9
0.1
001
50
2.0
0.1
0.0
0.0
10
14
22
0.6
0.0
10
12
0.
64
0.6
0.6
12
14
||8,00
P Oxi]la
[[ey[x] -p [X]
[|ey(x) P [X]l
T-D (initial)
(Early stop)
T-200 (Converge)
12
1.3
[[otx] (xlll
[10xx) "colls
0.-
0.-
0.6
0.6
0.-
00-
0.-
10
[xe
(X)ll2
[1ey(r) p (xillaa
11e,00)
(xiill
T=0 (Initial)
(Early stop)
T=200 (Converge)
[|qtx] ppppp Dx012
0.-
0.8
0.8
[140]
0.8
24
[|0,00
P OOllla
[keyDx) p [x]
[|ey1x) [x]]la
(Early stop)
T=200 (Converge)
1.-
[lobx) -P (x0ll2
[lqbx) -pp (xell2
3.8
3.-
0.5
02
02
02-
[le,(x) - (xx]
[|ey(x)
[|e,00) P (x]Ila
Figure 17: Distance gap of each sample under different supervision: label smoothing, KD, ESKD, and ground-truth training.
20

---

Published as a conference paper at ICLR 2022
G MORE ABOUT THE DECOMPOSITION AND NTK MODEL
Proof of Proposition I. Recall z(x) = f(w, x) is the vector of output logits, and = Softmax (z) is the output probability. We are taking step of SGD observing Xu. and observing the change in predictions on X We begin with Taylor expansion,
(xx)
Vwq (xo)lwe (w w') + +[ -w'l3).
Xx1
Kx1
Kxd
dx1
To evaluate the leading term, we plug in the definition of SGD and repeatedly use the chain rule:
Vwq (uq'(xx)lw>
(w w²) = V2q' (xo)22))
(xol(x)))wr)) (- V V (xu))w)
Rxd
dx1
KXK
kxd
1xd
V2q' Vwz' VZ V (xxu)x Viz'(xx)) (xu)lws)
KxK
Rxd
1x
axd
(x00(x))22 ( love (V (xu)
KxK
Kxd
dx
RX1
= r) (xo) KC' (xos xu): (Ptar(xu) -x(xx)))
For the higher-order term, using as above that
x+| w' = - wwz)xzz' (xw) (Pwrr(Xw) - q((xw)
and noting that, since the vectors are probabilities, (XPr)((x)) - q' (xxxx) )1 is bounded, we have that
O +1 w 13 0n? ( [I(Vwz(x.))ww I1lp |Pur(( - qx.(x))ll" -0721IVV O
In the decompasition,
[qi 91)
-9192
9:9K
-9271
Q2(1 (1))))
929K
A'(x)) V.q'(x.)lzz
(10)
:
:
Q
9x
OK (1 9K)
which is a symmetric positive semi-definite (PSD) matrix with trace tr(.A* (x.)) = (x ΣK
As we have Σ. 9: 1, it is easy check the trace of this matrix is larger at the beginning of training (when q tends to be flat) than that at the end of training (q tends to be peaky), as illustrated by most panels in Figure 18. Given that the trace of matrix is the sum of its eigenvalues and AA'(x,, (x.) is PSD,
smaller tr(A4((x.) = - Σ of means this matrix will tend to shrink its inputs more. Hence the change in predictions tends to decrease when q' becomes more peaky.
The second term in that expression, (²'(,,, x,,), is the outer product of gradients x, and x,- Intuitively, if their gradients have similar directions, this matrix is large, and vice versa. This matrix known as the empirical neural tangent kernel, and can change through the course of training as the network's notion of "similarity" evolves. For appropriately initialized very wide networks trained with very small learning rates, XC' remains almost constant during the course of training, and is almost independent of the initialization of the network parameters; the kernel it converges to is known as the neural tangent kernel (Jacot et al., 2018; Arora et al., 2019).
Learning Dynamics of (xx)) In Proposition 1, we decompose (x.) -q' (x.) into three parts; we use this to analyze what the learning path of a training sample might be like in Section 3.3. Here we will provide more detailed illustration of the two groups of force imposed on qxx.).
Specifically, q (x.) - (x.) will be influenced by two variables, i.c., updating sample x,, and time t. We discuss their effects separately. For x only the last two terms, i.c., (x,, x,) and (x.) - q' (x.,)) depends on it.
'The matrix A can be observed to he the covariance matrix of categorical distribution with item probabil- ities 9, and hence PSD.
21

---

Published as a conference paper at ICLR 2022
30
MAN Wishlr
77
1
12
29 Mke
21 2urge
47 Altas
27 k9
35. Caser
2.0
8 Eeerge
40.4 J0h
55' prige
praige
enige
2
38 onigg
Aade
I6's mirg
Zxthe
Rade
Mafe
aamm
$1
Figure 18: Upper panels: how tr(AA' (xx)) changes during training. Each panel represents a specific Xn. The panels are ordered by their integral difficulty, from left-to-right and up-to-down. The x- axis is the number of epochs, and the y-axis is (r cos(xo, x,) and tr(CC)). Panels are ordered by the integral difficulty of x,. The subtitle of the panel gives the sample ID, the class it belongs to (i.e., A, B, C) and the color of their corresponding class (i.e., A is blue, is orange and C is green).
Lower panels: the correlation between
In Section 3.3, we claim that x, and x,, are similar, the norm of (x", x.) might be large, and vice versa. Here we empirically demonstrate this using a toy Gaussian dataset, as illustrated in Figure 18. The figure shows how the similarity between x, and x,, correlates with tr(AC (Xo. xu)). Each panel represents one X which is claimed in the title of the subfigures. The x-axis represents the rank of the cosine similarity between observed x,, and all N training samples (including itself). The y-axis the trace of KC (x., (x). The color of each scatter point is the true label of x,- From the figure, we can observe clear decreasing trend, which means smaller similarity leads to larger tr (AC (xo, x,)). Additionally, the term (x..) q' (xx)) provides a direction that q' (x.) should move towards.
We claim in Section 3.3 that at any point in the input space, the labels of these input samples might follow the ground truth distribution, i.e., p² (y/ xx). Hence most of the neighbouring x might pull q(x.) towards its ground truth p" (v)x.). We will discuss the norm of this term when discussing the influence of t. In short, at any time, the neighboring x,, will impose stronger force on x, and the direction of the force roughly points to the ground truth p" (v(x.).
22

---

Published as a conference paper at ICLR 2022
As discussed, IC' is roughly constant over t in the very-wide limit. For finite width networks, how- ever, it adapts to reflect the network's 's new "understanding" of similarities. For instance, it might learn that certain types of images are more semantically similar than the randomly -inttalliiiiilalizzedd net- work thought. This does not fundamentally change our intuitions as long as it doesn't happen too often, but could potentially lead to more complicated zig-zag patterns the network's estimate of p" from neighboring points perhaps improves over time.
Over the course of training, A' (xx) and (x.) (o's))((xx))) none of these terms have an easy analytical expression w.r.t. d: q' is quite complicated. Thus, we pro- vide some intuition, with experimental verification. In Figure 18, we show how tx.))((x)) depends on q': flat q' leads to larger trace. A similar trend also exists in the norm of (Pur (x qxx)). As the initialized q tend to be flat, updates of any samples will influence network's ' parameters a lot. When the training progresses, those easy samples converge fast, so their | (x.) q' (x.) ll and tr(AA) become small. However, as the q' for the hard sample still far away from its ey, the large (ar((xx) q' q'(x.)ll and tr(AA) will finally drag of back towards the one-hot distribution, as illustrated in Figure 3.
(x.)) will also change. In practical regimes,
H COMPARISON TO LIU ET AL. (2020) AND HUANG ET AL. (2020)
The main claim of this paper is that better supervisory signals can enhance the generalization ability of the trained model. Inspired by the success of KD,, we find that the neural network can sponta- neously refine those "bad" labels during training by observing their learning path. The learning path of those hard samples will first move towards their true p(y(z)) and then converge to their label Ptar or ey. We explain why this phenomenon occurs by expanding the gradients of each training sample. This phenomenon is also explained in Proposition 1, and formally proved for a particular softmax regression model by Liu et al. (2020). As complement, we propose an explanation using an NTK model, and experimentally verify it by observing the learning path and distance gap during training.
Another difference between these two works that we consider the problem of refining supervisory signals, while Liu et al. (2020) consider correcting wrong labels (a special case of "bad" supervi- sion). Our work provides additional emphasis and empirical study for the clean-label case.
Regarding the algorithm, Liu et al. (2020) design effective regularization term inspired by early stopping regularization. They apply exponential moving average (EMA) on the model's output when calculating this regularization term to further enhance the performance. This method is similar to that proposed by Huang et al. (2020), who switch between optimizing the training loss and an objective based on the EMA of the model over the course of training.
Although bears significant similarity to these methods, Filter-KD does not change the course of training the teacher model. Rather, we propose (based the high variance of the zig-zag learning paths) to simply use the the EMA of that model's outputs as a teacher for later distillation.
We suspect that these three algorithms work because of essentially the same underlying principle, whether we think of this as being based on the zig-zag learning path or as early-stopping regulariza- tion. We expect that this principle will be helpful moving forward in the field's understanding of the learning dynamics of SGD methods for neural networks
I Low PASS FILTER ON NETWORK PARAMETERS
In Section 4, we point out the high variance issue of the traditional KD methods after observing the learning path of those hard samples. We then propose Filter-KD algorithm to smooth the output of each training sample during training. Such low pass filtering method is quite similar to the momentum mechanism used in self-supervised learning, e.g., from the classic method of Ruppert (1988) to MOCO (He et al., 2020), which conduct low pass filtering on each parameter of the network. As mentioned before, the proposed Filter-KD algorithm might require more memory when the dataset becomes larger, because 4smoath would record every training sample's 's prediction.
Conducting low pass filtering network parameters might be good way to solve this. To verify whether this method works, we train a ResNet18 8 on CIFAR100, using one-hot supervision. At the beginning of training, we copy the training model and call it tracking model. At the end of each
23

---

Published as a conference paper at ICLR 2022
0.8
Test accuraty
Test accuracy
0.6
Filter
OHT
Track KD
Track
Track
0.3
Epoch
Epoch
Figure 19: Behavior of the tracking model (with low pass filter each parameters). ResNet18 trained for 150 epochs on CIFARIO0.
update (i.e., each batch), the parameters of training model updated as usual while the tracking model's parameters are updated with momentum. Specifically, for the training model, w (+1
Irain
w v/V L, for the tracking model, w w (1 (1-ol a--aw 150 epochs, and show the learning curve of test accuracy the left panel of Figure 19. We see the performance of the tracking model converges faster than the training model, which reasonable because filtering parameters regularizing the time, find this tracking model is not good as 4smooth when teaching a new model, as illustrated in the right panel of Figure 19.
aw We train the model for
of this two models are the same. At the training process. However, the converging accuracy
As this paper mainly focuses on the behavior of the model's prediction, the discussion and ex- periments of filtering in parameter space is limited. However, as many papers demonstrates the effectiveness of this momentum mechanism, we think important explore the relationship
further.
Test ACC
Test ECE
Runl
Run2
Run3
Run4
Run5
Runl
Run2
Run3
Run4
Run5
OHT
95.35
95.30
95.42
95.23
95.42
0.027
0.027
0.026
0.025
0.025
KD
95.30
95.38
95.42
95.44
95.42
0.027
0.027
0.027
0.026
0.026
ESKD
95.29
95.41
95.39
95.58
95.42
0.026
0.029
0.025
0.028
0.027
FilterKD
95.66
95.68
95.49
95.76
95.58
0.005
0.006
0.008
0.011
0,006
OHT
78.27
78.31
77.97
77.78
78.02
0.053
0.053
0.052
0.053
0.054
KD
78.64
78.55
78.03
78.18
78.49
0.060
0.057
0.062
0.066
0.059
ESKD
78.84
78.73
78.85
78.97
78.74
0.065
0.066
0.067
0.070
0.066
FilterKD
79.87
79.93
80.19
80.22
80.23
0.028
0.026
0.034
0.028
0.031
Table Each run of results in Table 3. each run, the initialization of student networks for different methods are controlled to be the same.
24


Reviews:
[{"review_id": "THZ-8CeKLx", "reviewer": "Reviewer_bjwd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**This review has been updated since it was originally posted. See thread below.**\n\n## Strengths:\n- The paper is, for the most part, well-structured and easy to read. Argument and ideas are developed step-by-step, formulated and substantiated formally, based on previous work.\n- Establishing hypothesis up front.\n- Starting from a toy example, building up the contribution from initial observations, and finally moving on to more complex/realistic scenarios after.\n- Tackles a relevant problems for the ML community, namely knowledge distillation and noisy labels.\n\n## Weaknesses:\n- The concept of noise is not well-defined, to the point where the problem definition becomes confusing: is the noise big enough to flip the target label? More precisely, does `arg max_i p*(y|x)_i = arg max_i e_yi` always hold? If it does, the hard sample from Figure 3 does not seem to comply. If the equality doesn't hold, the problem definition is more about data with wrongly labeled samples; in this case, the insights from section 3 will be contingent on the number of samples for which the equality mentioned above doesn't hold (i.e., mislabeled samples). This aspect has not been addressed in the paper. Moreover, the concept of easy, medium and hard samples does not explicitly compensate for mislabeled samples, and hard samples *could* trivially correspond to the mislabeled samples.\n- KD, ESKD and LS depend substantially on a temperature hyper-parameter which the paper does not address. Having the wrong temperature can drastically affect the performance of these baselines.\n- The labels from ESKD can be seen as a special case of the proposed Filter-KD where alpha = 1. Therefore, one could expect the labels from an ESKD teacher to be similar to those of Filter-KD if the teacher were trained using less epochs (in other words, if the early-stopping criterion forces the teacher to stop sooner). Even though ESKD is a valid baseline, is it the most compelling one to justify the introduction of Filter-KD?\n- Does not use a real dataset (i.e., not a toy dataset) for which `p*` is available like CIFAR10H **[1]** to evaluate Hypothesis 1.\n\n### Detailed comments:\n- The abstract is not a clear description of the paper, lacks context and coherence. It is not clear what the contributions are (is it a criterion to measure generalization, a function to measure difficulty or a novel way to do knowledge distillation).\n\n- Increase text size in all figures (at least as big as the figure's caption).\n\n> In this paper, we first clarify what makes supervision good for a classification problem, and then explain two existing label refining methods, label smoothing and knowledge distillation (KD), in terms of our proposed criterion.\n\n*The paper clarifies or proposes a new criterion to quantify generalization performance?*\n\n> we look deeper into the learning path of the network’s predicted distribution for training samples with different difficulty.\n\n*What is the 'learning path'? How is difficulty defined? Make it clear that these are definitions that will be introduced later on.*\n\n- Evaluate labels produced by teachers trained on KD, ESKD, LS and Filter-KD wrt. human annotated labels from CIFAR10H **[1]**.\n\n- Proposition 3 from Menon et al.: Lacks context. I recommend explaining further where it comes from to make the argument more self-contained.\n\n- Figure 1b-c: The range on the y-axis is rather small (coverage of 0.7% for accuracy and 0.02 on ECE), exaggerating correlation between accuracy and similarity between `p*` and `p_tar`. Also, KD and LS depend on a temperature parameter, which according to the supplementary material, has only been tested with with one setup.\n\n> ...leads to better generalization performance, as measured either by accuracy (ACC) or expected calibration error (ECE)\n\n*It is critical to define the term \"generalization\" here. If these are the two criteria that are going to define generalization, or are distribution shifts also being considered?*\n\n- Figure 2: It would be interesting to see the [distribution of the entire dataset for each axis](https://matplotlib.org/stable/_images/sphx_glr_scatter_hist_001.png). The scatterplot can hide how dense the distribution is (and how thin the tails are).\n\n- Figure 3 (hard sample): How can the `p*()` label for the hard sample be so close to the bottom right corner (say [0.1, 0.8, 0.1]), and yet the one-hot encoding be all the way across, on the bottom left corner ([1, 0, 0])? It looks as if the samples were mislabeled. Refer to the first point under 'Weaknesses'.\n\n- Figure 3: How representative is this behavior across the dataset? How many easy/medium/hard samples are there?\n\n- Figure 3: The paths on the right are close to impossible to see and interpret.\n\n- Proposition 1: `x_o` has not been defined within the proposition.\n\n> however, for a harder sample (middle), it is very difficult to observe any patterns\n\n*Capitalization*\n\n- Figure 4: What does the black dashed line mean?\n\n- Section 5.1: What labels are used for the students' test set? Is it the original OHT or a label set by the corresponding teacher?\n\n- Section 5 & Figure 5: How does Filter-KD compare to models trained using LS and KD (with optimum temperature)? Any reason to leave them out?\n\n### References\n[1] Peterson, Joshua C., et al. \"Human uncertainty makes classification more robust.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.", "summary_of_the_paper": "This paper introduces a method for doing knowledge distillation with noisy labels. The contribution consists on representing distilled labels as a weighted moving average of the predicted labels from a teacher as it trains (using noisy one-hot encoded labels). Finally, a student is trained using the distilled labels.", "summary_of_the_review": "The paper is easy to follow and has contributions that are sufficiently motivated. Empirical evaluation shows that the proposed method is more robust to noisy labels. However, the problem definition (therefore the scope of the paper) is not clear, and the core contribution Filter-KD has not been evaluated to an extent that justifies its simplicity.\n\n**UPDATE: After discussing with the authors (see thread below) I have updated my review to reflect my view of the paper.**", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RSlxiX1fvYk", "reviewer": "Reviewer_QVYW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In terms of novelty, clarity, and significance, I think this paper is good. The question of why distillation improves performance is an important one. Analyzing the learning paths of individual examples is an interesting idea. To my knowledge, the visualization of learning paths is novel, as is the Filter-KD method. I enjoyed reading the paper, and I think the ideas described have the potential to inspire interesting future work.\n\nWith that said, I’m not entirely convinced that the explanation for the success of KD provided here is accurate for real-world KD use cases.\n\n1. A major claim is that KD’s success is related to the zig-zag pattern, but this claim does not seem to be sufficiently supported by the experiments involving real data. It is shown that, on some examples, there is a zig-zag in the learning path, but there is no direct demonstration that this zig-zag is related to the efficacy of KD. One idea is to measure the degree of the zig-zag for each example and then verify that it suffices to use KD only for these examples and one-hot training for others. This seems like the bare minimum thing to show, but the entropy of the model distribution at the early stopping point might be correlated with the degree of zig-zag, in which case it seems somewhat trivial that the low-entropy outputs could be replaced with one-hot labels. I am not sure exactly what a good empirical test of the “zig-zag” hypothesis would look like, but I don’t find the evidence provided compelling enough.\n\n2. It’s not clear to me whether distillation is performed at a temperature of 1 or a temperature that is tuned somehow. Assuming the magnitude of the final logits provides some indication of learning difficulty, I could see distillation at a temperature >1 providing some of the benefit of early stopping. This also seems like a relevant baseline for FilterKD, since both increasing the temperature and averaging predictions will have the effect of increasing the entropy of the target distribution.\n\n3. The numbers in Table 1 seem uncompelling. The caption does not say whether the +/- is standard error, standard deviation, or something else, but assuming it is standard error, I don’t think the improvements on CIFAR-10 or AG News would be statistically significant. The improvements from KD on these datasets are small to begin with, so they may not be particularly good benchmarks for this work. The performance with noisy labels is more compelling, but doesn’t help to validate that the success of KD on clean data is attributable to the phenomenon described.\n\n4. I’m suspicious of the idea of “base difficulty.” “Base difficulty” is defined as the L2 distance between the one-hot label and the true conditional distribution. A nice aspect of base difficulty is that it depends only on the true data distribution. However, it seems to capture only two kinds of “difficult” examples: those that are ambiguous (if the conditional distribution has high entropy) or incorrectly labeled (if the conditional distribution has low entropy). The proposed “integral difficulty” measure further captures the difficulty of _learning_ particular examples, and is somewhat similar to previously proposed metrics such as [C-score](https://arxiv.org/abs/2002.03206) (see also the learning speed based proxies in that paper, one of which is equivalent to integral difficulty) or [variance of gradients](https://arxiv.org/pdf/2008.11600.pdf). Intuitively, data points with low base difficulty could have high integral difficulty if they are located in regions of the distribution with low density. The labels in CIFAR-10/100 are relatively clean and the images are relatively unambiguous, so I suspect that most of the high integral difficulty examples do not actually have high base difficulty. \n\nMinor:\n- “Early-stopped knowledge distillation” is introduced without any further description. Only from the appendix is it clear that it means knowledge distillation from an early-stopped teacher, as opposed to early stopping during knowledge distillation.\n- IIUC, the intuition from Section 3.3 is that the reason for the “zig-zagging” in Figure 3 is that the label assigned to a given point changes not only based on the gradient of the loss for that point, but also on the gradients of the losses on other similar points. This point seems intuitive even without bringing up the $\\mathcal{A}^t$ or the empirical NTK, and I am not sure that Proposition 1 helps to understand it. Further analysis of the dynamics of $\\mathcal{A}^t$ and the stationarity of the empirical NTK might make this decomposition more interesting, but the analysis in Appendix E doesn’t seem conclusive.\n- Missing paren in definition of integral difficulty on p. 5.\n- The idea of taking a moving average of network weights, as explored in Appendix G, is much older than MoCo and it would be good to cite relevant literature, e.g., [Ruppert](https://ecommons.cornell.edu/bitstream/handle/1813/8664/TR000781.pdf?sequence=1), [Polyak & Juditsky](https://epubs.siam.org/doi/abs/10.1137/0330046), and the [Inception paper](https://www.cv-foundation.org/openaccess/content_cvpr_2015/html/Szegedy_Going_Deeper_With_2015_CVPR_paper.html), which AFAIK was the first or among the first papers to apply this method in deep neural network training.", "summary_of_the_paper": "This paper proposes an explanation for the success of distillation. It first experiments with synthetic Gaussian data. On synthetic data, it shows that distillation works better from an early-stopped model. It probes why, and finds that, when the one-hot label is far from the true conditional distribution, the early-stopped model’s distribution tends to be closer to the true distribution than either the one-hot label or the converged model. The early-stopped model is better because, over the course of training, the model tends to produce a distribution close to the true one before finally overfitting the one-hot label. Some examples show similar patterns for real-world networks/datasets, provided the predictions over training are smoothed. Motivated by these results, the paper proposes to use an averaged distribution of labels from different steps of training rather than the distribution at the end of training, and shows that this improves performance by a little bit.", "summary_of_the_review": "This paper is well-written, novel, tackles an important question, and contains potentially interesting ideas. However, the evidence provided is not quite sufficient to validate the claims made. The suggestion that zig-zag in the learning trajectory is important to the success of distillation on real (non-noisy) data is not well-demonstrated, and the benefits of the proposed FilterKD technique seem marginal on real-world data.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NYkWQpGf02", "reviewer": "Reviewer_8bUF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The theoretical explanation and the illustrative toy experiments are very suggestive of what may actually be happening with ESKD. These contributions are very novel and useful in my opinion. They show that for harder examples, compared to easier examples, the learning path initially takes the network output to the true $\\mathbf{p}^*$  before making a turn for the one-hot target. \n\n2. The experiments when performed more real-world datasets using a proxy called \"integral difficulty\" instead of the true $\\mathbf{p}^*$ (as it is unknown) still show very similar patterns, at least for the easy examples, compared to the toy Gaussian dataset where $\\mathbf{p}^*$ can be calculated exactly. This is intellectually satisfying.\n\n3. The experiments show how hard examples, as measured in terms of integral difficulty, show a very different -- highly erratic -- learning path compared to easier examples. More importantly, the authors show that by low-pass filtering, the path actually looks quite similar to the  case of medium hardness. This is very important as it allows the authors to propose an intuitive novel KD framework called FilterKD where the targets are smoothed teacher outputs. \n\n4. The FilterKD algorithm is novel, intuitive relatively easy to implement for smaller datasets.\n\n5. The experiments on CIFAR-10 and CIFAR-100 show that FilterKD is indeed better than using ESKD. \n\n6. Experiments with noisy labels are well-performed and provide good evidence for the theoretical ideas as well as FilterKD.\n\n\nQuestions and suggestions for improvement:\n\nExperimental results need to be improved. I would suggest the following additions\n\n1. Results of regular KD in addition to ESKD. \n\n2. There are results only with one teacher-student architecture pair. A wider set of teacher-student pairs is required before concluding that the proposed method is better.\n\n3. In Table1, both the teacher and student have the same architecture. One of the main reasons to do KD is for model compression. So having a larger teacher and smaller student would be quite useful for a larger audience. I would suggest looking at the benchmark and code provided by Tian et al. in Contrastive Representation Distillation, ICLR 2020 as an easy starting point.\n\n4. The memory requirements for storing smoothed targets in memory may be prohibitive. But why can't we load them from permanent memory as and when required, like we do with a dataloader for images and videos?\n\n5. Experiments on a dataset like ImageNet or at least TinyImageNet would strengthen the paper more. ", "summary_of_the_paper": "Based on empirical observations and the neural tangent kernel theory, the authors present an explanation of why Knowledge Distillation (KD) and Early Stopping KD (ESKD) are better than using just the one-hot labels as well as how hard samples have high variance curve in the output space. Based on these observations, they propose a novel method called Filter-KD which yields better performance over using just one-hot labels as well as EKSD for standard vision datasets.  ", "summary_of_the_review": "Overall, I believe there are solid contributions in this submission which can be further demonstrated with the suggested experiments. Hopefully, the authors will be able to address the weaknesses I have outlined. \n\nFor now, I am giving this a 6: marginal acceptance. I am happy to increase the rating based on the authors' response to the next level.\n\nUPDATE AFTER AUTHOR RESPONSE:\n\nI thank the authors for a clear response to my suggestions and the additional experiments. I am satisfied with the additions and modifications and I am increasing my score. Please try to add ImageNet results if possible, as it may help others in the field who use it as a common benchmark.\n\nI have also read the other reviews and the authors have also addressed them well too. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "DQqFbmKP-Yh", "reviewer": "Reviewer_JaQp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The observed \"zig-zag\" pattern is interesting and gives one reason that early stopping is effective in knowledge distillation. The improvement obtained by early stopping and the proposed FilterKD is rather minor. ", "summary_of_the_paper": "This paper analyzes the learning process in knowledge distillation and proposes a method called Filter-KD. It was evaluated by using CIFAR10, CIFAR100, and AG News.", "summary_of_the_review": "I am still not sure the proposed method is justified well in theory. It is based on NTK theory but it is applicable only for a sufficiently wide network. Also, several \"zig-zag\" patterns are shown in figures, but we are not sure of their definition and how often it appears in the real dataset. The improvement in accuracy is rather minor.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors made substantial  improvements to the originally submitted manuscript; however, reviewers initially remained reluctant to support the paper for acceptance based on the degree to which they were confident in the underlying arguments / position taken by the authors and the evidence provided to support their position and arguments. There are also concerns about the significance of the gains in performance afforded by the proposed approach.

During the author response period two reviewers became satisfied with the additions and modifications leading to an increase in the final score. It will be critical for the authors to try to add ImageNet results if possible in addition to other promises made to reviewers.

The AC recommends accepting this paper.
Decision: Accept (Poster)