Scaffolding a Student to Instill Knowledge
Authors: Anil Kag
Durmus Alp Emre Acar
Aditya Gangrade
Venkatesh Saligrama
Venue: ICLR 2023 poster (2023)
Keywords: knowledge distillation
tiny capacity student
large capacity teacher
budget constrained learning

Abstract: We propose a novel knowledge distillation (KD) method to selectively instill teacher knowledge into a student model motivated by situations where the student's capacity is significantly smaller than that of the teachers. In vanilla KD, the teacher primarily sets a predictive target for the student to follow, and we posit that this target is overly optimistic due to the student's lack of capacity. We develop a novel scaffolding scheme where the teacher, in addition to setting a predictive target, also scaffolds the student's prediction by censoring hard-to-learn examples. Scaffolding utilizes the same information as the teacher's soft-max predictions as inputs, and in this sense, our proposal can be viewed as a natural variant of vanilla KD. We show on synthetic examples that censoring hard-examples leads to smoothening the student's loss landscape so that the student encounters fewer local minima. As a result, it has good generalization properties. Against vanilla KD, we achieve improved performance and are comparable to more intrusive techniques that leverage feature matching on benchmark datasets.


Published as a conference paper at ICLR 2023
SCAFFOLDING A STUDENT TO INSTILL KNOWLEDGE
Anil Kag" ani 1 kaggbu..... edu
Durmus Alp Emre Acar" a ppaarrbb..ed edu
Aditya Gangrade"
agangra2@andrew. cmu edu
Venkatesh Saligrama'
srvêbu.edu
ABSTRACT
We propose a novel knowledge distillation (KD) method to selectively instill teacher knowledge into a student model motivated by situations where the student's s capacity is significantly smaller than that of the teachers. In vanilla KD, the teacher primarily sets a predictive target for the student to follow, and we posit that this target is overly optimistic due to the student's 's lack of capacity. We develop novel scaffolding scheme where the teacher, in addition to setting a predictive target, also scaffolds the student's prediction by censoring hard-to-learn examples. The student model utilizes the same information as the teacher's soft-max predictions inputs, and in this sense, our proposal can be viewed as a natural variant of vanilla KD. We show on synthetic examples that censoring hard-examples leads to smoothening the student's 's loss landscape so that the student encounters fewer local minima. As result, it has good generalization properties. Against vanilla KD, we achieve improved performance and are comparable to more intrusive techniques that leverage feature matching on benchmark datasets.
INTRODUCTION
fundamental problem in machine learning is to design efficient and compact models with near state-of-the-ar (SOTA) performance. Knowledge Distillation (KD) (Zeng & Martinez, 2000 Bucila jet al. 2006 Ba & Caruana 2014 Hinton et al.) 2015) is a widely used strategy for solving this problem wherein the knowledge from large pre-trained teacher model with SOTA performance is distilled onto a small student network.
Vanilla KD. Hinton al. 20155) proposed the popular variant of KD by matching the student soft predictions, s(x) with that of the pre- tre-trained teacher, t(x) on inputs x. Informally, during student training, an additional loss term, D z. (t(x), s(x)) is introduced that penalizes the difference between student and teacher predictive distributions using Kullback-Leeihe (KI.) divergence. This promotes inter-class knowledge learned by the teacher. We will henceforth refer to Vanilla KD as KD.
Capacity mismatch between student and teacher. One of the primary issues in KD is that the loss function is somewhat blind to the student's 's capacity to interpolate. particular, when the student's capacity is significantly lower than the teacher's, ss we expect the student to follow the teacher only those inputs realizable by the student.
We are led to the following question: What can the teacher provide by way of predictive hints for each imput, so that the student can leverage this information to learn to its full capacity?
Scaffolding Student to Distill Knowledge (DiSK). To address this question, propose that the teacher, during training, not only set predictive target, t(x), but also provide hints on hard to lear inputs. Specifically, the teacher utilizes its model to output guide function, g(x), such that the student can selectively focus only on those examples that it can learn.
Our Proposal:
(xx) 1, teacher discounts loss incurred by the student on the input x. g x
R 0, teacher signals the input x as learnable by student.
"CEE Department, Boston University, Boston, MA
Statistics Department, Camegic Mellon University, Pittshurgh, PA

---

Published as a conference paper at ICLR 2023
With this in mind we modify the KL distance in the KD objective and consider, DEs (t(x), ((s(x), g(x))) where o(s,g), which will be defined later, is such that, ((s. 0) = s the guide function g to ensure that only
if g offers no scaffolding. We must impose constraints hard-to-learn examples are scaffolded. In the absence of such constraints, the guide can declare all examples to be hard, and the student would no longer learn. We propose do so by means of budget constraint (ss,g) g) S. 8 to ensure that the guide can only help on a small fraction of examples. While more details are described in Sec. [ we note that, in summary, our proposed problem is to take the empirical linear combination of the aforementioned KL distance and cross-entropy term as the objective, and minimize it under the empirical budget constraint.
We emphasize that g(x) is used only during training. The inference logic for the student remains the same as there is no need for g(x) during inference. The guide function supported student training has three principal benefits. The benefits are explored in Sec.
Censoring Mechanism. Our guide function censors examples that are hard to learn for the student. In particular, when there is a large capacity gap, it is obvious that the student cannot fully follow the teacher. For this reason, the teacher must not only set an expectation for the student to predict, but also selectively gather examples that the student has the ability to predict.
Smoothen the Loss landscape. We also notice in our synthetic experiments that whenever scaffolding is powerful, and can correct student's 's mistakes, the loss landscape undergoes dramatic transformation. In particular, we notice fewer local minima in the loss viewed by the guided student. Good Generalization. The solution our constrained optimization problem in cases where the guide function is powerful can ensure good student generalization. Specifically, we can bound the statistical error in terms of student complexity and not suffer additional complexity due the teacher.
Contributions. We summarize our main results.
We develop a novel approach KD that exploits teacher representations to adjust the predictive target of the student by scaffolding hard-to-learn points. This novel scaffolding principle has wider applicability across other KD variants, and is of independent interest.
We design a novel response-matching KD method (Gou et al. [02T]) which is particularly relevant in the challenging regime of large student-teacher capacity mismatch. We propose an efficient c constrained optimization approach that produces powerful training scaffolds to learn guide functions.
Using synthetic experiments, we explicitly illustrate the structural benefits of scaffolding. In particular, we show that under our approach, guides learn to censor difficult input points, thus smoothening the student's 's loss-landscape and often eliminating suboptimal local minima in it.
Through extensive empirical evaluation, yields large and consistent accuracy gains over vanilla KD under large student-teacher capacity mismatch (upto 5% and 2% on CIFAR-10 100 and Tiny- Imagenet..
demonstrate that the proposed DiSK method;
produces student models that can get near- teacher accuracy with significantly smaller model complexity (e.g. 8x computation reduction with 2% accuracy loss on CIFAR-100). improves upon KD even under small student-teacher capacity mismatch, and is even competitive with modern feature matching approaches.
2 ILLUSTRATIVE EXAMPLES
We present two synthetic examples to illustrate the structural phenomena of the censoring mechanism and smoothening of student's 's loss landscape enabled by the scaffolding approach DiSK, which lead to globally optimal test errors. We defer exact specification of the algorithm to Sec
Example 1 (1D Intervals). Consider a toy dataset with one dimensional features E [0,9] and binary class labels y E (Red, Blue) as shown in Figure I There are two Blue labelled clusters as in [2, 3] and in [5, 7]. The remaining points are labelled Red. We sample 1000 i.i.d. data points as the training set and 100 data points as the test set with balanced data from both classes, We describe the details of the experiment setup such models and learning procedure in Appx. A.I
Teacher T belongs to the 2-interval function class, and the aapacity-constrain student S belongs to the 1-interval function class. Since teacher capacity is sufficient to separate the two classes without error, it learns the correct classifier (see Figure[[] In contrast, the best possible student hypothesis cannot correctly separate the two classes. Hence, the student will have to settle onto one of the many
2

---

Published as a conference paper at ICLR 2023
Loss Contour
DISK Contour
minima (100% Acc.)
Min-A (67% Acc.)
value (a.a4b)
2355
S Min-B (83% Acc.)
242
Min-C (87%
value in (a,a+b)
value in (a.a+b)
(a)
Figure la ID Intervals. Data distribution on x-axis [0, 9). Teacher T learns correct decision boundary with 2-intervals and is the global minima for this binary classification task. Student S has many bad local minima, and one global minima that best describes the dccision boundary with 1-interval, [b] KD training. Loss contour plot shows the various local minima exist. Tc DiSK training. Loss contour plot shows the bad local minima longer exist.
Table 1: The number of times each method lands on various local minima in toy problems for 100 runs.
Dataset
ID Intervals
Gaussians
Minima
(Globll
D (Global)
Accuracy
67%
83%
87%
70%
80%
90%
100%
Cross-Entropy
35
64
73
12
9
6
KD
30
67
31
57
DiSK
local minima. We show these minima and the contour plot for the student in Figure 1 We present the results of training student models with different initializations in Table
KD suffers from bad local minima. KD loss landscape contains many local minima (see Figure Due to a big gap between student and teacher capacity, it is unable to help the student discern between these minima. Hence, KD fails to distinguish between the different minima (see Table
DiSK censors interval [2, 3] and in addition focuses training on learnable datapoints. If we analyze the guide function at the end of the training, see that it covers (censors) the first Blue cluster. Indeed, both clusters are not simultaneously learnable with the available student capacity. Once we censor the interval [2, 3). then the problem becomes realizable for the student model. The guide function thus captures the excess capacity of the data.
DiSK smoothens loss landscape. The guide function and the budget constraint enable method to have smooth loss landscape thanks to the guide-function censoring points, which eliminate the local minima. Hence, DiSK solution lands in the global minimum with high probability.
Example (2D Gaussians). Consider another toy dataset with two dimensional features x E R2 and three class labels y E [Red, Green, Blue). Here we wish to show that DiSK can allow for globally optimal solutions reaching 100% accuracy, which appears unachievable with cross-entropy minimization regardless of data size.
Figure 2a shows the labelled data. There are six cluster centers, two with each class label. Data points are drawn using Gaussian balls around the cluster centers with small radii. We sample 1000 i.i.d. data points as the training set and 1000 data points as the test set with equal representation from all three classes. We provide details (hypothesis classes, learning procedure, etc.) in Appx. A.2.
The teacher is 3-layer neural network with 8, 16, and 3 neurons. The student is 2-layer neural network with and neurons. We point out that the teacher being an prer-parameterized network in this feature space, casily learns the correct decision boundary. While the student being severely constrained network suffers in learning the task. Different training runs lead to different local minima. We show teacher solution and student local minima in Figure 2a The contour plots for the student
models under KD loss and DiSK loss are shown in Figure 2b 2c using
Li et al, (2018).
The results similar to the ID example- KD converges poor local minimum with at least 43% of the initializations, while in contrast, DiSK escapes these by focusing on the learnable part of the input space (Fig. 2c), converging the global minimum nearly always (Table
3

---

Published as a conference paper at ICLR 2023
Data
S Min-A (Acc-0.7)
S Min-B (Acc-0.8)
KD Loss Corbour
DESK Lown Centour
T (Acc-1.0)
S Min-C (Acc-0.9)
5 Min-D (Acc-1.0)
0
(a)
(b)
(c)
Figure 2: 2a 2D Gaussians. Data distribution on R². Teacher T learns the correct decision boundary with
3 layer NN and it is the global minima for this whree-way classification task. While student S has many bad local minima, and one global minima that best describes the decision boundary with 2 layer NN. [2b KD training. Loss contour plot shows various local minima the loss landscape. [ DiSK training. Loss contour plot shows the bad local minima no longer exist (wider minima, join two adjust minima, remove bad local minima).
To conceptualize our findings in these examples, let us attempt to intuitively infer the example- censoring, mundsaape-smootheme that appear to be satisfied for these synthetic examples.
and good generalization, by utilizing the following conditions
Realizability. Suppose
in situation where the guide funetion g is sufficiently powerful that there a student and guide function capable of interpolation, i.e., predictions supported by the guide function, o(s, g), interpolates
mimic the labels.
Example: For instance, consider a binary classification problem with the labels E 1J. Let ((8,g) g) y(s + g) with s(x) E [-1, Our realizability condition is that always satisfy ((s(x) + g(x)) > O. As such, this leads to the condition that if ys(x) S 0, then vg(x) > 0.
Therefore, E [o) S E[4 [[] S E[1 [xo(x)>o]]-
Small Guide Function Capacity. In addition to realizability suppose the class of guide functions g E has a small capacity (for instance, small VC dimension). For our case this condition is satisfied because our guidance function is obtained by using an MLP on teacher's last layer features.
Example: Continuing with the example above, say we now have m training instances, (x;, Mi), i E [m]) ğ(x) is guide function output of DiSK. We can infer by standard statistical learning re- sults (Shalev-Shwartz & Ben- DaviD 2014) that, for the estimated function I E G. follows
[VC(G)+1oR
with probability greater than
that [[][))())=00] < EE [v:0(x))20) + V m
As a result, can say that if there is student, s(x) (not necessarily that output by DiSK), which complements ğ(x) and satisfies realizability, then with probability greater than m (V VC[C)+log
[o] 1 E. [v.0(x))00) +
Remarks. The key point is that the student capacity is considerably larger since we typically train an entire DNN, and student complexity-base bound can be vacuous. While the guidance function does bound the student generalization error in terms of guide function complexity, there strong caveats- we require the strong assumption of realizability on the entire domain, and additionally, while the guide function witness student error, are not in a position to precisely estimate it without additional training data. Furthermore, the RHS is a relaxed bound on the student training error. This motivates having budget constraint ensure that student learns with small training error.
DEFINITIONS AND FORMULATIONS
Notation. Let X' and y (1,..., .....,,) C} be the feature and label spaces respectively, focusing on C- class classification task. We assume that we have a training set of N i.i.d. data points D = {xi, 1 where x; E and y4 y. We use symbols S and to denote the student and teacher models respectively. Let Ig ls(x) RD and lx(x) R be the score vector, logits, predicted by S and T
4

---

Published as a conference paper at ICLR 2023
on imput We use as the temperature used to soften the probability distribution. We write the resulting softened student and teacher probabilities as s (x) and t² (x)x
s" s²(x) softmax (Is(x))
(Ir(x) 7
t=(x) softmax
The standard prediction probabilities correspond s (x) and (x). We will use s denote the yth coordinate in s" (x), and similarly (x). The hard prediction of the student ps(x) = arg my s (x), and similarly pr(x) = arg maxyey (x) for the teacher.
use g(x [0, 1] to denote the helper guide function for the student and teacher pair (S, T). Guide takes input and any other feature processed by (S, T') pair and decides whether not the student needs help on the input x. Finally, we define ReLU activation as (-)+ max(0,-).
3.1 VANILLA KNOWLEDGE DISTILLATION
KD relaxes the 0-1 error between the student predictions and the true labels y using the cross-entropy loss LCg. Then, KD denotes the distance between the student and teacher softened probability distributions using the KL divergence. We summarize the corresponding losses as,
N
1
N
(x;,) log ();)) tz(x))
LCE(S) = N i=1
log (x:); (s)
For hyperparameters [0, 1], 0, KD minimizes a mixture of the above losses, as shown below LKD aLce(s) a)Ckr(s)..
3.2 SELECTIVE KNOWLEDGE DISTILLATION.
KD attempts to transfer the knowledge from the teacher to the student on all training data points, which is sub-optimal objective when there is capacity mismatch between the student and the teacher. Instead, we propose distilling selective knowledge (DiSK) to allow the student to selectively ignore some hard-to-learn data points during training, transferring the teacher's knowledge only on easy-to-learn inputs, and matching the learning to student capacity. Our objective is to minimize
N
N
min 8.g.5
distance(t((
subject to.
(v. max, sy (x.)) S 8
(2)
N
Distance between and with help of
Support budget constraint
where é interpolates student predictions based on the guide's help. The divergence term helps in minimizing the distributional distance between the teacher and student probabilities after the guide included. While the budget term in the optimization constrains the helper provide help only when necessary, the amount of help given to the student should be within the budget ó E [0, 1].
Function g Construction.
previously stated, we use the teacher's last layer features and soft predictions input to the guide g. The guide is structured as weight three-layer neural network with these inputs, with a sigmoid activation at the last layer. We re-emphasise that is not used at inference time, and only aids training. More details are left to Appx A.4
Relaxed Losses, Lagrangian Optimization Algorithm. We relax Eq. 2 and construct La- grangian by integrating the constraint into the minimization.
Budget constraint relaxation. We relax the indicator loss the budget to a cross-entropy, and treat hyperparameter to get
Coudger (8,
Σ g(x;) log sy (xi) -
N
We view the scaffold as way for the student to interpolate the uncensored data. It suggests that good initialization for the budget is the error of cross- cooss-entropy trained model when the student does not have the teacher supervision. Thus, sean the budget in small interval around this initialization.
5

---

Published as a conference paper at ICLR 2023
Algorithm DiSK: Distilling Selective Knowledge.
1: Input: Training data D = {(x,, Vi) vi)}i=1r Teacher t,
2: Parameters: т, K, Amin. Amax: Number of iterations R. Xr cosine period, Budget 6,
3: Initialize: s, randomly initialize g.
Amin.
4: for r to R do
5: Randomly Shuffle Dataset D
6: g 4 arg ming Disk (8.9. U, (,)
7: S t min, aLCE(s) (1
dist (s. L)
À f Amnn + (Amax Amin) (1-008
AI
8:
9: :
Distillation objective. Motivated from KL loss, we construct distance loss with guide function as, dist (s, T.D
(x;) log (
(5 (s) + 1 y (x)))((x;)
We point out two modifications in the distillation loss. First, Caist explicitly adds guide value to softened student probabilities in selected class indices. The class indices guide funetion adds value are picked as top K classes based on the teacher probabilities for any input x; where is a hyperparameter of our method. The rest of the class indices do not get any value from g. Second, we use different temperature parameters for teacher and student. Temperature parameter for teacher, т, is hyperparameeer. The student temperature found by minimizing the KL loss between teacher softened probabilities and the student softened probabilities over the training dataset, .ic Tt.,DD = arg min,' E; K Ltt (x;), s (xx)).
Similar KD, we incorporate standard cross entropy loss between student model predictions and the ground truth labels for stability. We construct our Lagrangian by combining Eq. 3 and 4 as,
DiSK (s, g, A) (s) + (1 C
K (s, g) + XC XSuudget
(5)
where a is saper--ar and is the dual parameter of DiSK.
We optimize Obj. Jusing a primal dual update scheme as explained in Algorithm
Primal Parameter Updates (s, g). We learn the student and the guide function g using alternating minimization. We approximate arg min with running SGD for a small number epochs on the full dataset. In each iteration, we first learn the guide function g to select the data partition from which the knowledge needs to be distilled. Next, given the function we learn the student using the help We empirically found that not optimizing the student model on budget loss gives more stable results. Hence, we minimize the student model only on the distillation and cross-entropy losses.
Dual Parameter Update (A) Intuition. Although it is tempting optimize the above via dual ascent and primal descent scheme (wherein the dual parameter À is increased by residual term in the con- straint until constraint satisfaction), recent work, Sun & Sun (2021). has proposed to decrease the in the non-convex regime. Inspired by this, we update the dual parameter by fixed schedule between [Amin Amax]. Amin encourages exploration and allows student model distill knowledge from all points. On the other hand, Amax enforces the constraint and forces the student model to learn on uncen- sored inputs. We choose
4AT, so that the algorithm is exposed to few exploratory periods. For the final period, we increase À monotonically so that budget is more strictly enforced at termination.
Computational Efficiency. Algorithm | trains both student and guide networks. The guide network being small (three-layer MLP) relative to the student (CNN model), the additional cost in training the guide relatively insignificant, and as such DiSK efficiency is similar to KDD
EXPERIMENTS
We evaluate DiSK in various capacity mismatch scenarios on benchmark datasets. We avail our code at https:// qi con/an. Ikagak2/Disk http:///ttab..cmmmmmmaaaaaaa////
Distilling Scaffoldes Knowledge
Datasets. We use publicly available CIFAR- 100 (Krizhessky 2009). Tiny-Imagent (Le & Yang 2015) datasets. CIFAR-100 contains 50K training and 10K test images from 100 classes with size

---

Published as a conference paper at ICLR 2023
Table 2: Model Statistics. We compute the storage (number of parameters) and computational requirements (number of multiply-addition operations) of the models used this work.
CIFAR- 100
Tiny-Imagenet
CIFAR- 100
Architecture
Architecture
MACs
Params
MACs
Params
MACS
Params
ResNerlO-E
64M
1.25M
255M
1.28M
ResNet10
253M
4.92M
1013M
5M
ResNet32x4
1083M 327M
7.4M
Teacher
ResNet18
555M
11.22M
2221M
11.27M
Wide-ResNet-4022
2.25M
ResNet34
1159M
21.32M
4637M
21,38M
ResNet10-xxs
2M
13K
8M
15K
ResNet8x4
177M
1.2M
ResNet1O-xs
3M
28K
12M
31K
ShuffleNetV2
44.5M
1.4M
Student
ResNetlO-s
4M
84K
16M
90K
Wide-Res Made-ResNet-1622
101M
700K
ResNet10-m
16M
320K
64M
333K
Wide-ResNet-40-1
83M
S70K
MobileNetV2x2
22M
2.4M
32 32 x 3. While Tiny-Imagenet contains 100K training and 10K test images from 200 classes with size 64 x 64 x 3. We provide the dataset setup and data augmentations used in detail in Appx. A.3
Models. We evaluate standard convolutional models on these datasets including ResNet(He et al. 2016). Wide-ResNede ResrrutZaorruykk & Komodakis 2016). PobileNet((andlle) et al.) 2018)) and Shuf- fleNet al., 2018). Sshos the storage and computational requirements of all the models used in this work. We provide explicit model configurations in Appx. A.4 including the tiny models we generate from the ResNet architectures.
Methods.
We study performance against standard cross-entropy (CE) based learning and the vanilla KD methods. For each method, we train models for 200 epochs using SGD as the optimizer with 0.9 momentum and 0.1 learning rate. See Appx. A.5 for more training details. We have recorded the mean in our results as the variance of 3 trials in our experiments is not larger than 0.1 in most cases.
We perform evaluations in different settings. Below, explain individual setups. We cover more ablative experiments in Appx. due to page limit.
Large Capacity Mismatch Setting. We distill knowledge from teacher model into a student model where the student has much less capacity compared to the teacher model. We use four large capacity ResNet teachers and five tiny ResNet students and train these students using CE, KD, and DiSK methods. Performances, and the gains of DiSK are reported in Table 3
Small Capacity Mismatch Setting, While DiSK has been designed for the scenario when student capacity is very low, we further evaluate in the setting where teacher and student capacities are similar, to probe how far the power of the method extends. The model classes used are the standard choice for this scenario (Chen et al. 2022 Tung & Mori 2019). Performance is reported in Table 4
Table 4 further reports the results of the feature matching distillation methods: FitNets (Romero et al. 2015), SemCKD (Chen et al. (021), and SimKD (Chen et al. (022). Such methods can often outperform response matching KD on large students, due to student representations that are more aligned with the teacher, but typically at an increased training cost. While feature matching methods are not the main focus of our work (and in principle scaffolding idea can be extended them), we observe that DiSK often improves upon their performance without any direct feature matching.
Experiment results. Below, we highlight salient features of DiSK based on empirical data.
DISK outperforms the baselines uniformly across all datasets and student sizes. As shown by Table 3 DiSK significantly improves the student performance in CIFAR- and the (more challenging) Tiny-Imagens dataset, respectively showing accuracy gains of up to 5% and 2% compared to KD. These gains are consistent across wide range of student and teacher capacities.
DiSK achieves better performance with worse teachers than KD does with even the best teachers. In Table 3 we point out that the student performance increases for KD as the teacher complexity is increased for given student. But note that for the same student, DiSK achieves much better performance with even worse teacher. For instance, for the 'ResNet1O-m' student, KD accuracy increases from 66.96% to 68.09% by using high capacity teachers. But 1ResNet trained with even the worst teacher ( ''ReNet10-E"" achieves 70.03% accuracy. This saves a lot of resources in any application large teacher requires more training time, and larger compute resources.
DiSK is competitive even in small capacity difference setting. As shown by Table 4 DiSK does not loose its competitive edge over the KD even when the student is relatively similar sized as
7

---

Published as a conference paper at ICLR 2023
Table 3: DiSK performance under large capacity mismatch CIFAR- 100 & Kiny-lmagenet We draw mismatched teachers and students from the ResNet family, and report accuracy of CE trained teachers and students, performance of students distilled using KD and DiSK, and gains of the latter relative to KDD.
C-FAR-100
Tiny-Imagenet
Architecture
Accuracy (%)
Accuracy (%)
Teacher
Student
Teacher
CF
KD
DISK
Gain
Teacher
CE
KD
DISK
Gain
ResNetTO-xxs
$2.05
32.64
37.56
4.92
17.44
17.59
18.62
1.03
ResNet10-E
71.99
52.16
54,92
58.14
3.22
52.14
34.65
35.77
37.43
1.66
ResNetl0-m
65.24
66.96
70.03
3.07
44.74
46.01
48.03
2.02
ResNet10-xAs
32.05
34.25
37.84
3,59
17.44
17.96
18.55
0.59
ResNet10
ResNet1O-s
75.25
52.16
54,95
58.36
3.41
56.04
34.65
36.11
37.37
1.26
ResNetlO-m
65.24
67.27
70.15
2.88
44.74
46,08
48.19
2.11
ResNet10-xxs
32.05
34.16
37.8
3.64
17.44
17.47
18,53
1.06
ResNet18
ResNetlO-s
76.56
52.16
55.76
58.11
2.35
62.48
34.65
35,59
37.5
1.91
ResNet10-m
65.24
68.09
69.86
1.77
44.74
45.91
47.7
1.79
ResNet1O-xxs
32.05
33.93
37.78
3.85
17.44
17.67
18.91
1.24
ResNer34
ResNet10-s
80.46
52.16
54.19
58.02
3.83
63.06
34.65
35.43
37.68
2.25
ResNet10-m
65.24
66.78
69.89
3.11
44.74
45.89
47.6
1.71
Table 4: DiSK performance with small capacity mismatch on CIFAR-100. We pick standard student and teacher
configurations used in the KD literature, and report accuracies and gains similarly to Table 3
Feature matching
KD baselines are due to Chen et al. (0022).
CIFAR-100
Architecture
Response Matching KD
Feature Matching KD
Accuracy (%)
Accuracy (%)
Teacher
Student
Teacher
CE
DiSK
Gain
FitNet
SemCKD
SimKD
ResNer8x4
73.89
76.25
76.92
0.67
74.32
76.23
78.08
ShuffleNetV2
73,74
79.13
80.23
1.1
75.82
77.62
78.39
ResNet32xd
81.45
Wide-ResNet-16-2
74.26
76.28
77.67
1.39
74.70
75.65
77.17
MobilcXetV2x2
69.24
76.05
77.24
1.19
73.09
73.98
75,43
ResNer8x4
73.89
75.15
76.05
0.9
75.02
75.85
76.75
ShuffleNetV2
73.74
75.81
78.33
2.52
Wide-ResNet-40-22
78.41
Wide-ResNet-40-1
72.81
74.44
75.92
1.48
74.17
74.4
75.56
MobileNetV2x2
69.24
73.92
76.32
2.40
*SimKD accuracy not emphasized as it employs additional layers beyond the given student architecture and thus not directly comparable to other methods.
teachers, and shows gains of up to 2.5% relative to KD. We conjecture that the observed gains arise from the fact that DiSK provides scaffolding for hard points to the student in initial training stages, which promotes the student to learn easy examples first. As training progresses, DiSK removes the discounted help from hard inputs. As a result, the student evolves from simpler hypothesis to the ones consistent with both easy and hard inputs. This justifies our dual parameter (A) update in Algorithm T wherein we periodically increase and decrease À to enforce and relax the budget constraint.
DiSK students achieve near teacher accuracy while saving up 8x MACs & 5x Params. As reported in Table 3 student "ResNet1O-m' trained with the teacher "ReeNNtl0-E") achieves close to the teacher accuracy of 71.99%. In this process, it saves 4x compute and requires 4x less parameters. Similarly, student ("ShuffleeetV2") trained with the teacher ("ResNet32x4)) achieves close to the teacher accuracy of 11.45. In this process, it saves 24x compute and requires 5x less parameters.
DiSK cleverly selects a subset of datapoints and smoothens the loss landscape. As illustrated in Figure and 2 DiSK judiciously selects subset of hard-to-leamn data points for the students and provide discounted help to the student focus on easily learnable inputs. As result, it eliminates some bad local minima in the student landscape, and smoothens tihs surface.
DiSK enables the student to reach saturation capacity. In Table 3 the performance of KD often suffers as the teacher size is increased, e.g., student ("ResNet10-s' accuracy decreases substantially with the teacher 'ResNet34" versus the teacher 'ResNer18.. In contrast, DiSK saturates the student performance across different teachers. For instance, student ("ResNet1O-m') accuracy is R 70% for all the teachers. Thus, we point out that DiSK enables the student to reach saturation. This may be due to the fact that guide g identifies the same set of 'easy' points across different teachers.
5 RELATED WORK
We refer the reader to Gou al. (021) for comprehensive survey on knowledge distillation.
8

---

Published as a conference paper at ICLR 2023
Response Matching, Zeng & Martinez 2000); Bucila et al. (2006) distill the response of ensemble of classifiers into a single neural network by creating a pseudo-labeled dataset using the ensemble of classifiers. Ba & Caruana (014)) extend this to the setting with the neural network the teacher. Hinton et al. (0155) propose vanilla KD that distills knowledge from an ensemble of neural networks into single network by matching their output logits. This work provided simple recipe for aligning the teacher and student predictive distributions using the Kullback-Leibler (KL) divergence. Recently, Beyer et al. (202T) modify the KD procedure include patient and consistent teacher resulting in substantial gains. Knowledge consistency is enforced by using the same aggressive data augmentation and image views in the student as in the teacher. Patience is promoted using a very long training schedule. This results in computationally very expensive training process.
Stanton et al. 222T)) analyze response matching KD and suggests that difficulty in optimization leads to poor knowledge distillation. Thus, the teacher and student predictions do not always match, even on the training data. Cho Hariharan (2019) study vanilla KD through the lens of mismatched student and teacher capacities. They show that small students are unable to mimic complex teachers. They propose early stopping teacher training as to remedy to achieve a student-learnabble teacher. The above works fail miserably when the gap between student and teacher complexities is large. Specifically, the student cannot learn the complex teacher decision boundaries primarily due to the small student capacity. It becomes imperative selectively choose only easy-to-learn data points and transfer the teacher knowledge from these points and ignore the hard-to-learn data points during distillation. Thus, our proposal targets the problem of severe capacity gaps between student and teacher models. Additionally, our experiments with standard student and teacher configurations show that DiSK is still competitive when the capacity difference small.
Feature Matching, In response matching, teacher supervision is limited to its logits. We can enforce intermediate layer feature matching for refined teacher supervision. FitNets (Romero et al., 2015) extend the KD by including the feature matching in the middle layers. Zagoruyko & Komodakis (2017) use feature map attention as the teacher supervision. TTnng & Mori preserve pairwise similarity in feature maps amongst data points during the distillation. [Chen et al. (2022) modify the student by projecting the student features onto the teacher feature space and by reusing the teacher classifier. While our work focuses primarily on selective distillation in vanilla KD for simplicity. We can easily extend the proposed framework to incorporate it into feature- matching distillation.
Curriculum Learning & Hard Instance Mining. Curriculum Learning (CL) (Bengio et al. 2009
Hacohen & Weinshall 2019; Graves et al., 2017) sorts the data based on their hardness as measured by some scoring function ex., predictive entropy, softmax margin score, etc. ). It presents the data points during training in the order of increasing hardness. Similarly, Hard Instance Mining (HIM) (Zhou et al. 2020) reduces the weight of the easy example and increases the weight on hard inputs to promote hard-example learning. We point out that our method is only conceptually related to these works via input hardness. Our method helps the student with hard examples by providing explicitly discounted help g. We learn the helper function (through teacher representation) that decides whether the student needs help on given input. Thus, we do not prioritize learning hard examples keeping in mind the fact that student capacity is much smaller than the teacher.
6 CONCLUSION
We develop a new knowledge distillation method that utilizes teacher predictions in novel ways combining predictive targets with scaffolding the student hard-to-learn points by means of a guide function. Our method is particularly relevant when there is large gap between student and teacher capacities. We show that our method allows for convergence better minima based on two key properties. Our guide function allows for censoring hard-to-learn examples, and the predictive targets set by the teacher on remaining points allow for eliminating bad local minima and smoothening the resulting student loss landscape. Against vanilla KD, we achieve improved performance and are comparable to more intrusive techniques that leverage feature matching on benchmark datasets.
ACKNOWLEDGMENNSS
This research supported by Army Research Office Grant W91 the National Science Foundation grants 2CF-2007350 and CCF-1955981, AFRL Contract no. FA8650-22-C-1000 and a gift from ARM Corporation.

---

Published as a conference paper at ICLR 2023
REFERENCES
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In
Proceedings of the 26th Annual International Conference Machine Learnin.- ICML '09, pp. 41-48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 978 178160558116.. doi: 10.11445/553374415533388 /tt//ttpa:://oi.orr///////////// 1ttp:///...1111115510 145/ 1153380
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent, 2021. URL htt.tttss:/// https://sss..iit
org/abs/2105.0m 05237
Cristian Bucila, Rich Caruana, and Alexandru Niculcscu- Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '06, pp. 535-541, New York, NY, USA, 2006. Association for Computing Machinery. doi: 1150402. 1150464
ISBN 1595933395.
15445/115440.111555 150464..
Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang. Yan Feng, and Chun Chen. Cross- layer distillation with semantic calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7028-7036, 2021.
Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang. Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11933-11942, 2022.
Jang Hyun Cho and Bharath Hariharan. On the efficacy knowledge distillation. 2019 IEEE/CVF International Conference Computer Vision (ICCV), pp. 2019.00489.
2019. doi: 10.1 109/CCV.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Terrance De and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: /708. 04552, 2017.
Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, (786)):18--188111 mar 2021. doi: 10.1007/ sl 1116--022-01445 URL https: /tattps://doi..om bttpttts::/doii.
Alex Graves, Marc G. Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. Automated curriculum leaming for neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp- 1311-1320. PMIR, 06-11 Aug 2017. URL m1 wtt.ppes/////gggep graeesravvsss 7a html
Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 200525355555 2544. PMLR, 09-15 Jun 2019. URL http///pooeeednneeeeeee ml r press/v977 hacoheni 9a. html
K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 00.1009CVPR.2206.0000
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015.
htR.:t:::: /ttps:///axiiii org/ abs /00...1503..00 02531
10

---

Published as a conference paper at ICLR 2023
Anil Kag and Venkatesh Saligrama. Time adaptive recurrent neural network. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15149-15158, June 2021.
Anil Kag, Ziming Zhang, and Venkatesh Saligrama. Rnns incrementally evolving on an equilibrium manifold: A panacea for vanishing and exploding gradients? In International Conference on Learn- ing Representations, 2020. URL htttt:::
net 2or
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge, 2015.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Neural Information Processing Systems, 2018.
David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privileged information. In ICLR (Poster), 2016. URL
03643
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018
Adriana Romero, Samira Ebrahimi Kahou, Polytechnique Montréal, Bengio, Université De Montréal, Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitmets: Hints for thin deep nets. in International Conference on Learning Representations (ICLR, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-222 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Shai Shalev-Shwarzz and Shai Ben-David. Understanding Machine Learning From Theory to Algorithms. Cambridge University Press, 2014.
Samuel Don Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew Gordon Wilson. Does knowledge distillation really work? In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL net pettps://openreet www.:e/iwwweeeeetummneum. Foo.mmiid--a. EKoXiReA
Kaizhao Sun and Andy Sun. Dual descent alm and admm, 2021. URL https::/arxiv..rr
abs/2109. 13214
Tung and G. Mori. Similarity-preservii knowledge distillation. In 2019 IEEE/CVF Inter- national Conference Computer Vision (ICCV), pp. 1365-1374, Los Alamitos, CA, USA, nov 2019. IEEE Computer Society. doi: 10. 10.110091CCC22219.000444 URL https://doi
Teeecomputersocieyyyooo wttp:////................ 1010////0..1000/1/..///.........
00145
Vladimir Vapnik and Rauf Izmailov. Learning using privileged information: Similarity control and knowledge transfer. Journal of Machine Learning Research, 16(61):2023-2049, 2015. URL
http:://mlr woroor//ppa.... wwgppapers...peeeeoeooeeeeee /vvvvvl. wwvl/vvvppp http/////aaiil...ttt...
Ross Wightman. Pytorch image models.
Lytorch--iage-modee 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017. URL
https::/erxiv..r http:/awwwwii..rra////aaaaa/a/// 03928

---

Published as a conference paper at ICLR 2023
Xinchuan Zeng and Tony R. Martinez. Using neural networks to approximate ensemble of classifiers. In Neural Processing Letters, pp. 2000, 2000.
Tianyi Zhou, Shengjie Wang, and Jeffrey Bilmes. Curriculum learning by dynamic instance hardness. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 8602-8613, Curran Asso- ciates, Inc., 2020. URL http:///eee.popeeeeee.eee./eeeea 6GC0ee550l6667 33322277777777700
file/
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 DETAILS FOR ILLUSTRATIVE EXAMPLE (ID INTERVALS)
Dataset Overview. We generate synthetic toy dataset with one dimensional features x [0, and binary class labels y E {Red, Blue). We use (I) to denote the sigmoid function with a scaled by parameter R > 0, ie.. (x) 1-exp( (x))
Function Classes. Let H be the 1-interval function class parametrized by two variables {a, b), i.e. for h E H
M(2; a, b) = a(z a) o(x - b); 0 < a <<< Similarly, let FF be the 2-interval family parametrized by four variables {a, b, c, d), ie., for f E
ie.,
[2; a, b, d) h(a; a, b) + e,);;,d); 0<a < b d <9
Note that any function in H. behaves as an indicator for the interval (a, b). Similarly, any function in F behaves an indicator for two exclusive intervals { {a, b), (c. d)).
Data Generation. We assume that the data is generated using the function f E F with parameters (a" b" ee", d²). Dataset is sampled with balanced data from both classes. We label x red if (x) < 0.5, otherwise we label the point as blue. We sample 1000 i.i.d. data points as the training set and 100 data points as the test set. Figure D shows the train data. We draw an independent validation set of 100 data points for hyper-parameter tuning.
Large Capacity Teacher T belongs the 2-interval function class F and is learnt with all the training data points. We learn the teacher with cross-entropy loss. We use the SGD optimizer with momentum 0.9, learning rate 0.1, weight decay 0.01, and minimize the loss for 200 epochs. Note that the teacher recovers the underlying function f as shown by the two intervals in Figure []
Capacity Constrained Student S belongs to the 1-interval function class H and it has access to all the training dataset. Note that best possible hypothesis in H. cannot recover the performance of the function f and hence the student will have settle on one of the many local minima. We show these minima as well the contour plot for the student in the Figure [ We learn the student with three different loss functions ( cross-entropy LCE. vanilla KD CRD similar training setup as the teacher in terms of the optimizer and training steps. For DiSK method, our guide function g has similar capacity as the student but utilizes the teacher features to learn the decision as to which points are hard-to-learn for the student. For both, KD and DiSK, we scan the apper-parameter over the range {0.0, 0.1, 0.5, 0.9, 1.0]. Similarly, we scan the temperature T in the range {1,2,
(s) and DiSK Algorithm I). We use
in the following ranges: (a) T2 E (1,2,4), (b) K E 31, (c) Amin {0.01, 0.1, 5, 10), (d) Amax E (1, 5, 10, 20, 50, 100, 1000), (e) Budget E (0.1,0.05 0.0). and (f) Ar E (20,50). We replace the arg min Algorithm with three gradient steps.
For DiSK, we scan the different
Note that although the hyper-parameter scan looks daunting, the default hyper-parameters: T, (teacher temperature). K = 2, Amin 0.1, Amax 50, 0.0 (approximate error of the global minima), XT = 50, work well in this setup as well as the 2D gaussian example described below.
Vanilla KD suffers from local minima. The loss landscape of the Vanilla KD contains many local minima (see Figure [b)). Since there is big gap between student and teacher capacity, the teacher unable to help the student discem between these bad minima. Hence, Vanilla KD leads to one of the bad local minima with high probability (see Table
DiSK removes bad local minima. In contrast, DiSK deletes harder points from the landscape and as a result settles onto the global minima for S with high probability (see Figure Tec) and Table [ where one cluster of blue points have been removed). Note that this also removes bad minima from the loss landscape. Finally, DiSK learns the student that has the best performance.
A.2 DETAILS FOR ILLUSTRATIVE EXAMPLE (2D GAUSSIANS)
Dataset Overview & Data Generation. We generate a synthetic toy dataset with two dimensional features E R and three class labels y E (Red, Green, Blue]. We generate six cluster centers. We
13

---

Published as a conference paper at ICLR 2023
assign color to each cluster center and spread input features around these centers. Below we list the cluster centers along with their class labels.
(0, 0), Red
(1.5, 0), Blue
(3, 0). Green
(0, 1.5), Blue
(1.5, 1.5), Green
(3, 1.5), Red
Given cluster center draw input features x using Gaussian ball with radius r 0.05 around center using multi-variate Gaussian (c, rT]), where I the Identity matrix.
Figure 2a shows the labelled data. We sample 1000 i.i.d. data points as the training set and 1000 data points as the test set with equal representation from all three classes.
Function Classes. We use two forwafo neural networks as function classes in this example. Let <(.) denote the Batch-Norm followed by ReLU operation.
Let H be the two feed-forward layer neural network. Any h E H can be written as
h(z) = W20(W1
where W1 E R and W2 E 33x2 Note that h has only two neurons and hence a very small network.
Let F the three feed forward layer neural network. Any f E can be written as
((x) (a
where W1 R8x2 W₂ IR 16x8 and W3 3x32
Note that ff has 8 neurons in first and 16 neurons in the second layer. The final layer in above networks the classifier layer that transforms the features into the class probabilities.
Large Capacity Teacher T' is a layer neural network with 8, 16 and 3 neurons. In between each feed-forward layer, we have batch-norm and ReLU activation non-linearity. We point out that the teacher being an over parameterized network in this feature space, easily learns the correct decision boundary. We show this decision boundary in the Figure 2a We learn the teacher with cross-entropy loss. We use the SGD optimizer with momentum 0.9, learning rate 0.1, weight decay 0.01, and minimize the loss for 200 epochs.
Capacity Constrained Student S is a layer neural network with 2 and 3 neurons. Similar to the teacher, we have batch-norm and ReLU non-linearity in between the feed-forward layers. Since the student is severely constrained compared to the teacher, it suffers in learning the task. Different training rums lead to some popular local minima. We show the teacher solution as well as the student local minima in Figure 2a For DiSK method, student but utilizes the teacher features to learn the decision as to which points are hard-to-learn for the student. The contour plots for the student models under KD loss and DiSK loss shown in Figure using the visualization toolkit described in Li ct al. 2018). We following similar setup for syper-parameter tuning in Sec. A.I
guide function g has similar capacity the
We see a similar result as in ID example. KD suffers from bad local minima and converges to the global minima with only 43% of the initializations. Differently, DiSK escapes the local minima solutions and focus on the learnable part of the input space as shown in Figure 2c Our method converges the global minima with very high probability see Table Tp.
A.3 DATASET DETAILS
We publicly available CIFAR- 100 (Krizhessky 2009),
(Le & Yang 2015)) and ImageNet- IK (Russakovsky et al. 2015) datasets. CIFAR- 100 contains 50K training and 10K test images from 100 classes with size 32 x 32 x While Tmay-nt contains 100K training and
14

---

Published as a conference paper at ICLR 2023
10K test images from 200 classes with size 64 x 64 X 3. Imagenet contains 1.2M training and 100K test images from 1000 classes with size 224 x 224 x 3.
For the CIFAR-100 and Tiny- Ting-emaeenet datasets, 'RandomCrop"" "RandomHooizontllFiip 'AutoAugment' (Cubuk et al. 2019). 'Cutout" (De & Taaloor 2017), and Mean-SSd-W Add-Normaizzation' We use the same augmentation strategy in all our experiments, across baselines and different model families.
the standard data augmentations including
For the Imagenet- IK dataset, following the previous work Chen et al. (2022), use the 'Random- Crop', "RandomHHorizontlFFiip and Alean-Std-NNrmaliinn
A.4 MODEL DETAILS
In this section, we list the model characteristics as well as their accuracy obtained using standard cross-entropy (CE) loss. Table lists all the models used in large capacity mismatch setting. While Table 6 lists all the models in the small capacity mismatch setting. Below, we describe individual model for completeness.
Large Student-Teacher Capacity Mismatch All models in the Table belong to the same ResNet family and the standard 'BasicBlock' the building block. It consists of a convolutional block, followed by four residual block stages, followed by the adaptive average pooling layer and the classifier layer. Different capacity models in this family differ only in the number of repetitions of the residual block and the number of filters in each stage. Below, we write the different of repetitions and the number of filters for the four different residual stages.
ResNer34 has (64, 128, 256, 512)
filters and repeats the 'BasicBlock' [3,4, 6, times.
ResNer18 has (64, 128, 256, 512]
filters and repeats the 'BasicBlock' 22, times.
ResNer10 has (64, 128, 256. 512] filters and repeats the "BasicBlock" 1, times.
R0- has [32, 64, 128, 256
filters and repeats the 'BasicBlock' 1,, times.
ResNerlO-m has [16, 32, 64, 128]
filters and repeats the 'BasicBlock" 1, times.
ResNerlO-s has [8, 16, 32, 64] filters and repeats the 'BasicBlock' [1,1,1,1] times.
ResNerlO-xs has [8, 16, 16, 32) filters and repeats the 'BasicBlock" [1, 1, times.
ResNerlO-xxs has [8, 8, 16, 16) filters and repeats the 'BasicBlock' [1, times.
Small Student- Teacher Capacity Mismatch Definitions of all models in the Table borrowed
from Chen et al. (202)) We refer the reader to their official github repository (https: /iithbb com/ De f FangChen/ SimKD.. qit) for the exact definition. We trained these models using the data augmentations mentioned above and found that our cross-entropy baseline as well as the vanilla KD baselines are much better than the reported in their work.
end
Guide Function Our guide function g is a three layer feed-forward network. uses the last layer features and logits of the teacher as the input. has 64, 128, and neurons in the three layers. We include batch-norm followed by ReLU non-linearity between these layers. The final layer contains sigmoid activation to contain the scaler output in the range [0, 1].
Warm Start We note that we warm start each student model by first training them with cross entropy loss without teacher. We observe that the warm start benefits both DiSK and KD. Note that, do not change the algorithms. We only start from a CE pre-trained student model.
Table 5: Models used in large capacity mismatch setting along with storage and computational requirements.
CIFAR-100 MACS
Tiny- Finy-Imagenet
Architecture
CE Acc.
Params
CE Acc.
MACS
Params
ResNet1(-F
71.99
64M
1.25M
52.14
255M
1.28M
ResNet10
75.25
253M
4.92M
56.04
1013M
5M
Teacher
ResNet18
76.56
5S5M
11.22M
62.48
2221M
11.27M
ResNer34
80.46
1159M
21.32M
63.06
4637M
21,38M
ResNet10-xxs
32.05
2M
13K
17.44
8M
15K
ResNet1O-xs
42.99
3M
28K
25.89
12M
31K
Student
ResNer10-s
52.16
4M
84K
34.65
16M
90K
ResNet10-m
65.21
16M
320K
41.74
64M
333K
15

---

Published as a conference paper at ICLR 2023
Table 6: Models used in in small capacity mismatch setting along with storage and computational requirements.
CIFAR-100
Architecture
CE Acc.
MACs
Params
ResNet32x4
81.45
1083M
7.4M
Teacher
Wide-ResNet-40-2
78.41
327M
2.25M
ResNet8x4
73.89
177M
1.2M
ShuffleNetV2
73.74
44.5M
1.4M
Wide-ResNet-1644
74.29
101M
700K
Student
Wide-ResNet-40-1
72.81
83M
570K
MobileNetV22x2
69.24
22M
2.4M
Table Imagenet-IK. We pick student and teacher configurations show that scale DiSK ImageNet dataset with significant improvements in Top-1 accuracy. We borrow model definitions from timm Wightman (2019) repository including the convolutional and transformer vision models.
Teacher
Student
CE
KD
DiSK
ResNet50
ResNet18
69.73
71.29
72.35
ViT-Large (Patch 16, Res. 224)
ViT-Tiny (Patch 16, Res. 224)
75.45
77.86
A.5
For both, KD and DiSK, we scan the a hyper--praaaa paper-parameter over the range 0.1, 0.5, 0.9, 1.0 As per recommendations from previous works works(Chen al. 2022 Cho & Hariharan) 2019 Tung Mori 2019) we use 7 = 4 as the temperature in Eq.
For DiSK, we scan the different hyper-parameters in the following ranges: (a) T, E (1,2, (b) K E (1,3, 3, 5, 10, 20, 50). (c) Amin E {0.01, 1,5, 10), (d) Amax E {1,5, 10, 20, 50. 100, 1000]. (e) Budget 8 within 0.2 distance from the cross-entropy trained student model's error, and (f) ÀT {20, 50}. We replace the arg min in the Algorithm I with three SGD steps over the entire dataset. For our experiments (both KD and DiSK), we use the popular cosine learning rate scheduler for the SGD optimizer 0.1 learning rate, 0.9 momentum and 5e 4 weight decay. We use 200 as the batch size.
Note that the default hyper-parameters: = (teacher temperature), K 20, Amin 0.1, Amax = 50, = approximate error of the global minima (replaced by the cross-entropy error), ÀT 50, work well in most of our experiments.
We point out that the denominator N the budget constraint should be calibrated for the correct numerical implementation. Instead of N
the number of wrong student predictions as the normalizer, i.e., Σ M4 # arg max, sy (2;) as this the term that appears in the Eq. alongside the g term in the budget constraint.
We note that we use similar parameters train with CE in ResNet based models. Improving CE training would improve DiSK and KD as well since both are initialized with the CE trained model.
A.6 DIFFERENT EXPERIMENT SETUPS
Scaling upto ImageNet setting. We show that DiSK scales easily to the large-scale ImageNet- K dataset. We train two configurations with DiSK, namely (a) ResNet18 student and ResNet50 teacher, and (b) ViT-Tiny student and ViT-Large teacher. We borrow these models from the timm (Wightman 2019) library. Table 7 shows the DiSK performance on these two configurations along with the baseline. It clearly shows that DiSK scales well to ImageNet- 1K setup and achieves significant improvements over the baselines.
A.7 RELATED WORK CONTINUED
We continue section of related work here due to the page constraint in the main part.
Privileged Information. Vapnik Izmailov (2015) propose the 'learning under privileged informa- tion' (LUPP) framework wherein a support vector machine is trained using privileged information
16

---

Published as a conference paper at ICLR 2023
Table 8: DiSK performance against feature matching KD 100: Similar setup in Table We
integrate DiSK within SimKD (Chen et al. 2022) (see Appx. A The gains using DiSK over and using SimKD + DiSK over SimKD are reported. Feature matching KD
baselines are due to Chen et al. (022).
Architecture
Response Matching KD
(%)
Fenture Matching KD
Teacher
Student
Teacher
CE
KD
DISK
Gain
FRLNet
SemCKD
SunKD
SimKD DISK
Gain
Wide-ResNet-00-2
Wide-ResNet-401
78.41
72.81
73.89
75.15
74.44
73,92
1.48
74.17
73.85
74.4
75.56
70.75
76.21
0.65
unavailable during the inference stage. Later, Lopez-Pazz et al. (2016) unify LUPI and vanilla KD into generalized distillation, wherein the teacher is learned using the privileged information. Next, the student is trained using the ground truth and the teacher labels. These works rely on privileged information the application domaín and are shown to work on toy setups. Since our guide function, g is available only during training, it can be thought of privileged information from teacher.
A.8 EXTENSIONS BEYOND VANILLA KD.
In this section, we discuss potential extension of our method beyond Vanilla KD to feature based distillation as well as self-supervised distillation.
Feature based KD methods. Let fa and fr denote the features for the student and teacher respectively and denote the operator such that (fa) lies in the same feature space ft- Commonly used feature transfer strategy is to minimize the distance between these two representations via loss function such as mean-squared error as shown below by the loss Ltc.
N (
N Lit-g = N E (L-
Σ Ct== = N
(6)
simple extension of this feature alignment loss to the selective distillation is shown by the loss Lit-g that weighs each data point with the helper function decision. Table shows this feature matching extension for the SimKD(Chen et al. (022) scheme. We leave question of finding better selective distillation losses in feature alignment future work.
Self-Supervision. We can substitute the teacher with a moving average of the student to help guide in the distillation process surrogate that helps in learning the hints).
Other Domains. Although our work focuses mainly on models in the vision domain. We point out that conceptually the function g is applicable models in other domains albeit with some modifications. For instance, we can utilize the censoring mechanism in sequential decision making with recurrent neural networks (Kag et al., 2020 Kag Saligrama| 202I). We already show that DiSK can be applied to the vision transtormers in the [ We leave the application and modifications to DiSK for transformers language domain for future research.
17


Reviews:
[{"review_id": "V_I-kWWW_bg", "reviewer": "Reviewer_sPvE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an approach to distillation where the teacher provides both a predictive target and scaffolds the student's prediction by censoring hard-to-learn examples. In the case where the student has far fewer parameters than the teacher this scaffolding leads to a smoother loss landscape for the student and thus the student encounters fewer local minima. The authors call their approach DiSK (distilling selective knowledge). ", "summary_of_the_review": "The authors provide a novel approach to performing distillation in which scaffolding allows hard to learn examples to not impact the student. The authors note that an implementation would be provided in the final version, and as a reader I desperately wanted it. I believe it would clarify some questions i have in regards to the guide functions implementation.", "strength_and_weaknesses": "The greatest strength of the work is the considerable gain in accuracy compared to cross-entropy or kullback-leibler. It is very impressive for such resource constrained models. Figure 1 and the example provided is excellent to provide an overview of what can be achieved in regards to simplifying the loss landscape. \n\nOne of the weakness in the paper is that it is unclear what the guide function model is. In one part of the paper it is described as \"obtained by using an MLP on softmax outputs of the teacher\" in another place it is described as taking the teacher's last layer features and the prediction probabilities as input.\n\nAnother weakness is that decisions are presented without an explanation. Why relax the guide function to be continuous? Why is the budge constraint necessary?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T1nLRxeIMd", "reviewer": "Reviewer_XDpR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a novel knowledge distillation (KD) method aims at addressing the capacity mismatch between teacher and student models. The authors propose to additionally learn a weight function to mask hard to learn examples, and thus increase the performance of student models. ", "summary_of_the_review": "This paper provides a novel approach for knowledge distillation by distinguish hard to learn samples for students, and proves to provide a student model with smoother landscape, fewer local minima, and thus better generalization errors. I think the idea proposed is novel, and the paper is well motivated with very intuitive examples to present the ideas. It is a good paper for ICLR. ", "strength_and_weaknesses": "Strength\n- The paper is well motivated. I especially like the introduction section (despite being long) which clearly conveys the important ideas with two illustrating examples.\n- The methodology is well formulated. \n- Experiments are comprehensive with detailed discussions. \n\nWeaknesses\n- The selection of budget is an important factor in the whole algorithm, but there seems not to be discussions on how users should choose in practice. \n- The stability and efficiency of the proposed algorithms are not discussed. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8v1ISjCRkt", "reviewer": "Reviewer_DdZ6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for improving knowledge distillation from a high capacity teacher model to a low capacity student model. In addition to training the student to match the teacher's outputs with cross-entropy, the paper proposes learning a 'guide' (or essentially a gating) function from the teacher's output that reduces the weight placed on difficult examples when training the student. Experiments reveal that this smooths the optimization landscape for the student, enabling it to find the best minimum even when it does not have the capacity to represent the ground truth function. In addition to results on toy environments, the experiments examine CIFAR-100 datasets with standard architectures, and show an improvement over vanilla KD and other KD techniques. ", "summary_of_the_review": "Overall, the paper is poorly organized but the results are compelling. I think it will be of interest to the community. ", "strength_and_weaknesses": "A strength of the paper is in the quality of the experiments. Starting with the toy experiments, the authors clearly demonstrate how their approach is valuable in helping a low capacity student find the best minimum point in the loss optimization landscape; Figure 1 convincingly shows how the gated teacher loss function reduces local minima. Beyond the toy experiments, the paper tests on real datasets with standard large resnet models, and shows consistent improvements over prior work. \n\nA weakness of the paper is its organization. The intro extends for almost 4 pages, and includes detailed experiments and proofs that are wholly inappropriate for the intro. The method has not been fully explained, and yet the paper is diving into experiments and theoretical arguments to show that it works. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster