Teacher's pet: understanding and mitigating biases in distillation
Authors: Michal Lukasik
Srinadh Bhojanapalli
Aditya Krishna Menon
Sanjiv Kumar
Venue: ICLR 2022 Submitted (2022)

Abstract: Knowledge distillation is widely used as a means of improving the performance of a relatively simple “student” model using the predictions from a complex “teacher” model.  Several works have shown that distillation significantly boosts the student’s overall performance; however, are these gains uniform across all data sub-groups? In this paper, we show that distillation can harm performance on certain subgroups, e.g., classes with few associated samples, compared to the vanilla student trained using the one-hot labels. We trace this behavior to errors made by the teacher distribution being transferred to and amplified by the student model. To mitigate this problem, we present techniques which soften the teacher influence for subgroups where it is less reliable. Experiments on several image classification benchmarks show that these modifications of distillation maintain boost in overall accuracy, while additionally ensuring improvement in subgroup performance.

Under review as a conference paper at ICLR 2022
TEACHER'S PET: UNDERSTANDING AND MITIGATING BIASES IN DISTILLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Knowledge distillation is widely used as a means of improving the performance of relatively simple "student" model using the predictions from complex "teacher" model. Several works have shown that distillation significantly boosts the student's s overall performance; however, are these gains uniform across all data subgroups? In this paper, we show that distillation can harm performance on certain subgroups, e.g., classes with few associated samples, compared to the vanilla student trained using the one-hot labels. We trace this behaviour to errors made by the teacher distribution being transferred and amplified by the student model. To mitigate this problem, we present techniques which soften the teacher influence for subgroups where is less reliable. Experiments on several image classification benchmarks show that these modifications of distillation maintain boost in overall accuracy, while additionally ensuring improvement in subgroup performance.
INTRODUCTION
Knowledge distillation is a technique for improving the performance of "student" model using the predictions from "teacher" model. At its core, distillation involves replacing the one-hot training labels with the teacher's ' predicted label distribution. Empirically, distillation has proven successful for model compression (Bucilă et al., 2006; Hinton et al., 2015), improving the performance of a fixed model architecture (Anil et al., 2018; Furlanello et al., 2018), and semi-supervised learning (Ra- dosavovic et al., 2018). Theoretically, several works (Lopez-Paz et al., 2016; Mobahi et al., 2020; Tang et al., 2020; Menon et al., 2020; Zhang & Sabuncu, 2020; Ji & Zhu, 2020; Allen-Zhu & Li, 2020; Zhou et al., 2021; Dao et al., 2021) have studied how distillation affects learning. Put together, both strands of work further the understanding of when and why distillation helps.
In this paper, we are similarly motivated to better understand the mechanics of distillation, but pose a slightly different question: does distillation help all data subgroups uniformly? Or, do its overall gains come at the expense of degradation of performance on certain subgroups? To our knowledge, there has been no systematic study (empirical or otherwise) of this question. This consideration is topical given the study of fairness of machine learning algorithms on under-represented subgroups (Hardt al., 2016; Buolamwini & Gebru, 2018; Chzhen et al., 2019; Sagawa et al., 2020a).
Our first finding is that even in standard settings - e.g., on image classification benchmarks such as CIFAR distillation can disproportionately harm performance on subgroups defined by the individual classes (see Figure 1). To discern the source of this behaviour, we ablate the teacher and student architectures ($3.2), dataset complexity ($3.3), label frequencies ($3.4). These point to the potential harms of distillation when the teacher makes confident mispredictions on subgroup. We emphasise that this phenomenon is not simply the teacher's 's worst class accuracy translating to student; instead, distillation amplifies worst class imbalance much more than the teacher.
Having identified a potential limitation of distillation, we present two simple techniques to remedy it. These apply per-subgroup mixing weights between the teacher and one-hot labels, and per-subgroup margins respectively ($4). Intuitively, these limit the influence of teacher predictions on subgroups models poorly. Experiments image classification benchmarks show that these methods typically maintain a boost in overall accuracy, while ensuring a more equitable improvement across subgroups.
In sum, this work provides novel insights into distillation performance, with three main contributions:

---

Under review as a conference paper at ICLR 2022
Subgroup
Accurscy improvmmmnt
Accuracy
Subgroup Avia age Worst
Baodling
Adapeive miking
Basdling
Adoptive mixing
(a) CIFAR-I00-LT.
(b) ImageNet.
Figure I: Illustration of the potential deleterious effects of distillation on data subgroups. train a ResNet-56 teacher CIFAR- R-100-LT, onngtailed version of CIFAR- 100 (Cui et al., 2019; Cao et al.. 2019) where some labels have only a few associated samples, and a ResNet-50 teacher on ImageNet. For each dataset, self-distill to student ResNet of the same depth. On CIFAR-100 LT, as is often observed, distillation helps the overall accuracy over one-hot student training (~2% absolute). However, such gains come at significant classes: on the len rarest classes, distillation harms performance by ~~% Similarly, on ImageNet, distillation harms the average accuracy of the worst-10 classes by ~3%. Our proposed techniques ($4) can roughly preserve the overall accuracy, while boosting subgroup performance.
subgroups defined by the individual
we identify hitherto unexplored issue with distillation, namely, that its improvements in overall accuracy may come the expense of harming accuracy on certain subgroups ($3.1). Such finding is topical given the widespread practical use of generic learning paradigms such as distillation, and the increasing societal applications of learning systems more broadly.
iii) we ablate potential sources for the above phenomenon ($3.2, $3.3, $3.4), and in the process identify certain characteristics of data (e.g., skewed label distributions) where it can manifest. This systematic empirical analysis aims at understanding non-trivial phenomenon, rather than empirically comparing different learning methods.
(iii) propose two simple modifications of distillation that mitigate the above problem, based applying per-subgroup mixing weights and margins (54); these perform well empirically ($5),
While the paper's analysis empirical rather than theoretical, we that our focus is on systematic analysis aiming at understanding non-trivial phenomenon, rather than merely empirical comparison: Section $3 is devoted carefully understanding the extent of, and causes for, the non-uniform gains of distillation. This is in line with works which empirically analyse neural network phenomena, e.g.
Zhang al. (2017); Müller al. (2019); Nakkiran al. (2020); Neyshabur al. (2020).
BACKGROUND AND RELATED WORK
Knowledge distillation. Consider multi-class classification problem instances X and labels = - (1...., L.). Given a training set = {(r,, yn) ) drawn from some distribution P. we seek classifier h: X + that minimises the misclassification error Evvg (h) = P(((() * g) In (xx)), where é loss function such the softmax cross-entropy, which for softmax probabilities Py(x:) exp(fy(x)) is fyy, f())) = log P. (z). One may then classify the sample via h(x) = arg [a (x).
practice, one may learn logits f X - L to minimise p((f) t +
Knowledge distillation (Bucilä al., 2006; Hinton et al., 2015) employs the logits ft: X + R of "teacher" model to train a "student" model. The latter learns logits f= X RR minimise:
1 Σ [11 a) - f(xn) N
Redist (f)
Pi ely. ((,))( 2'E(Z]
(I)
where a E [0, 1]. Here, one converts the teacher logits to probabilities p² : A,. for simplex A. e.g. via softmax transformation pt, (x:) exp(St, f (z:)). The second term smooths the student labels
2

---

Under review as a conference paper at ICLR 2022
based on the teacher's confidence that they explain the sample. The first term includes the original label, so as to prevent incorrect teacher predictions from overwhelming the student. One further important trick is temperature scaling of the teacher logits, so that (x) exp(T" fy (x)). Setting T >
makes p' more uniform, thus preventing overconfident predictions (Guo et al., 2017).
Average versus subgroup performance. The above exposition treats the misclassification error Eavg(h) as the fundamental performance measure of interest. However, suppose the data contains subgroups 9 = {l...., .......),, G}. Defining the per-subgroup errors ( = P(h(2r) # g), we have Favg(h) (h) = =gge P(g) errg(h) (h), which may mask errors on samples with P(g) ~ (Sagawa et al., 2020a;b; Sohoni et al., 2020). To this end, one may instead measure the balanced error (Menon et al., 2013) (ha = Σ 2e [9] -erryhh) which treats the subgroup distribution as uniform, or the worst-subgroup error (Sagawa et al., 2020a;b; Sohoni et al., 2020) Emax (h) = max err, (h), which focusses on the worst-performing subgroup. An intermediary is the average of the & worst- performing subgroups (Williamson & Menon, 2019): for ith largest per-subgroup error (h). (h) = 1/2 E (h).
1
The definition of subgroups a domain-specific consideration. One special case is where each label defines subgroup (i.e., 9 y), and P(v) is skewed. In such long-tail settings (Buda et al., 2017), classifiers with good average performance can perform poorly on "tail" labels where P(y) ~
Related work. There is limited prior study that dissects distillation's 's overall gains per subgroup. Zhao et al. (2020) showed that in incremental learning settings, distillation can be biased towards recently observed classes. We show that even in offline settings, distillation can harm certain classes. Recently, Zhou et al. (2021) studied the standard aggregate performance Favg of distillation, which was tied to certain subset of "regularisation samples". By contrast, primary concern is to understand the subgroup performance of distillation.
Study of the fairness of machine learning algorithms on under-represented data subgroups has received recent attention (Dwork et al., 2012; Hardt et al., 2016; Buolamwini & Gebru, 8; Chzhen al., 2019). This has prompted dissection of the performance of techniquess such as dimensionality reduction (Samadi et al., 2018). increasing model capacity (Sagawa et al., 2020a), and selective classification (Jones et al., 2021). We follow the general spirit of such works, studying a more delicate setting involving two separate models (the student and teacher), each with their own inductive biases. We present more discussion of related directions in $6. In related effort, Hooker et al. (2019) explore how model compression may harm subgroup performance compared to the original model.
3 ARE DISTILLATION GAINS UNIFORM?
We demonstrate that the gains of distillation are not uniform across subgroups: specifically, consider- ing subgroups defined by classes, distillation can harm the student's performance on the "hardest" few classes ($3.1). To understand the genesis of this problem, we perform ablations (ef. Table 1) that establish its existence in settings where there are insufficient samples to model certain classes, either due to the number of classes being large ($3.3), or the class distribution being skewed ($3.4). We then identify that the student may amplify the teacher's 's errors ($3.6). Finally, we corroborate these results for a more general notion of subgroup in a fairness dataset ($3.5).
3.1 DISTILLATION HURTS SUBGROUP PERFORMANCE
To begin, we consider the effect of distillation on a standard image classification benchmark, namely, ImageNet. We employ a self-distillation (Furlanello et al., 2018) setup, with ResNet-34 teacher and student models, trained with standard hyperparameter choices (see Appendix B). Following Cho & Hariharan (2019), we early stopping on the teacher model. We now ask: what is the impact of distillation on the student's 's overall and per-class performance? The first question has an expected answer: distillation improves the student's average accuracy by +0.4% (see the Baseline setting in Table 1). Judged by this conventional metric, distillation is thus a success.
A more nuanced picture emerges when break down the source of the above improvement. We compute the per-class accuracies for the one-hot and distillation models, to understand how the overall gains of distillation are distributed. Figure shows that these gains non-uniform: distillation
3

---

Under review as a conference paper at ICLR 2022
Setting
Dataset
Avg acc
Worst subgroup
Cumulativve accuracy gain
0.00
Baveline (83.1)
ImageNet
Stronger weccher ($3.2)
ImageNet
+0.17
-1.19
-0.02
Long tail ($3.4)
CIFAR 110AR-10011 L.T
+2.17
-1.46
ImageNet LT
+0.21
-0.32
Fairness (83.5)
UCI Adult
+3 10
-3.94
-0.44
Reduce aclasses (83.3)
CIFAR 100 ImageNet-
+1.93
+3.13
200
400
600
800
1000
Index
Figure 2: Cumulative gain of ResNet-34 self- distillation on ImageNet. For index k, we compute the gain in average accuracy the k worst classes. While average accuracy (k 1000) improves by +0.4%, for k S 40, distillation harms over the one-hot model (evi- denced by the negative gain).
Table I: Summary of findings in the ablation anal- ysis of distillation's subgroup performance ($3). In range of different settings, distillation seen to hurt the hardest subgroup accuracy (worst-k class accuracy or worst subgroup according an at- tribute), despite improving the average accuracy (upper rows). Decreasing number of labels helps improve the hardest classes (bottom row).
Teacher
Student
Average accuracy
Worst-11 accuracy
Teacher
Student
Average accuracy
Worst-10 accurscy
Teacher
Student
Average accurscy
Worst-10
accuracy
EffoNet
Ros-0
40.17
-1.19
Res-50
Res-50
40.39
-1.43
Res-34
Res-34
40,42
-2.00
EdfeNNe
Ros-34
0.00
-0.80
Res-50
Res-34
+0.39
-2.05
Res-34
Res-
+0.15
-3.00
EffoNet
Res-18
10.05
Res-50
Res-18
-1.00
Res-18
Res-18
-3,00
Table 2: Summary of effect of distillation on different teacher and student architectures considered for the ImageNet dataset. The comparison is with respect the one-hot (i.e., non-distilled) student. We find that distillation consistently hurts accuracy of the worst 10 classes.
in fact hurts the worst-k class performance for k < 40. (See Appendix C for detailed per-class breakdown.) Thus, distillation may harm the student on classes that already finds difficult. At the same time note that distillation does improve many classes, such the classes with relatively high accuracies from the one-hot student.
Given that average accuracy improves, it is worth asking whether the above is cause for concern: does it matter that performance on subgroups corresponding to the "hardest" classes suffers? While ultimately domain-specific consideration, in general exacerbating subgroup errors may lead to issues from the fairness perspective. Indeed, we shall see that distillation can also harm in settings where the subgroups correspond to sensitive variables; we discuss this further in $3.5.
At this stage, it is apposite to ask whether the above is an isolated finding, or indicative of a deeper issue. We thus study each of the following turn: (i) does the finding hold in settings beyond self-distillation? (ii) does the finding hold for other datasets, or is it simply due to the idiosyncrasies of ImageNet? (iii) what are some general characteristics of settings where the problem manifest?
3.2 IS DISTILLATION BIASED BY THE MODEL SIZE OR ARCHITECTURE?
Having begun with self-distillation setup, we now demonstrate that similar findings hold when the student and teacher architectures differ. Continuing with the ImageNet dataset, in the second row Table we report statisties for the overall average accuracy and average accuracy over the worst 10 classes when distilled ResNet-50 student from stronger teacher: Efficient-Net (Tan & Le, 2021) teacher achieving 85.7% accuracy. Again, we see improved average accuracy and harmed Worst subgroup accuracy, composed of the 10 classes with the lowest accuracy according to teacher's performance. In Table 2, we report statistics when varying teacher and student architectures. The detrimental effect of distillation hard class performance holds across all scenarios: thus, earlier results were not specific to relf-distillation.
For self- eelf-distillation settings, smaller models appear incur greater losses on the worst-class error. When distilling between different architectures (e.g., from ResNet-50 to ResNet- 18), we observe that average accuracy may not improve, as noted in Cho & Hariharan (2019). There however
4

---

Under review as a conference paper at ICLR 2022
clear trend between the difference in architectures and drop in worst class performance. Finally, note that there is little change in the teacher's 's and student's 's worst-k classe; see Figure 6 (Appendix).
3.3 Is DISTILLATION BIASED BY A LARGE NUMBER OF CLASSES?
Having seen that ImageNet consistently demonstrates a performance degradation on certain classes, we now repeat the same analysis on a smaller image classification benchmark. We return to the self- distillation setup, using ResNet--6 models on CIFAR-
On this dataset, Table 1 shows a (perhaps more expected) result: distillation boosts both the average and worst-1 class performance. This indicates that, at a minimum, the behaviour of distillation's 's performance gains are problem-specifie; on CIFAR, distillation appears complete win for both the average and subgroup accuracy.
One plausible hypothesis is that the tension between average and subgroup performance only mani- fests on problems with many labels, which might informally be considered "harder". To confirm this further while fixing the dataset, we randomly select 10% of classes from the ImageNet dataset, and only keep examples corresponding to those classes across the train and validation sets. Consistent with the results for CIFAR-100, we again find that the worst- 10 class accuracy is not harmed under distillation (see the right side of Table 1 where we report results on ResNet-34 34 ceff-distillation..
The above indicates that for problems with a few, balanced labels, there may not be a tension between average and worst-subgroup performance under distillation. However, problems with relatively few labels, one may harm subgroup performance if there is label imbalance.
show that even for
3.4 IS DISTILLATION BIASED BY CLASS IMBALANCE?
We now consider a long-tail setting, where the training label distribution P(y) is highly non-uniform. Following the long-tail learning literature (Cui et al., 2019; Cao et al., 2019; Kang et al., 2020), we construct "long-tailed" (LT) versions of the above datasets, wherein the training set is down-sampled so to achieve a particular label skew. For ImageNet, we use the long-tailed version from Liu et al. (2019). For other datasets, we down-sample labels to follow Fy = i) x 1 for constant f and i [L] (Cui et al., 2019). The ratio of the most to least frequent class is set to 100.
From the Long tail setting in Table 1, we note that on both 1IFAR-100-LT and ImageNet-LT, accuracy over the hardest classes drops (we report results from self-distillation using ResNet-56 for CIFAR-100 and ResNet-50 ImageNet). The former is particularly interesting, given that the standard CIFAR-100 shows gains amongst the hardest classes. This provides evidence that for some "harder" problems e.g., where there are insufficiently many samples from which model a particular class there may be a tension between the average and subgroup performance.
3.5 BEYOND CLASSES: OTHER CHOICES OF SUBGROUPS
Our analysis thus far has focused on subgroups defined by classes. This choice is natural for long- tailed problems, where it is important to ensure good model performance on classes (Kang et al., 2020). In other problems, different choices of subgroups may be appropriate. For example. in problems arising fairness, one may define subgroups based on certain sensitive attributes (c.g., sex, race). In such settings, does one similarly see varying gains from distillation across subgroups?
We confirm this can indeed hold the UCI Adult dataset using random forest models (details in Appendix C.3). This data involves the task of predicting if an individual's ' income is 2 50K or not, and possesses subgroups defined by the individual's race and sex. Akin the preceding results, we find that distillation can significantly improve overall accuracy, at the expense of degrading accuracy on certain rare subgroups, e.g., Black women; see Table 1, and Table (Appendix). This further corroborates basic observation on the non-uniform distribution of distillation's 'ss gains.
A distinct notion of subgroup was recently considered Zhou et al. (2021), who identified the impact of certain "regularisation samples" on distillation. These are subset of training samples which were seen to degrade the overall performance of distillation. It is of interest whether such a subgroup relates regularisation samples in such subgroups, which might explain the poor performance of distillation? In Appendix C.4, study the relationship between regularisation samples, and the per-label subgroups
previously studied subgroups of "hard" classes; e.g., is there abundance of
5

---

Under review as a conference paper at ICLR 2022
0.75
0.50
0.4
0.00
-0.25
-0.50
-0.75
5
5
4
5
Class bucket
Class bucket
Class bucket
(a) Accuracy.
(b) Log-loss.
(c) Margin.
Figure 3: Logit statistics on CIFAR-100 LT, for the teacher and distilled student under self- distillation setup (ResNet-56 + ResNet-56). We show the statistics on 10 class buckets: these are created by sorting the 100 classes according to the teacher accuracy, and then creating 10 groups of classes. As expected, the student follows the general trend of the teacher model. Strikingly, we observe that the teacher model tends to systematically confidently mispredict samples in the higher buckets, thus incurring a negative margin; such misplaced confidence is largely transferred to the student, whose accuracy suffers on such buckets. Note that we consider statistics on the test set.
from our analysis; we find that, in general, these may be complementary notions. We further analyze the effect of the technique proposed in Zhou et al. (2021) on average and subgroup accuracies in $5.
3.6 WHY DOES DISTILLATION HURT CERTAIN SUBGROUPS?
The above has established that in range of scenarios, distillation can hurt performance on subgroups defined by individual classes. However, a firm understanding of why this happens remains elusive. To study this, we consider ResNet-56 self-distillation on CIFAR-100-LL - which showed a stark gap between the average and subgroup (i.e., worst-1 class) performance - and dissect the logits of the teacher and distilled student. (See the Appendix for plots where the teacher and student architectures differ.) Across classes, we seek to understand: (i) how aligned are the student and teacher accuracies? (ii) how reliable are the models' probability estimates? (iii) how do the models' confidences behave?
For a test' example (Ir, y) and predicted label distribution p(x) E A we thus compute each models' accuracy, log-loss Elog(y. p(z)) 2002) lmarg (y, p(z)) = (x( max,'y Pt, (xr). Note that the latter may be negative if the model predicts the incorrect label for the example. Figure 3 shows these metrics on 10 class buckets: these are created by sorting the 100 classes according to the teacher accuracy, and then creating 10 buckets of classes. Within each bucket, we compute the average of the metric specified above.
log P. (2)), and margin (Koltchinskii & Panchenko,
Remarkably, for 5 out of 10 class buckets, average margins are negative, suggesting that the teacher is often wrong yel confident in predicting these classes. On these buckets, the student accuracy generally worsens compared to the teacher. Further, log-loss increases across all buckets (including those where accuracy improves), indicating reduced confidence in the true class of the distilled student. This points at potential source of the poorer performance on the worst-1 accuracy. Recall that the distilled student's aim is to mimic the teacher's logits on the training samples. This is a proxy to the student's 's true goal, which is mimicking these logits on test samples, so as to attain similar generalisation performance as the teacher. When such generalisation happens, the student can thus be expected to roughly inherit the teacher's per-class performance; in settings like the above, this unfortunately implies it will perform poorly on those classes with negative teacher margin.
IMPROVING SUBGROUP PERFORMANCE UNDER DISTILLATION
We now study simple means of correcting distillation to prevent the degradation of subgroup per- formance identified above. These leverage the insight that the behaviour is potentially a result of
'The choice of test, rather than train, example is crucial: an overparameterised teacher will likely correctly predict all training samples, thus rendering the above statistics of limited use. To leverage the insights from the above analysis in practice, shall use holdout set that can be carved from the training set.

---

Under review as a conference paper at ICLR 2022
the teacher confidently mispredicting some subgroups. In the following, for concreteness and simplicity, we focus on subgroups that are given by the individual classes.
4.1 DISTILLATION WITH ADAPTIVE MIXING WEIGITT
In $3.6, we saw that distillation can hurt classes where the teacher is inherently inaccurate. Such inaccuracy may in fact be amplified by the student, which is hardly desirable. An intuitive fix is to simply rely less the teacher for classes where it performs poorly, is otherwise not confident; instead, the student can simply fall back onto the one-hot training labels themselves. Formally, for per-class mixing weights (a1...., aL) E [0, 1]". the student can minimise
N
Raist (1) (1)
E (1-ay_) oyy))) E(yn, (xn)) + Oly.." N ==
(ty ell y'E[L]
(2)
This objective introduces a mixing weight Og per-class, which allows us weigh between teacher predictions and one-hot labels for each class independently. By contrast, in the standard distillation setup equation we only have a single weight a that is common for all classes.
How do we choose the weights azy In the standard distillation objective equation 1, one only needs to tune single scalar a, which amenable to, e.g., cross-validation. By contrast, equation involves single scalar for each label, which makes any attempt at grid search infcasible. Following the observations in $3.6, we propose the following intuitive setting of 0y given teacher predictions p":
0g = max (0, Exly [Tavg (V. (v.P' (x)))) Tavg (x)) p)) L-
Σ (x). V'
(3)
In words, Equation 3 places greater faith in the teacher model for those classes which it predicts correctly with confidence, i.e., with large average margin Yavg- When this margin is negative so that the teacher is incorrect average, which can occur classes that are rare in the training set we set dty = 0, and completely ignore the teacher predictions. The above requires estimating the expectation Em [-]. which requires access labelled sample. This may be done using holdout set; we shall follow this in our subsequent experiments.
4.2 DISTILLATION WITH PER-CLASS MARGINS
Our second approach for improving distillation harder classes is leverage recent developments in loig-tail learning. where the goal is to improve performance on rare classes. Specifically, Khan al. (2018); Cao al. (2019); Tang et al. (2020); Ren et al. (2020); Menon al. (2020); Wang et al. (2021) proposed a variant of the softmax cross-entropy with margins Pyv' between label pairs:
Ely, f(x)) = log 1 +
Σ Pay (t)-(() V'žy
Intuitively, this penalises predicting label !/ instead of y when Pgy is large. For training label distribution म. Cao et al. (2019) proposed to set Pyy' & exp(mT 1/4 so that rare labels receive a higher weight when misclassified. Khan et al. (2018); Ren al. (2020); Menon et al. (2020); Wang et al. (2021) showed gains with Pay x Ty' so that rare labels are not confused with common ones. T.
We adapt such techniques to our setting, with the intuition that we ought to increase the student penalty for misclassifying those "hard" classes that the teacher has difficulty modeling. We thus choose Pyy' Sy, where Ot is the adaptive per-class mixing weight from the previous section. This 0,' discourages the model from confusing "hard" labels y with "easy" labels y', when ay > Oty- To avoid a division by issue, we add a small offset to 0g when it becomes 0.
We may understand the effect of Equation 4 by studying how it impacts the Bayes-optimal student model predictions, i.e., the optimal predictions in the infinite sample limit, and without a model capacity restriction. We have the following.
Lemma 1. Let & be the loss of Equation 4, with Pwv'
Szi, Let be the Bayes-optimal student oy
scores that minimise Raist (f) = (p' (x)) (x)())). Then, Vo e X. [L]. (z) -log (x(x) 0,
7

---

Under review as a conference paper at ICLR 2022
Table Summary of student's average accuracy using one-hot and distilled labels. Worst is denotes accuracy over the worst k classes. Global and adaptive temperatures ay selected using held out dev set. The proposed AdaAlpha technique improves both mean and worst class accuracy over vanilla distillation. For AdaMargin on CIFAR-10K 100 LT and ImageNet LT, we observed divergence during training, presumably due to this method being sensitive to the selection of hyperparameters, which in turn are estimated on very small number of examples per class.
Method
Mean
Per-class accuracy statisties
Worst-1
Worst-10
Dataset
Method
Per-class accuracy statistics
Worst-1
Worst-10
Worst-100
CIFAR-100
One
73.31
45.67
CIFAR- LT
One-bot
43.22
0.00
2.33
N/A
Distillation
75.24
49:00
54.65
Distillation
45.39
0.00
0.87
NIA
AduAlpha
75.43
49.33
56.42
AdaAlpha
48.57
0.67
4.20
N/A
AdaMargin
75.15
51.33
56.62
AdaMargin"
Training diverges
ImageNet
One-hot
76.38
14.00
23.64
ImageNet LT
One bot
45.41
0.00
0.00
1.39
Distillation
76.35
13.00
22.02
Distillation
45.98
0.00
0.00
1.10
AduAlpha
76.57
13.00
23.22
AdaAlpha
46.15
0.00
0.00
1.08
AduMargin
76.36
14.00
23:20
AdaMargin
Training diverges
Lemma illustrates that using per-class margins encourages the student to mimic the teacher predictions p² (x)) but with an important modification: we up-weight the probabilities for classes that the teacher does poorly (a, ~ 0). Intuitively, this makes it easier for the student to improve performance on classes with small teacher margin.
Relation to existing work. Previous works varied distillation supervision across examples towards improving average accuracy. Proposals included weighting samples based on the ratio (Tang et al., 2019; Zhou et al., 2021), and difference (Zhang al., 2020) between student and teacher score. Similarly, Zhou et al. (2020) proposed to only apply distillation on samples the teacher gets correct.
RESULTS FOR ADAPTIVE DISTILLATION METHODS
We now present results that further corroborate the potential non-uniform gains of distillation, and the ability to mitigate this with the techniques of the previous section. We emphasise here that our goal is expressly not to improve over the statc-of-the-ar in distillation techniques; rather, we wish to verify the key principles identified in the preceding study, which considers distillation from novel angle (i.e., in terms of subgroup rather than average performance).
Setup. We report results the datasets used in $3: CIFAR-100, ImageNet; and long-tailed (LT) versions of the same. For brevity, we report results under deff-disiillatinn. (For results with varying architectures, see the Appendix.) Thus, for each dataset, train one-hot teacher ResNet model, which is distilled to a student ResNet of the same depth. We use ResNet-56 models for CIFAR, and ResNet-50 models for all other datasets. We employ the same syper-parameten we use non-early stopped teachers for consistency across datasets; see the Appendix for details.
used in $3, except
We compare: (i) one-hot training of the student (ii) standard distillation, i.e., minimising Equation 1 (iii) AdaAlpha, our proposed distillation objective with adaptive mixing between one-hot and teacher labels Equation 2, and a as per Equation 3 (iv) AdaMargin, our proposed distillation objective with adaptive margins (Equation 4), and Pyy' as standard mean accuracy over all classes; (ii) the accuracy over the worst-1 class; and (iii) the mean accuracy over wo (and wo0 for the LT datasets) classes.
ay' We summarise each method by reporting: (i) the
For the Ada-* methods, per $4, creating the label-dependent ag requires estimating the generalisation performance of the teacher. To do this, create random holdout split of the training set. For non-LT datasets, we randomly split into 80% (new train) - 20% (dev). For LT datasets, for each class we hold out l examples into the dev set (k: 50 for Imagenet-LT, k 20 for CIFAR-100-LT)) or half of examples for a class if the total number of per class examples is at most 2k. We train an initial teacher on the new train slice of data, and estimate its per-class performance the holdout dev slice. These are used to estimate Ot as per, e.g., Equation 3.
Table 3 summarises the results for all methods. We make the following observations.
8

---

Under review as a conference paper at ICLR 2022
Table 4: Ablations of design choices in the proposed methods: 1) remove distillation signal from the bottom 10% of classes, according to confidence; 2) randomly shuffle per-class a values; 3) weight distillation based on student and teacher confidence Zhou al. (2021).
Per-elass accuracy statisties
Datasel
Method
Mean
Worst-1
Worst-10
CIFAR-100
AdaAlpha
75.52 + 0.10
49.33 + 3.09
56.10 0.44
remove hardest 10%
75.40
0.04
48.33 1: 1.89
55.79 1.19
shuffle temperatures
74.56
0.93
46.00 1 1.91
53.10 1.22
Zhou et al. (2021)
75.14 + 0.29
15.11 # 1.66
53.89 0.66
AdaAlpha improves mean accuracy over vanilla distillation. The proposed AdaAlpha method consistently and significantly improves standard mean accuracy over vanilla distillation. Thus, AdaAlpha does not sacrifice the gains offered by distillation on average class performance, which desirable. Other techniques sometimes perform slightly worse than standard distillation on this metric; however, as we now see, this is compensated by gains on other important dimensions.
AdaAlpha improves worst-accuracy over distillation. The proposed method consistently improves the worst-class accuracy compared to standard distillation: Thus, the technique largely fulfil their design goal of improving performance on "hard" classes, while not overly sacrificing average-case performance. In most cases, these improve both the average and worst-class accuracy, indicating that softening the teacher influence can be broadly beneficial.
Comparison of AdaAlpha and AdaMargin. In the Appendix, we report per class statistics for CIFAR- 100 LT. AdaMargin flattens both the margin and log-loss distributions, reducing confidence on the poorly classified, tail classes. AdaAlpha consistently increases log-loss across classes, and improves margins on few buckets, leading to a positive margin on one bucket where all other methods give negative margins. Intuitively, AdaMargin tries to more aggressively control the worst-class accuracy; when this succeeds, there is a large payoff, but there is also greater risk of overfitting.
Additional ablations. We confirm that the success of AdaAlpha is not immediately replicated by simpler baselines: (i) remove hardest 10%, which removes the distillation loss component on bottom 10% labels according the per class margins found using Equation 3. It helps analyze whether there is any additional gain beyond simply removing teacher's supervision where is arguably wrong. (ii) shuffle temperatures, which randomly shuffles the per-class ay values used in AdaAlpha. This determines whether the precise choice of which labels to up- or down-weight is important; (iii) the adaptive distillation scheme of Zhou et al. (2021), where distillation is weighted differently across examples depending on the teacher and student scores.
In Table 4, we find that the first two methods work worse than the proposed AdaAlpha method, indicating that the precise choice of which labels to up- or down-weight is important, and that it does not suffice to merely ignore the teacher on entire subgroups. The adaptive distillation scheme Zhou et al. (2021) is also not as effective AdaAlpha; see Appendix for more such results.
6 DISCUSSION AND OTHER APPROACHES
Our goal of ensuring equitable performance across classes can be seen encouraging fairness subgroups defined by the classes. This is subtly different to the classical fairness literature (Calders & Verwer, 2010; Dwork et al., 2012; Hardt et al., 2016), wherein the subgroups defined by certain sensitive attributes. Broadly, fairness techniques attempt to learn models that predict the target label accurately, but the subgroup label poorly; these inadmissible for setting, wherein the two labels exactly coincide. Ensuring fairness across subgroups defined by the classes has been studied in Mohri al. (2019); Williamson & Menon (2019); Sagawa al. (2020a), who proposed algorithms to explicitly minimise the worst-class (as opposed to the average) loss. Adapting such algorithms to the distillation setting of interest for future work. More broadly, the intent of the analysis this paper is to better understand settings where distillation can implicitly hurt certain rnder-represented subgroups.
fairraeessn across
9

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
This work studies settings wherein distillation can harm performance on subgroups. The aim of such a study is to help identify and mitigate potential adverse effects from distillation in practical applications where fairness considerations are important. Further, our proposed techniques were shown to improve the subgroup performance distillation, and thus could be useful for preventing such adverse behaviour. As a limitation, we considered natural definitions of such subgroups, but an important practical consideration is settings where subgroup membership unknown.
REPRODUCIBILITY STATEMENT
For all experiments, we used the default hyperparameters from previous works (see Appendix B for details).
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. CoRR, abs/2012.09816, 2020. URL
abs/2011 09816.
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E. Dahl, and Geoffrey E. Hinton. Large scale distributed neural network training through online distillation. In International Conference on Learning Representations, 2018.
Cristian Bucilä, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA, 2006. ACM.
KDD '06, pp- 535-541
Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study the class imbalance problem in convolutional neural networks. arXiv:1710.05388 [cs, stat], October 2017.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Sorelle A. Friedler and Christo Wilson (eds.), Conference on Fairness, Accountability, and Transparency. volume 81 of Proceedings of Machine Learning Research, pp- 77-91, New York, NY, USA, 23-24 Feb 2018, PMLR.
Toon Calders and Sicco Verwer. Three Naive Bayes approaches for discrimination-free classification. Data Mining and Knowledge Discovery, 21(2):277- 2010.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Aréchiga, and Tengyu Ma. Learning imbalanced datasets with aebellddstribution-awwa margin loss. In Advances in Neural Information Processing Systems 32, pp. 1565-1576, 2019.
J. H. Cho and B. Hariharan. On the efficacy of knowledge distillation. In 2019 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pp. 4793 4801, 2019.
Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, Luca Oneto, and Massimiliano Pontil. Leverag- ing labeled and unlabeled data for consistent fair binary classification. In H. Wallach, H. Larochelle. A. Beygelzimer, F. d' Alché Buc, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 12760-12777 12770. Curran Associates, Inc., 2019.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balancee loss based effective number of samples. In CVPR, 2019.
Tri Dao, Govinda M Kamath, Vasilis Syrgkanis, and Lester Mackey. Knowledge distillation as semiparametric inference. International Conference on Learning Representations, 2021. URI https: /ttp///epeeeeeeeeeeeeeeee/eeeeet ew net/ forum? ? d=m40CC24+0Y.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Faimess through awareness. In Innovations in Theoretical Computer Science Conference (ITCS), pp. 214-226, 2012.
10

---

Under review as a conference paper at ICLR 2022
Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandku- mar. Borm-agan neural networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässann Stockholm, Sweden, July 10-15, 2018, pp. 1602-1611, 2018.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, pp. 1321- 1330, 2017.
Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems (NIPS), December 2016.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, abs/1503.02531, 2015.
Sara Hooker, Aaron C. Courville, Yann N. Dauphin, and Andrea Frome. Selective brain damage: Mcasuring the disparate impact of model pruning. CoRR, (bss/1911.05248, 2019. URL http:
//arxiv. org/ abs /1911.911111 05248.
Guangda Ji and Zhanxing Zhu. Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems, 2020.
Erik Jones, Shiori Sagawa, Pang Wei Koh, Ananya Kumar, and Percy Liang. Selective classification can magnify disparities across groups. In International Conference on Learning Representations, 2021.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng. and Yan- Kalantidis. Decoupling representation and classifier for long-tailed recognition. In Eighth International Conference Learning Representations (ICLR), 2020.
Salman H. Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A. Sohel, and Roberto Togneri. Cost-sensitive learning of deep feature representations from imbalanced data. IEEE Transactions Neural Networks and Learning Systems, 29(8):3573-3555 3587, 2018. 2732482.
doi: 10. N0.1109TNNLS.20111
V. Koltchinskii and D. Panchenko. Empirical margin distributions and bounding the generalization error of combined classifiers. Ann. Statist., 30(1):1-50) 02 2002.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-scale long-tailed recognition in an open world. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 2537-2546. Computer Vision Foundation IEEE, 2019.
D. Lopez-Paz, B. Schölkopf, L. Bottou, and V. Vapnik. Unifying distillation and privileged informa- tion. In International Conference on Learning Representations (ICIR), November 2016.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, 2017.
Aditya Krishna Menon, Harikrishna Narasimhan, Shivani Agarwal, and Sanjay Chawla. On the statistical consistency of algorithms for binary classification under class imbalance. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, pp. 603-611, 2013.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment, 2020.
Hossein Mobahi, Mehrdad Farajtabar, and Peter L. Bartlett. dittiiistilattinn amplifies regularization in hilbert space, 2020.
11

---

Under review as a conference paper at ICLR 2022
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In Interna- tional Conference on Machine Learning, 2019.
Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems 32, pp. 4696-4705, 2019.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. International Conference on Learning Representations, 2020.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What being transferred in trans- fer learning? In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 512-523. Curran Asso- ciates, Inc., 2020. URL
ngs. neurin pdf.
neurips cc/papprr/222//11e.
Ilija Radosavovic, Piotr Dollár, Ross B. Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: Towards omni-supervised learning. 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 4119-4128, 2018
Jiawei Ren, Cunjun Yu, shunan sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and hongsheng Li. Balanced meta-softmax for long-tailed visual recognition. In H. Larochelle, M. Ranzato, R. Hadsell, M. E. Balcan, and Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 4175-4186. Curran Associates, Inc., 2020.
S. Sagawa, P. W. Koh, T. B. Hashimoto, and Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations (ICLR), 2020a.
S. Sagawa, A. Raghunathan, P. W. Koh, and P. Liang. An investigation of why overparameterization exacerbates spurious correlations. International Conference on Machine Learning (ICML), 2020b.
Samira Samadi, Uthaipon Tantipongpipat, Jamie H Morgenstem, Mohit Singh, and Santosh Vempala. The price of fair pea: One extra dimension. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Biancer and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 31, pp. 10976- 17977-1977.. Curran Associates, Inc., 2018.
N. Sohoni, J. Dunnmon, G. Angus, A. Gu, and C. Ré. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. In To appear in Conference on Neural Information Processing Systems (NeurIPS), 2020.
Mingxing Tan and Quoc V. Le. Efficientnetv2: Smaller models and faster training, 2021.
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H. Chi, and Sagar Jain. Understanding and improving knowledge distillation. CoRR, abs/2002.03532, 2020.
Shitao Tang, Litong Feng Wenqi Shao, Zhanghui Kuang, Wayne Zhang, and Zheng Lu. Learning efficient detector with semi-supervised adaptive distillation. In 30th British Machine Vision Con- ference 2019, BMVC 2019, Cardiff. UK, September 9- 2019, pp. 215. BMVA Press, 2019. URL papers 0145-paper.pppppp
https: httpc/cw2222.m org/ wtt//prgg//oooooooo wop-conttnt/upleeoc
Grant Van Horn and Pietro Perona. The devil is in the tails: Fine-grained classification in the wild. arXiv preprint arXiv: 1709001450 2017.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8769-8778, United States, December 2018. Institute Electrical and Electronics Engineers (IEEE). ISBN 978-1-5386- 6421-6. doi: 10.1 1199/CVR..00 8.00914. 14. URL http: ccvpr0111 thecvf com/ hom/.00 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018 ; Conference
date: 18-06-2018 Through 22-06-2018.
12

---

Under review as a conference paper at ICLR 2022
Jiaqi Wang, Wenwei Zhang, Yuhang Zang, Yuhang Cao. Jiangmiao Pang. Tao Gong, Kai Chen, Ziwei Liu, Chen Change Loy, and Dahua Lin. Seesaw loss for long-tailed instance segmentation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2021.
Robert C. Williamson and Aditya Krishna Menon. Faimess risk measures. Proceedings of the
36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, 2019.
California, USA. pp.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Youcai Zhang, Zhonghao Lan, Yuchen Dai, Fangao Zeng, Yan Bai, Jie Chang, and Yichen Wei. Prime- aware adaptive distillation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 678. Springer International Publishing. 2020. ISBN 978-3-030-58529-7.
Zhilu Zhang and Mert R. Sabuncu. Self self-distillation as pustance-speciffe label smoothing. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and fairness in class incremental learning. In 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13205-13214. IEEE, 2020.
Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. Interna- tional Conference on Learning Representations, 2021.
Zaida Zhou, Chaoran Zhuge, Xinwei Guan, and Wen Liu. Channel distillation: Channel-wise attention for knowledge distillation. CoRR, abs/2006.01683, 2020. URL https: //arxiv.. org/abs/ 2006. 01663
13

---

Under review as a conference paper at ICLR 2022
A PROOFS
Proof of Lemma 1. We may write
(V. = log
1
By
cfv (x))))oo(a))
4'zy Og
log
Σ
alaay Iy (1)(1))
4'
( fy(x)
log
1
eFola)
=- log
Ey (x)
where fy(x)= = fz(x) + In ay. The population loss is
(f) Ex [(='())TT((((()))
Eg [KL [z)
where (x) x exp(Jy(x))- Thus, at optimality we must have 5² (x)) = pp or Fo(Ø) = log (3r).
By definition of f, we thus that fy = log p'g log fy llog px)(x)
dv
B DETAILS OF EXPERIMENTS
B. ARCHITECTURE
We use ResNet with batch norm (He et al., 2016) for all our experiments with the following configura- tions. For CIFAR, we experiment with ResNet-56 and ResNet-32. For ImageNet, we use ResNet-50. We list the architecture configurations in terms of (Mayers Dateers stride) corresponding to each ResNet block in Table 5.
Architecture
Configuration: [(Daayers Baterr stride)]
CIFAR ResNet-32
[(5, 16, 1), (5, 32, 2), (5, 64, 2)]
CIFAR ResNet-56
[(9, 16, 1), (9, 32, 2), (9, 64, 2)]
ImageNet ResNet-18
[(2, 64, 1), (2, 128, 2), (2, 256, 2), (2, 512, 2)]
ImageNet ResNet-34
[(3, 64, 1), (4, 128, 2), (6, 256, 2), (3, 512, 2)]
ImageNet ResNet-50
[3,, 64, 1), (4, 128, 2), (6, 256, 2), (3, 512, 2)
Table 5: ResNet Architecture configurations used in our experiments (He et al., 2016). [*] Note that ImageNet ResNet-50 uses larger blocks with 3 convolutional layers per residual block compared to ResNet- 18 and 34. We refer He al. (2016) for details.
B.2 TRAINING SET
For all datasets, we train using SGD and weight decay 10 for CIFAR, and 0.5 x 10-4 for Imagenet datasets. We have the following dataset specific settings.
CIFAR-100. We train for 450 epochs with an initial learning rate of 1.0, with linear warmup in the first 15 epochs, and annealed learning rate schedule. We drop the learning rate by a factor of 10 at epochs number: 200, 300 and 400. We use a mini-batch size of 1024. We use SGD with Nesterov momentum of 01..
14

---

Under review as a conference paper at ICLR 2022
Accuracy Butto 10%
0.1
Wargin
0.2
0.
0.5
8
Tall dess Sub-bucket
Tal dass SuD bucket
Tall class sub
(a) Accuracy (tail classes).
(b) Log-loss (tail classes).
(c) Margin (tail classes).
Logit statistics for ResNet-50 delf-distillation on ImageNet, for the (early-stopped) teacher, self-distilled student, and one-hot (non-distilled)) student. Per Figure 3, we first create 10 class buckets. We zoom in the "tail" bucket (comprising the 100 "hardest" classes), and further split them into 10 "tail sub-buckets". As in Figure 3, the teacher is seen to confidently mispredict most samples on the last few buckets, with such misplaced confidence being transferred to the student.
Figure 4:
For our distillation experiments we train only with the cross-entropy objective against the teacher's logits. For each method we find the best temperature from the list of values: {1, 3, 4.53.
ImageNet. We train for 90 epochs with an initial learning rate of 0.8, with linear warmup in the first 5 epochs, and an annealed learning rate schedule. We drop the learning rate by a factor of 10 at epochs number: 30, 60 and 80. We use a mini-bate batch size of 1024.
For our distillation experiments we train with the distillation objective as defined in Equation setting = 0.2. For each method we fix the temperature to 0.9.
Long-tail (LT) datasets. We follow setup as in the non-long tail version, except for the learning rate schedule, which we change to follow the cosine schedule (Loshchilov & Hutter, 2017).
C ADDITIONAL EXPERIMENTS
We present additional experiments to those in the body.
C.I FURTHER VARYING DATASETS AND MODEL ARCHITECTURES
On Imagenet, we summarise statistics for three models: the carly-stopped teacher, distilled student, and the one-hot (non-distilled) student. As with CIFAR-100-LT,, we sort classes by teacher accuracy, and bucket them into 10 groups. Owing to the larger number of labels, we further zoom into the "tail" bucket (comprising the 100 "hardest" classes), and split them into 10 sub-buckets. From Figure the distilled student performs worse than its one-hot counterpart on the last bucket; this is in keeping with our results in Table
Figure 5 shows logit statistics for additional settings to considered in the body. On ImageNet-LT, e.g., we see again that the margin of the teacher model systematically worsens and becomes negative on the hardest classes.
In Table 6 we report results from the inherently long-tailed iNaturalist 2018 dataset (Van Horn Perona, 2017). Our observations made for other considered datasets hold: adaptive margin method improves over both one hot and plain distillation in terms of the worst class accuracy. We also observe, how the average accuracy improves.
C.2 LOGIT PLOTS UNDER ADA- METHODS
Figure shows the logit statistics under the proposed AdaMargin and AdaAlpha methods on CIFAR- 100 LT. We see that AdaMargin can generally improve the student margin and accuracy the hardest classes, while also reducing the los-l This confirms that the gains of the method come from improving behaviour of the scores on these hard classes.
15

---

Under review as a conference paper at ICLR 2022
CIFAR-1 10 res56
CIFAR-100
2.5
student
studintt student one hot teacher
student one hot teacher
Margh
2.5
0.4
0.5
5
9
0
8
Class bucket
Class bucket
araaneet res5o
res50 0.9]
Imagenct res50 0.9]
2.0
student disil
studert one-hot
0.8
Sludent one-hot
student distil
1.5
0.6
Margin
L.0
0.4
0.5
0.2
0.0
5
Class bucket
Class bucket
Imagenet-LLT res50 res18 IT 0.9]
Imagenet-LT res50
res18 8 IT 0.91
studient ditill
student distil
student
Log 10g-
0.2
0.0
-0.4
bucket
Figure 5: Logit statistics for the teacher, student with one-hot labels, and student with distilled labels across: datasets and architectures.
1.0
Distiled Distiledaccurae accuracy
Distilled accuracy
0.8
0.6
0.4
0.0
One-hot accuracy
One-hot accuracy
Figure 6: Per-class accuracies for one-hot and self-distilled ResNet- 18 (left) and ResNet-34 (right) ImageNet. The diagonal denotes classes where both models achieve the same accuracy. Distillation tends worsen performance on "hard" classes for the one-hot model, i.c., those with low accuracy (red rectangle).
C.3 RESULTS ON ADULT DATASET
We report the results of an experiment on the UCI Adult dataset. This data comprises 48K examples, with the target being a binary label denoting whether or not an individual has income 50K. The data is mildly imbalanced, with 24% of samples being positive.
Inspired by Dao et al. (2021), we consider a random forest based distillation setup: we use a teacher model that is random forest classifier comprising 500 trees with maximum depth of 20, and student model that is random forest regressor comprising tree with a maximum depth of 20. The teacher model achieves test (balanced) accuracy of 81.8%.
16

---

Under review as a conference paper at ICLR 2022
Per-class accuracy statisties
Method
Mean
Worst 20
Top 20%
20
One-hot
53.00
5.00
100.00
48.00
Distill
52.67
5.00
100.00
47.67
AdaMargin
53.33
8.33
98.33
45.00
AdaAlpha avg
54.67
13.33
100.00
41.33
Table 6: Self-distillation experiments (from Resnet-50 to Resnet-50) the iNaturalist dataset Van Horn et al. (2018) with student's average accuracy using one-hot and distilled labels. Worst 20 denotes accuracy averaged over worst 20 classes A20 denotes the difference between the mean accuracy and the worst 20 classified classes. The proposed AdaMargin technique improves mean and worst-class accuracy over both one-hot training and standard distillation.
AdUAISh Addhaarg standard
0.75
studere ASuAlsto
Atarlorgin
0.6
AdvMaggin
Heacher
0.4
0.00
-0.25
0.2
0.0
Class budkat
Class backet
Class bucket
Accuracy.
(b) Log-loss.
(c) Margin).
Figure 7: Logit statistics for ResNet-56 self-distillation CIFAR-100 LT, for the teacher, self- distilled student, and our adaptive methods. Per Figure 3, we create 10 class buckets. AdaMargin flattens both the margin and log-loss distributions. AdaAlpha increases log loss across classes, while improving margins on few buckets, including flipping the bucket 5 to have positive margin.
Table 7: Difference between distillation and one-hot performance on Adult dataset. Here, subgroups defined by the sex and label. A refers the difference between the distilled and one-hot student's accuracy on the subgroup.
label
A
Male
-2,222
Female
0
0.393
Female
2.373
Male
8.384
We perform distillation by feeding the student model the teacher's prediction scores, mixed in with the binary training labels with weight 0.9. Distillation improves the student's overall (balanced) accuracy significantly, from 76.2% to 79.3%. However, this gain is not distributed uniformly: using per-label subgroups, find that distillation helps the positive class by +7.4%, but hurts the negative class by- -1. 1..2% While by itself suggestive of asymmetry distillation performance, the data admits an arguably more natural subgroup creation, based on available sex and features. For example, we find that amongst low-income males, distillation hurts by 2.2%% further restricting to those who are Asian Pacific-Islandee the degradation -5.9. This confirms that in scenarios where fairness may be consideration, naïve application of distillation may be inadmissible.
C.4 ANALYSIS OF REGULARISATION SAMPLES
Recently, Zhou et al. (2021) proposed the notion regularisation samples to understand how distillation's 's performance teacher's 's prediction on the training label is less than the distilled student's 's prediction on this label; these may be shown to correspond to cases where certain notion of "variance reduction" dominates
be improved. In brief, such samples correspond cases where the
17

---

Under review as a conference paper at ICLR 2022
Table 8: Difference between distillation and one-hot performance Adult dataset. Here, subgroups are defined by the sex, race, and label. A refers to the difference between the distilled and one-hot student's accuracy on the subgroup.
race
SeX
label
A
Amer-Indian-
Female
-66.667
Asian-Pac-Islante
Male
0
-5,941
Other
Male
0
-4.347
Black
Female
-2.381
White
Male
-2.248 0
Black
Male
0
-1.639
Black
Female
O
-0.140
Ssian-Pac-Islander
Female
0
0.000
White
Female
0
0.388
Other
Female
0
2.439
White
Female
2.724
Amer-Indian-Eskee
Female
0
6.349
Amer-Indian-Esoe Eskimo
Male
6.493
Asian-Pac-Islandee
Female
7.692
White
Male
7.796
Black
Male
9.489
Other
Male
15,000
-ain--eac-lslande
Male
19.626
Other
Female
20.000
Amer-Indian- -eeia--ndda--
Male
25.000
notion of "bias reduction". Given our analysis above of the asymmetric effects of distillation on certain subgroups, is natural consider whether or these relate the presence of regularisation samples in these groups
Figure 8(a) visualises the distribution of regularisation samples inside subgroups defined by 10 label buckets. where the labels are sorted in descending order of label frequency. Here, we compare the predicted probabilities of the teacher and final distilled student models on all training samples (as was done in the analysis of Zhou et al. (2021)). Interestingly, we see that the tail buckets tend to have very few regularisation samples; i.e., for rare labels, the teacher prediction on the training label is generally higher than that of the distilled student model. We confirm this in Figure 8(b).
While the analysis Zhou et al. (2021) was primarily for training samples - since the aim in identifying regularisation samples was to mitigate their influence during training identify the breakdown of such samples on test data. Figure 9(a) shows that, compared to the training set, there in absolute terms more such samples across nearly every label bucket; however, there is again no clear correlation between the label bucket and the fraction of such samples. In particular, the tail bucket is again the one with the fewest regularisation samples. This is corroborated by the probability scores of the teacher and student in Figure 9(b).
may also
Overall, this results suggest that the existing notion of regularisation samples may not, by themselves. be sufficient to predict the poor performance of distillation on certain subgroups defined by labels.
C.5 IMPACT OF REPEATED DISTILLATION
In the body, we showed that performing distillation once can harm worst-class accuracy. However, what is the effect of repeating this process, and distilling using the resulting student as new teacher? Does the worst-class accuracy get further harmed?
Table 9 shows that on CIFAR- 100 LT, repeating distillation can indeed harm worst-class performance, even though average performance remains roughly similar. This further highlights the potential tension between average and worst-case performance under distillation.
18

---

Under review as a conference paper at ICLR 2022
1.0
0.3
62
Probabiity
0.6
30
20
varibble
Teacher
Studert
Label bucket
Label bucket
(a) Distribution of regularisation samples per lahel bucket.
(b) Teacher versus distilled student probail- ities training label.
Figure 8: Study of regularisation samples on training set, CIFAR- 100 LT.
70
0.8
Teacher
Student
40
Indbablig
0.5
0.2
10
4.0
2.0
5.0
5.0
Label
Distribution of regularisation samples per label bucket.
(b) Teacher versus distilled student probail- on test label.
Figure Study regularisation samples on set, CIFAR- 1-00 LT.
Per-class accuracy statistics
Method
Mean
Worst 10
Top 10%
A10
One-hot
44.16
3.00
87.70
41.16
Distillation 1 x
45.49
0.90
88.10
44.59
Distillation 2x
45.22
0.00
88.40
45.22
Distillation 3x
44.80
0.00
87.60
44.80
Table 9: Results of repeated distillation CIFAR-100 LT. Using a distilled student as teacher for subsequent round of distillation is seen to further hurt worst-class accuracy.
19


Reviews:
[{"review_id": "jSLocArpyi", "reviewer": "Reviewer_t1Uw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is a pleasure to read, very well written, clear and convincing with their arguments, and do some good work (with small caveats) regarding the experimental support of the claims. I also like that ImageNet is thoroughly used rather than sticking to datasets like CIFAR.\n\nMy main issues however are 1) while biases in learning is clearly a topic that has relevance nowadays, the focus of this paper ends up being on some specific aspect of a specific technique that might cause bias amplification. Thus, the focus is quite narrow 2) The results are not impressive. Performance improvements, even on the specific focus group (lower-performing classes) the gains are not very strong or consistent. In more detail:\n\nRegarding 1, generally speaking, I think the community has too much of a fascination with the logit matching knowledge distillation method. Knowledge distillation is widely used and very important, but the specific logit matching flavour of KD is a 2015 paper that has been thoroughly improved multiple times. I am always a bit confused about why the community keeps producing a large corpus of works dissecting small aspects of logit matching KD, while forgetting about the whole literature that followed that seminal paper. (I have to admit here though that logit matching KD still performs admirably against some of the other methods that are supposed to work better, and that it also tends to give gains when combining with other KD methods). \n\nStill, nowadays the state of the art is well above logit matching KD. I would be much more interested in the paper if it was applicable across KD methods (or at least on top of a couple of state of the art methods). For example, is it relevant for feature distillation (Heo et al., A Comprehensive Overhaul of Feature Distillation, ICCV'19)? or can it be combined with Xu et al., Knowledge Distillation Meets Self-Supervision, ECCV'20? \n\nIssue 2: The main results, shown in Table 3, are not very impressive. On ImageNet, (logit matching) distillation marginally hurts overall performance. The proposed method shows that it can keep the performance to roughly \"no distillation\" level, and for ImageNet LT the worst-1 and worst-10 accuracy are basically 0% for all (thus no improvement) and the worst-100 actually takes a small hit (quite irrelevant given the overall poor performance, but still). Also, the average performance is improved marginally. So all in all, the CIFAR-100 are the only convincing set of results.\n\nComparison with other approaches is also somewhat absent, e.g. Zhou et al (2021) (I think the most related method?) is only compared in one setting, and a table with a wide set of comparisons against other state of the art distillation methods is also absent (see for example Tables 3 and 4 of the aforementioned Zhou et al (2021) paper).\n\n\nOther aspects (not really important for the overall score and not necessary to reply):\n- On Fig. 1 caption: shouldn't it be 2% absolute instead of 1% absolute?\n- Page 2, bottom: the effect of the temperature is different (inverted) if > 1 or < 0, so the last phrase of the page is incorrect.\n- Eq. 2 is quite trivial, I'm not sure it is necessary to spell it out.\n- Section 3.1, the hyperparameter choices are not in appendix C, but in B.2 I believe.\n- Same paragraph, +0.4 improvement over the baseline is hardly a success... I'm not sure this kind of claim is too healthy\n- Table 1 could do with a CIFAR baseline so we can compare with line 3. Also, when comparing the long tail behavior with the standard behavior, we see less variance (less avg. gain, better worse performance), which feels kind of contradictory with the train of thought of the paper?\n- The last paragraph of section 3.4 looks unsubstantiated to me.\n- The analysis, especially for the experiments involving less classes, could do with some statistics - if the per-class performance is drawn from a normal distribution, it is clear that the worst performing one will depend on the number of classes (i.e. samples drawn from the distribution), so it is not surprising that reducing the number of classes of ImageNet means the performance of the worse class improves.\n- It feels like the formulation is calling for a per-sample weighting... in the end the average per-class performance is used as proxy but it seems to me that what matters is the behavior of the teacher for a specific instance.\n- Zhou et al. (2021) looks like the most related work, but I feel it is not explained properly in the related work. Explaining that method better, and highlighting the differences to the current proposal would improve the paper in my opinion.\n\n", "summary_of_the_paper": "The paper proposes to have a closer look at (logit matching) knowledge distillation, and explores whether the gains are uniform across all portions of the test data or if instead there are gains on some portions of the data but damaged performance on others. This is justified by the potential of KD to amplify biases. The authors quickly show that indeed the gains are not uniform, some classes being affected negatively by KD so that the bad performance of the teacher on some classes is amplified on the student. Then the authors conduct a number of experiments to study the impact of different aspects of training (e.g. architectures involved and aspects of data like class imbalance) over this amplification of class-specific poor performance. They are then able to identify what classes are likely to be negatively affected and follow that with the proposal of two modifications of the basic KD formulation with the aim to improve KD effect on the worse-performing classes. Experiments focus on CIFAR-100 and ImageNet, including long-tail variants.", "summary_of_the_review": "The paper has some interesting aspects and does a good deep-dive into some very specific aspect of knowledge distillation, but the focus is very narrow and the improvements obtained do not look impressive enough. The score would be around 4 if that existed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iP8aWwIN-w6", "reviewer": "Reviewer_pMdP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nThe identified problem, distillation performance on different subgroups, is highly related to the fairness of machine learning, which is an interesting and important research topic. Experiments on balanced and long-tailed datasets further evaluate how distillation performs diversely on different subgroups.\n\nThis paper proposed two methods for improving subgroup performance, AdaWeight with adaptive mixing weights and AdaMargin with per-class margins.\n\nThis paper is clearly written and well organized. The writing quality is satisfying.\n\n**Weaknesses**\n\nThe improvements of AdaAlpha and AdaMargin on ImageNet LT, a larger dataset for long-tailed classification, are marginal. Specifically, AdaAlpha achieves only an improvement of 0.15 mean per-class accuracy, and the training of AdaMargin diverges. It is not clear whether AdaAlpha works on large datasets. This marginal improvement limits the contribution of the proposed methods. Another large-scale dataset, iNaturalist 2018, may be helpful to further evaluate the effectiveness of AdaAlpha and AdaMargin.\n\nThe contribution of AdaMargin is somehow limited. As shown in Table 3, AdaMargin achieves a significant improvement of worst-1/10 per-class accuracy on CIFAR-100. However, the improvement on ImageNet compared to one-hot and distillation are very marginal. Also, the training of ImageNet LT diverges. These results show that AdaMargin is not a good choice in practice, which limits its technical contribution. \n\nFor the long-tailed setting, this paper evaluated the performance on worst-1, worst-10, worst-100 subgroups. The popular subgroups in long-tailed recognition are many-shot, medium-shot and few-shot. It is okay to choose not to report the performances on the normal subgroups, but an explanation on why not using these subgroups is welcomed.", "summary_of_the_paper": "This paper identified an underexplored problem with distillation, fairness of distillation on different subgroups. This paper further proposed two methods to tackle this challenge. Experiments are conducted on CIFAR-100, ImageNet, and their long-tailed variants.", "summary_of_the_review": "Overall, this paper proposed an interesting and important perspective on the fairness of distillation, which will inspire future works to pay more attention to the fairness of distillation. The writing quality is good. The contributions to empirical analysis are acknowledged by my side.\n\nThe weakness comes from the technical contributions. In particular, it seems that the proposed AdaMargin does not work well in practice. This limits the contributions of the methods part. The contribution of AdaAlpha is clear.\n\nConsidering both the strengths and weaknesses, I vote for acceptance at this stage. My initial score is between 6 and 8.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "by4ZkepEc-a", "reviewer": "Reviewer_QhBi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The authors aim at improving the performance of the underperforming groups of classes, which can be easily observed. Thus, for me, this observation is not surprising enough. So in section 3 (are distillation’s gains uniform), the authors use big space of the main paper to tell us the conclusions, which are already known by the researchers in the field of distillation. These conclusions are mainly roughly drawn from the simple experiments and lack the theoretical support. In this way, this paper is more or less like an experimental report.\n\n- In the main paper, it is said that the labels that are not well learned by the teacher performance even worse in the student network with direct distillation. But I just wonder if it is also a kind of knowledge to be learned? Is it necessary to just simply eliminate this?\n\n- The proposed two methods are too straightforward and intuitive and not new. So, the contribution of the main method is limited. \n\n\n- Then about the experiments. Does the author mainly focus on the self-distillation? I don’t think in the situation of self-distillation, the problem of the unbalanced performance should be well focused. What about the experiments in transfer learning? There are more poorly performed labels.\n\n\n", "summary_of_the_paper": "- This paper is established by a discovered phenomenon that in distillation, not all the classes’ performance is improved although the student is improved significantly. \n\n- Thus, the authors explore a method that focuses on promoting the performance of the poorly-performed labels, which also lead to the overall promotion while distillation.\n\n- The authors propose two methods to achieve it, one is the distillation with adaptive mixing weights and the other is the distillation with per-class margins.\n", "summary_of_the_review": "- please restate the contribution of the proposed method.\n\n- should need more experiments. like in transfer learning. And need more analysis, like what is the performance of the best-performance group of labels.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BvD7s37pht", "reviewer": "Reviewer_bvA3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n+ The paper is well-written and easy to follow and brings out an interesting issue with distillation\n+ An easy to implement solution is proposed for addressing the issue highlighted.\n- It is not clear if the same classes are \"problematic\" for each model. In other works, if I train a model 10 times do we expect the drop in performance to be in the same 10 classes? \n- I would expect a more detailed statistical analysis to validate that there is indeed a consistent decrease in classes in a statistically significant manner. This should include training multiple models and evaluating their performance.\n- When measuring the worst1/10 performance, we measure the performance on the initial worst 10 classes or there might be a reduction in the performance in other classes? In any case, the average performance is increasing. However, it would be useful to know if the proposed method can negatively impact some classes.\n- AdaAlpha/AdaMargin are not compared with more advanced distillation methods. I understand that the issue highlighted by the authors is important. However, it is not clear if this also affects the more recent distillation approaches (e.g., those performing online distillation, distillation from ensembles, using knowledge from intermediate layers, etc.) as well or if it only concerns vanilla distillation.\n", "summary_of_the_paper": "In this work the authors examine the impact of distillation on certain sub-groups of the data. More specifically, they demonstrate that even though distillation can increase the overall accuracy of a model, it can harm the accuracy on certain subclasses. This includes both self-distillation, as well as regular distillation setups. To overcome this, the authors propose using class-specific weights, that have been tuned according to a holdout set, to tune the importance of each class during distillation, as well as using softmax cross-entropy with margins tuned according to the difficulty of each class. Furthermore, it is also demonstrated that these phenomena can also impact the fairness of the model.\n", "summary_of_the_review": "Overall, I enjoyed reading this paper, which brings out an important issue in distillation. However, I would feel more confident in the reported results if a more detailed statistical analysis was provided, as well as some comparisons with more recent distillation approaches, which leaves me with many open questions. Furthermore, there is no discussion on why distillation brings out this behavior from a theoretical standpoint.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies knowledge distillation and explores why distillation gains are not uniform. Reviewers consistently find this paper an interesting read, but had common concerns on generalizability and limited improvements/contributions.
In general, reviewers mostly gave a score that is below the acceptance threshold, or expressed concerns otherwise. Summing these up, we conclude this paper is of interest to the ICLR audience, but current form is not ready yet for acceptance.  

Summary Of Reasons To Publish:
interesting analysis of the causes of non-uniform gains in distillation 

Summary Of Suggested Revisions:

 (1) the improvements are marginal and (2) the contribution of AdaMargin is limited, (3) generalizability to other KDs
Decision: Reject