Teacher’s pet: understanding and mitigating biases in distillation
Authors: Michal Lukasik
Srinadh Bhojanapalli
Aditya Krishna Menon
Sanjiv Kumar
Venue: Accepted by TMLR ()

Abstract: Knowledge distillation is widely used as a means of improving the performance of a relatively simple ``student'' model using the predictions from a complex ``teacher'' model. Several works have shown that distillation significantly boosts the student's \emph{overall} performance; however, are these gains uniform across all data subgroups? In this paper, we show that distillation can \emph{harm} performance on  certain subgroups, {e.g., classes with few associated samples}, compared to the vanilla student trained using the one-hot labels. We trace this behaviour to errors made by the teacher distribution being transferred to and \emph{amplified} by the student model, and formally prove that distillation can indeed harm underrepresented subgroups in certain regression settings. To mitigate this problem, we present techniques  which soften the teacher influence for subgroups where it is less reliable. Experiments on several image classification benchmarks show that these modifications of distillation maintain boost in overall accuracy, while additionally ensuring improvement in subgroup performance. 

Published in Transactions on Machine Learning Research (11/2022)
Teacher's pet: understanding and mitigating biases in distillation
Michal Lukasik Google Research
miukaskk@google..om
Srinadh Bhojanapalli Google Research
barinadh&@googje
Aditya Krishna Menon Google Research
Sititaakmenon@@ggogge.cco
Sanjiv Kumar Google Research
sanjjvkk@google.com
Reviewed on OpenReview: https: openreview. net/ forum? id- ph3.pph33aa.mm YXpuEb
Abstract
Knowledge distillation is widely used as a means improving the performance of a relatively simple "student" model using the predictions from a complex "teacher" model. Several works have shown that distillation significantly boosts the student's overall performance; however, these gains uniform across all data subgroups? In this paper, we show that distillation can harm performance certain subgroups, e.g., classes with few associaled samples, compared to the vanilla student trained using the one-hot labels. We trace this behaviour to errors made by the teacher distribution being transferred to and amplified by the student model, and formally prove that distillation can indeed harm underrepresented subgroups in certain regression settings. To mitigate this problem, we present techniques which soften the teacher influence for subgroups where it is less reliable. Experiments on several image classification benchmarks show that these modifications of distillation maintain boost in overall accuracy, while additionally ensuring improvement in subgroup performance.
1 Introduction
Knowledge distillation is technique for improving the performance of "student" model using the predictions from "teacher" model. At its core, distillation involves replacing the one-hot training labels with the teacher's predicted label distribution. Empirically, distillation has proven successful for model compression (Bucilă et al., 2006; Hinton et al., 2015), improving the performance of fixed model architecture (Anil et al., 2018; Furlanello et al., 2018), and semi-supervised learning (Radosavovic et al., 2018). Theoretically, several works (Topez-Paz et al., 2016; Mobahi et al., 2020; Tang et al., 2020; Menon et al., 2020; Zhang Sabuncu, 2020; Ji & Zhu, 2020; Allen-Zhu Li, 2020; Zhou et al., 2021; Dao et al., 2021) have studiod how distillation affects learning. Both strands of work further the understanding of when and why distillation helps.
In this paper, we similarly motivated to better understand the mechanics of distillation, but pose slightly different question: docs distillation help all data subgroups uniformly? Or, do its overall gains come at the expense of degradation of performance on certain subgroups? To our knowledge, there has been no systematic study (empirical or otherwise) of this question. This consideration topical given the study of fairness of machine learning algorithms on under-represented subgroups (Hardt al., 2016; Buolamwini & Gebru, 2018; Chzhen et al., 2019), and the study of the tension between average and subgroup performance of common learning algorithms (Samadi et al., 2018; Sagawa al., 2020a; Jones al., 2021).

---

Published in Transactions on Machine Learning Research (11/2022)
Subgroup
Average
Worst
improvement
Accuracy improvement
2
-1.0
Subgroup
Average
-1.5
Worst
Baseline
Adaptive mixing
Baseline
Adaptive margin
Method
Method
(a) CIFAR-100-LT.
(b) ImageNet.
Figure 1: Illustration of the potential deleterious effects of distillation on data subgroups. We train a ResNet-56 teacher on CIFAR-100-LT, a long-tailed version of CIFAR-100 (Cui et al., 2019; Cao et al., 2019) where some labels have only a few associated samples, and ResNet-50 teacher on ImageNet. For each datasot, we self-distill a student ResNet of the same depth. On CIFAR-100-LT, as often observed, distillation helps the overall accuracy over one-hot student training (~2% absolute). However, such gains at significant cost subgroups defined by the individual classes: on the ten rarest classes, distillation harms performance by 1%% Similarly, on ImageNet, distillation harms the average accuracy of the worst-10 classes (as determined by the teacher's performance) by 11.5% Our proposed techniques ($4) can roughly preserve the overall accuracy, while boosting subgroup performance. Notice that the label frequency may not be always consistent with performance or performance improvement in an unbalanced dataset. However, we find them be strongly related: for CIFAR- LT, Spearman rank correlation cocfficient between the ordering according frequencies and according teacher's performance is equal to 0.84.
Our first finding that even in standard settings - e.g., on image classification benchmarks such as CIFAR distillation can disproportionately harm performance on subgroups defined by the individual classes (sce Figure 1). Specifically, compared the teacher model, distillation can worsen the performance on hard classes, and amplisy the gap between worst- and average-class performance. To discern the source of this behaviour, we ablate the teacher and student architectures ($3.2), dataset complexity ($3.3), label frequencies ($3.4). These point to the potential harms of distillation when the teacher confidently mispredicts on a subgroup.
Having identified a potential limitation of distillation, we present two simple techniques to remedy it. These apply per-subgroup mixing weights between the teacher and onc-hot labels, and per-subgroup margins respectively ($4). Intuitively, these limit the influence of teacher predictions on subgroups it models poorly. Experiments image classification benchmarks show that these methods typically maintain boost in overall accuracy, while ensuring more equitable improvement across subgroups.
In sum, this work provides novel insights into distillation performance, with the following contributions:
(i) we identify a hithorto unexplored issue with distillation, namcly, that its improvements in overall accuracy may come at the expense of harming accuracy on certain subgroups (53.1). Such a finding topical given the widespread practical use of generic learning paradigms such as distillation, and the increasing societal applications of learning systems more broadly.
(ii) we ablate potential sources for the above phenomenon ($3.2, $3.3, $3.4), and in the process identify certain characteristies of data (e.g., skewed label distributions) where it manifest.
iii) we provide a theoretical explanation for why distillation can hurt performance on rare subgroups certain settings ($3.8), building on the self-distillation analysis for kernel methods developed in Mobahi ct al. (2020).
(iv) we propose two simple modifications of distillation that mitigate the above problem, based on applying per-subgroup mixing weights and margins (54); these perform well empirically ($5).
2

---

Published in Transactions on Machine Learning Research (11/2022)
At the outset, we note that while the paper's analysis is primarily (though not exclusively) empirical, our focus is on systematic analysis aiming at understanding non-trivial phenomenon, rather than merely empirical comparison. Specifically, $3 is devoted to carefully understanding the extent of, and causes for, the non-uniform gains of distillation. This is in line with works which empirically analyse neural network phenomena (Zhang et al., 2017; Müller et al., 2019; Nakkiran et al., 2020; Neyshabur et al., 2020).
2 Background and related work
Knowledge distillation. Consider a multi-class classification problem over instances X and labels y [L]= = (I....,Li L}. Given a training set S = {(x yn) drawn from some distribution IP, we seek a classifier h: X -> y that minimises the misclassification error Eavg(h) = P(h(x)) + y). In practice, one may learn logits f: T + +R to minimise A(f)= = N Em= Uyyn f(z))), where 8 is a loss function such as the softmax cross-entropy, which for softmax probabilities (a(x)) x exp(fy(x)) is E(y, fx)) = - Po)). One may then classify the sample via h(x)- = arg m fy(r).
Knowledge distillation (Bucilà et al., 2006; Hinton et al., 2015) employs the logits fr: x - R= of a "teacher" model to train a "student" model f²: x + RR, via minimising
1 Σ [11 a) ((En) + a. Edy(( ((N') (znn)), N
Raiie (J)
(1)
n=1
VE(L]
where a E [0, 1]. Here, one converts the teacher logits to probabilities p²: X + Ar, for simplex A, e.g. via a softmax transformation (x) x exp(f) (z)). The second term smooths the student labels based on the teacher's confidence that they explain the sample. The first term includes the original label to prevent incorrect teacher predictions from overwhelming the student. One further important trick is temperature scaling of the teacher logits, so that (z) xx exp(T-1, fīy (zr))- Setting T > O makes p' more uniform, thus preventing overconfident predictions (Guo et al., 2017).
We focus here on logit-based distillation, where the training data for both the teacher and the student is the same set of examples. This setup widely considered when studying fundamental properties of distillation (Mobahi et al., 2020; Menon al., 2021; Dao et al., 2021; Zhou et al., 2021), and is employed by the current to-he-o distillation technique of Beyer et al. (2021). There are other compelling setups to consider, including distillation based on transferring feature representations Heo et al. (2019) self-supervised tasks Xu et al. (2020); Chuanguang Yang (2021); Yang et al. (2021) We leave these for future work, as the study of fairness aspects of distillation remain undeveloped even in the logit-based setting.
Average versus subgroup performance. The above exposition treats the misclassification error Eavg (h) as the fundamental performance mcasure of interest. However, suppose the data contains subgroups 9 {1,..., G). Desining the per-subgroup errors err, (h) = Ph((x) # y [9), we have Eavg( = Egeg -(g)))rr,, (h), which may mask errors on samples with P(g) 0 (Sagawa et al., 2020a;b; Sohoni et al., 2020). To this end, one may instead measure the balanced error (Menon et al., 2013) (ha = Eyes [ST errz(h) which treats the subgroup distribution as uniform, or the worst-subyroup error (Sagawa et al., 2020a;b; Sohoni et al., 2020) Emax (h) = max (fr, which focusses on the worst-performing subgroup. An intermediary the average of the h worst-pe subgroups (Williamson & Menon, 2019): for ith largest per-subgroup error (rr))(h)) Frop- (h)) 1 Σ (n).
&
The definition of 9 is domain-specific consideration. One special case is where each label defines subgroup (i.e., 9 = Y), and P(y) is skewed. In such long-tail settings (Buda et al., 2017), classifiers with good average performance can perform poorly on "tail" labels where PPy) ~ O.
Related work. There is limited prior study that dissects distillation's overall gains per subgroup. Zhao et al. (2020) showed that in incremental learning settings, distillation can be biased towards recently observed classes. We show that even in offline settings, distillation can harm certain classes. Recently, Zhou al. (2021) studied the standard aggregate performance Eavg of distillation, which was tied to a certain subset of "regularisation samples". By contrast, our goal is to understand the subgroup performance of distillation.
3

---

Published in Transactions on Machine Learning Research (11/2022)
Setting
Dataset.
Avg acc
Worst subgroup
acc
Baseline ($3.1)
ImageNett
+0.39
-0.43
Stronger teacher ($3.2)
ImageNet
+0.17
-1.19
CIFAR-100 LT
+2.17
-1.46
Long tail ($3.4)
ImageNet LT
+0.21
-0.32
Fairness ($3.6)
UCI Adult
+3.10
-5.94
CIFAR-10
+0.55
-0.90
Reduce #classes ($3.3)
CIFAR-10 LT
+1.92
-4.40
CIFAR-100
+1.93
-3.33
ImageNet-100
+0.09
-0.06
Table 1: Summary of findings in the ablation analysis of distillation's subgroup performance ($3). In range of different scttings, distillation is seen to hurt the hardest subgroup accuracy (worst-k class accuracy or worst subgroup according to an attribute), despite improving the average accuracy (upper rows). Decreasing number of labels helps improve the hardest classes (bottom row).
Study of the fairness of machine learning algorithms under-represented data subgroups has received recent attention (Dwork et al., 2012; Hardt et al., 2016; Buolamwini & Gebru, 2018; Chzhen et al., 2019). This has prompted dissection of the performance of techniquess such as dimensionality reduction (Samadi et al., 2018), increasing model capacity (Sagawa et al., 2020a), and selective classification (Jones et al., 2021). We follow the general spirit of such works, studying a more delicate setting involving two separate models (the student and leacher), each with their own inductive biases. We present more discussion of related direct in $6. In related effort, Hooker et al. (2019) explore how model compression may harm subgroup performance compared to the original model.
3 Are distillation's gains uniform?
We demonstrate that the gains of distillation are not uniform across subgroups: specifically, considering subgroups defined by classes, distillation harm the student's performance on the "hardest" few classes (63.1). To understand the genesis of this problem, perform ablations (ef. Table 3.1) that establish its existence in settings where there are insufficient samples to model certain classes, either due to the number of classes being large ($3.3), or the class distribution being skewed ($3.4). We then identify that the student may amplify the teacher's errors ($3.7). Next, we corroborate these results for a more general notion of subgroup in a fairness dataset ($3.6). Finally, we show an analysis indicating that this behaviour is potentially result. of the teacher confidently mispredicting on some classes ($3.7). We conclude with formally proving that distillation can indeed harm rarest subgroups in the context of kernelized methods ($3.8).
3.1 Distillation hurts subgroup performance
To begin, we consider the effect of distillation on a standard image classification benchmark, namely, ImageNet. We employ self-distillation (Furlanello et al., 2018) setup, with ResNet-34 teacher and student models, trained with standard hyperparameter choices (see Appendix B). Following Cho & Hariharan (2019), we use carly stopping on the teacher model. We now ask: what is the impact of distillation on the student's overall and per-class performance? The first question has an expected answer: distillation improves the student's ' average accuracy by +0.4% (see the Baseline setting in Table 3.1). Judged by this conventional metric, distillation is thus a success.
A more muanced picture emerges when we break down the source of the above improvement, We compute the per-class accuracies for the one-hot and distillation models, to understand how the overall gains of distillation
4

---

Published in Transactions on Machine Learning Research (11/2022)
Cumulative accuracy gain
0.00
-0.01
-0.02
-0.03
-0.04
-0.05
0
200
400
600
800
1000
Index k
Cumulative gain of ResNet-34 self-distillation on ImageNet. For index k, we compute the gain in average accuracy over the & worst classes. While average accuracy (k = 1000) improves by +0.4%, for k S 40, distillation harms over the one-hot model (evidenced by the negative gain).
Figure
Teacher
Student
Average accuracy
Worst-
Teacher
Student
Average accuracy
Worst-
Teacher
Student
Average accuracy
Worst-
accuracy
accuracy
EffuNet
Res-50
#0.17
-1.19
Res-50
Ree-50
+-0.34
-1.58
Res-34
Res-34
+0.42
-2.0
EffuNet
Res-34
Res-50
Res-4
Res34
Res-18
EffuNet
R8
+0.05
-1.60
Res-50
Res-18
-0.09
-1.00
Res-18
Res-18
-0.13
-3.80
Table 2: Summary of effect of distillation different teacher and student architectures considered for the ImageNet dataset. The comparison is with respect to the one-hot (i.e., non-distilled) student. Distillation consistently hurts accuracy of the worst 10 classes.
are distributed. Figure 2 shows that these gains non-uniform: distillation in fact hurts the worst-k class performance for & < 40. (See Appendix C for detailed per-class breakdown.) Thus, distillation may harm the student on classes that it already finds difficult. At the same time we note that distillation does improve many classes, such as the classes with relatively high accuracies from the onc-hot student.
Given that average accuracy improves, does it matter that performance on subgroups corresponding to the "hardest" classes suffers? While ultimately domain-specific consideration, in general exacerbating subgroup errors may lead to issucs from the fairness perspective. Indeod, we shall see that distillation can also harm in set settings where the subgroups correspond to sensitive variables; see $3.6.
At this stage, it is apposite to ask whether the above is an isolated finding, or indicative of a deeper issue. We thus study each the following in turn: (i) does the finding hold settings beyond self-distillation? (ii) does the finding hold for other datasets, or is it simply due to the idiosyncrasies of ImageNet? (iii) what are some general characteristics of settings where the problem manifest?
3.2 Is distillation biased by the model architecture?
Having begun with a self-distillation setup, we now demonstrate that similar findings hold when the student and teacher architectures differ. Continuing with the ImageNet dataset, in the second row in Table 3.1 we report statistics for the overall average accuracy and average accuracy over the worst classes when distilled ResNet-50 student from stronger teacher: Efficient-Net (Tan & Te, 2021) (more specifically, Efficient-Net L) teacher achieving 85.7% accuracy. Again, we see improved average accuracy and harmed Worst subgroup accuracy, composed of the 10 classes with the lowest. accuracy according to teacher's performance, In Table 2, report statistics when varying teacher and student architectures. The harming of hard class performance under distillation holds across all scenarios: thus, our earlier results were not specific to self-distillation.
For self-distillation settings, smaller models appear to incur greater losses on the worst-class error. When distilling between different architectures (e.g., from ResNet-50 to ResNet-18), we observe that even average accuracy may not improve, as noted in Cho & Hariharan (2019). There however no clear trend between the difference in architectures and drop in worst class performance. Finally, we note that there is little change in the teacher's and student's 's worst-k classes; Figure 7 (Appendix).
5

---

Published in Transactions on Machine Learning Research (11/2022)
3.3 Is distillation biased by large number of classes?
Having seen that Tet consistently demonstrates a performance degradation certain classes, we now repeat the same analysis smaller image classification benchmark. We return to the self-distillation setup, using ResNet-56 models on CIFAR-100. On this dataset, Table 3.1 shows a (perhaps more expected) result: distillation boosts both the average and worst-1 class performance. This indicates that, at a minimum, the behaviour of distillation's performance gains are problem-specffic; on CIFAR, distillation appears a complete win for both the average and subgroup accuracy.
One plausible hypothesis is that the tension between average and subgroup performance only manifests on problems with many labels, which might informally be considered "harder". To confirm this further while fixing the dataset, randomly select 10% of classes from the ImageNet dataset, and only keep examples corresponding to those classes across the train and validation sets. Consistent with the results for CIFAR-1000 we again find that the worst-10 class accuracy is not harmed under distillation (see the right side of Table 3.1 where we report results on RosNet-34 self-distillation...
Next, we also consider the setting where the number of train examples is fixed but the number of classes varies. To this end, we contrast results from experiments on CIFAR-10 and CIFAR-100, and CIFAR-10 LT and CIFAR-100 LT. In each of these pairs of datasets, the set of the train examples is fixed but the number of classes changes from 10 to 100. As reported in Table 3.1, we find that the accuracy over the worst 10% classes according to the teacher does not drop for neither CIFAR-10 nor CIFAR-10 LT upon distillation.
The above indicates that for problems with a few, balanced labels or with sufficiently few labels with respect there may not be a tension between average and worst-subgroup performance under distillation. However, we now show that even for problems with relatively few labels, one may harm subgroup performance if there is label imbolance.
3.4 Is distillation biased by class imbalance?
We now consider a long-tast setting, where the training label distribution P(y) highly non-uniform. Following the long-tail learning literature (Cui et al., 2019; Cao et al., 2019; Kang al., 2020), we construct "long-tailed" to achieve particular
(LT) versions of the above datasets, wherein the training set is down-sampled label skew. For ImageNet, we use the long-tailed version from Liu et al. (2019). For other datasets, we 1 for constant μ and i E [L] (Cui et al., 2019). The ratio of the down-sample labels to follow Pyy i) x jet most to least. frequent class is set to 100.
From the Long tail setting in Table 3.1, we note that on both CIFAR-100-LLY and Image accuracy over the hardest classes drops (we report results from self-distillation using ResNet-56 for CIFAR-100 and ResNet-50 ImageNet). The former is particularly interesting, given that the standard CIFAR-100 shows gains amongst the hardest classes. This provides evidence that for some "harder" problems e.g., where there are insufficiently many samples from which to model particular class there may be a tension between the average and subgroup performance.
3.5 Is distillation biased by hyperparameter choice?
We study the impact of hyper paramcters on the bias of distillation in the case of CIFAR-100 LT and present the results in Figure: 3. We find that, across student architectures, for all temperalures and choices for the mixing parameter a, distillation improves the average accuracy but harms the worst-10 class accuracy.
3.6 Beyond classes: other choices subgroups
Our analysis thus far has focused on subgroups defined by classes. This choice is natural for long-Lailed problems, where it important to ensure good model performance on rare classes (Kang et al., 2020). In other problems, different choices of subgroups may be appropriate. For example, problems arising fairness, one may define subgroups based on certain sensitive attributes (e.g., sex, race). In such settings, does similarly varying gains from distillation across subgroups?

---

Published in Transactions on Machine Learning Research (11/2022)
42.5
1.6
1.4
worst accuracy
mean accuracy
41.5
1.2
1.0
0.8
40.5
0.6
0.4
39.5
0.2
0.0
0.2
0.6
1.0
0.0
0.2
0.4
0.6
0.8
1.0
alpha
alpha
1.4
42.0
1.2
worst accuracy
mean accuracy
41.5
1.0
ResNet-32
41.0
0.8
ResNet-20
Sissiet-32--ResNet..
40.5
0.6
Restiet-32--ResMM
40.0
39.5
0.2
0.0
1.5
2.0
2.5
4.0
4.5
1.0
2.0
2.5 temperature
3.0
4.5
5.0
temperature
Figure Impact of hyper parameters the bias of distillation in the case of CIFAR-100 LT. We find that, across student architectures, for all temperatures and choices for the mixing parameter a, distillation improves the average accuracy but harms the worst-10 class accuracy.
We confirm this can indeed hold on the UCI Adult dataset using random forest models (details in Appendix C.3). This data involves the task of predicting if an individual's income is 50K not, and possesses subgroups defined by the individual's race and sex. Akin to the preceding results, we find that distillation improve overall accuracy, at the expense of degrading accuracy on certain rare subgroups, e.g., Black women; see Table 3.1, and Table 9 (Appendix). This further corroborates our basic observation on the non-uniform distribution of distillation's gains.
significantly
A distinct notion of subgroup was recently considered in Zhou et al. (2021), who identified the impact of certain "regularisation samples" on distillation. These are subset of training samples which were seen to degrade the overall performance of distillation. It is of interest whether such subgroup relates to previously studied subgroups of "hard" classes; e.g., is there abundance of regularisation samples in such subgroups, which might explain the poor performance of distillation? In Appendix C.4, we study the relationship between regularisation samples, and the per-label subgroups from our analysis; we find that, in general, these may be complementary notions.
3.7 Why distillation hurts subgroups: margin view
The above has established that in range of scenarios, distillation can hurt performance subgroups defined by classes. However, firm understanding of why this happens remains clusive. To study this, we consider which showed a stark gap between the average and subgroup (i.e., worst-1 class) performance - and dissect the logits of the teacher and distilled student. (See the Appendix for plots where the teacher and student architectures differ.) Across classes, we seek to understand: (i) how aligned are the student and teacher accuracies? (ii) how reliable are the models' probability estimates? (iii) how do the models' confidences behave?
ResNet-56 self-distillation on CIFAR-100-LT

---

Published in Transactions on Machine Learning Research (11/2022)
For test' example (z, 3) and predicted label distribution p(æ) AL, we thus compute each models' accuracy, log-loss flog((: (V. p(x)) = - Py(2), and maryin (Koltchinski & Panchenko, 2002) Emarg (V)(())) Py(z) maXy' Pyr Note that the latter may be negative if the model predicts the incorrect label for the example. Figure 4 shows these metrics on 10 class buckets: these are created by sorting the 100 classes according to the teacher accuracy, and then creating 10 buckets of classes. Within each bucket, we compute the average of the metric.
Remarkably, for 5 out of 10 class buckets, average margins negative, suggesting that the teacher is often wrong yet confident in predicting these classes. Here, the student accuracy generally worsens compared to the teacher. Further, log-loss incrcases across all buckets (including those where accuracy improves), indicating reduced confidence in the true class of the distilled student. This points at a potential source of the poorer performance on the worst-1 accuracy. Recall that the distilled student's aim is to mimic the teacher's logits on the training samples. This is a proxy to the student's true goal, which mimicking these logits on test samples, so as to generalise similar to the teacher. When such gencralisation happens, the student can be expected to roughly inherit the teacher's per-class performance; in settings like the above, this unfortunately implies poor performance classes with negative teacher margin.
3.8 Why distillation can hurt subgroups: theory
To complement the preceding empirical analysis, we now seek to theoretically establish certain conditions under which distillation can provably harm performance on subgroups. We build upon the recent analysis of self-distlllatio for kernelised models in Mobahi et al. (2020). Specifically, consider the ridge regression problem:
Σ (xw(xx)) - 2n)² A// EW
min
(2)
where F= X w w N and { is a set of target labels. Let f denote the minimizer of (2) where each 2n = Yn- Similarly, let fdlst denote the minimizer where each 2n = f (z(n), i.c., the self-distillation solution. Mobahi et al. (2020) showed how such self-distillation can be interpreted as a regulariser, in the sense pdist implicitly down- weighting low variance directions in the data.
Our analysis holds for the following data distribution: suppose we have data from L subgroups, and targets yn E R.. Let (o:)ke[z) be L orthonormal vectors in RD,, corresponding the input representation x, from different subgroups. Let M, denote all samples with subgroup i, and d; = [M,]. Let G, d. ΣREM. (f(())) yx)² denote the mean error of examples from subgroup i, and eqist _
प. ΣxEM, (f (fdist (#x) yx) the mean
error from subgroup i of the distilled model. Here, the error is normalised by the mean subgroup target v = di ΣkEM, vē.
Thcorem 1. Let the subgroups be in decreasing order according to their frequency with < implying d; > dz. For any < j < L, if /didj, then,
dist S dist
Ej E E
Theorem 1 shows that distillation has a worse effect on subgroup j compared to i. Thus, the less represented directions of the data (e.g. those corresponding to the rare subgroups) would suffer more from the induced regularisation, as long as the regularisation strength À is sufficiently small. This is consistent with the observation of hardest classes (often the rarest classes) suffering the most from distillation (and in particular, self-distillation, as we extensively study in $5).
3.9 Discussion
We have found that distillation's gains in average accuracy can be at the expense of degradation in "hard" class accuracy, which can amplify the gap between worst- and average-class performance. Given a widespread
The choice of test, rather than train, example crucial: overparameterised teacher will likely correctly predict all training samples, thus rendering the above statistics of limited use. To leverage the insights from the above analysis practice, we shall use holdout that can carved from the training set.
8

---

Published in Transactions on Machine Learning Research (11/2022)
student teaccher
student
0.50
teacher
0.6
0.25
Margn
0.00
0.2
student
2
-0.50
-0.75
Class
bucket
(a) Accuracy.
(b) Log-loss.
(c) Margin.
Figure Logit statistics on CIFAR-100 LT, for the teacher and distilled student under a self-distillation setup (ResNet-56 ResNet--56) We show the statisties on 10 class buckets: these are created by sorting the 100 classes according to the teacher accuracy, and then creating 10 groups of classes. As expected, the student follows the general trend of the teacher model. Strikingly, we observe that the teacher model tends to systematically confidently mispredict samples in the higher buckets, thus incurring negative margin; such misplaced confidence is largely transferred to the student, whose accuracy suffers on such buckets. Note that we consider statistics on the test set.
use of distillation, we believe this systematic drop in worst class accuracy is of interest. Further, we emphasise that this is a more worrisome phenomenon than distillation potentially not helping worst class as much as it helps the easiest classes.
4 Improving subgroup performance via adaptive distillation methods
We now study simple means of correcting distillation to prevent the degradation of subgroup performance. These leverage the insight that the behaviour is potentially a result of the teacher confidenlly mispredicling some subgroups. In the following, for concreteness and simplicity, focus on subgroups that given by the individual classes.
4.1 Distillation with adaptive mixing weights
In $3.7, we saw that distillation can hurt on classes where the teacher is inherently inaccurate. Such inaccuracy may in fact be amplified by the student, which is hardly desirable. An intuitive fix is to simply rely less on the teacher for classes where it performs poorly, or is otherwise not confident; instead, the student can simply fall back onto the one-hot training labels themselves. Formally, for per-class mixing weights (a1,...,ar.) [0, 1]4, the student can minimise
N
Retist (
1 N
Σ
[(0
[11-a a.y.)) ((En)) + at. D'E(L]
(x) Eay..ly.... f(r))(
This objective introduces mixing weight ay per-class, which allows to weigh between leacher predictions and one-hot labels for each class independently. By contrast, in the standard distillation setup equation we only have single weight a that is common for all classes.
How do we choose the weights ? In the standard distillation objective equation 1, one only needs to lune a single scalar a, which is amenable to, e.g,, cross-validation. By contrast, equation 3 involves a single scalar for each label, which makes any attempt at grid search infeasible. Following the observations in $3.7, we propose the following intuitive setting of ay given teacher predictions p²:
By max (0. Euly [Tavy (v. p(())))
(4)
Yavg (v. (x)) = xz(()) L
Σ (2).
(5)
y'Fy
9

---

Published in Transactions on Machine Learning Research (11/2022)
Equation 4 places greater faith in the teacher model for those classes which it predicts correctly with confidence, T.e., with large average margin Tavg. When this margin is negative - so that the teacher is incorrect on average, which can occur on classes that are rare in the training set - we set Og = 0, and completely ignore the teacher predictions.
The above requires estimating the expectation Exly [l which requires access to a labelled sample. This may be done using holdout set, which we follow in our experiments. It is worth noting the limitations of such an approach. First, such scheme is, of course, not feasible in settings where the teacher model is used to label large unlabelled pool of samples. While this is important practical setting, further study of alternate means of mitigating the amplification of biases is required. Further, when certain data subgroups only have very few associated samples, it may not be feasible to hold out a part of the data. This issue also affects popular approaches for mitigating subgroup bias, e.g., Sagawa et al. (2020a). It would be of interest for future work to develop methods not requiring a holdout set for hyperparameter estimation. Nonetheless, the next section we report improvements from such a scheme even for long tail datascts.
4.2 Distillation with per-class margins
Our second approach for improving distillation on harder classes is to leverage recent developments long-toil learning, where the goal to improve performance on rare classcs. Spocifically, Menon et al. (2020) proposod variant of the softmax enssopy with label margins Puy/:
fly.f log 1 E Pov cfu
(6)
v'ty
Intuitively, this penalises predicting label s/ instead of y when Pay is large. For training label distribution n, Cao et al. (2019) proposed to set Pay' X exp(my1/4 so that rare labels receive higher weight when misclassified. Khan et al. (2018); Ren et al. (2020); Menon et al. (2020); Wang et al. (2021) showed gains that rare labels are not confused with common ones.
with Pay Fv.
nyy
We adapt such techniques to our setting, with the intuition that we ought to increase the student penalty for misclassifying those "hard" classes that the teacher has difficulty modeling. We thus choose Pyy' 0g where ay the adaptive per-class mixing weight from the previous section. This discourages the model from confusing "hard" labels y with "easy" labels v/, when ay 0y- To avoid division by 0 issue, we add small offset to ag when it becomes 0.
"y
We may understand the effect of Equation 6 by studying how it impacts the predictions, i.c., the optimal predictions in the infinite sample limit, and without a model capacity restriction. We have the following.
student model
ay Lemma 2. Let E be per (6), with Puy' = oxl Let f be the minimiser of Rais( (f) = (=) Then, Vr X. [L], (x) log (x)
ay
Lemma 2 illustrates that using per-class margins encourages the student to mimic the teacher predictions p²(), but with an important modification: we up-weight the probabilities for classes that the teacher does poorly on (a, ~ 0). Intuitively, this makes it easier for the student to improve performance on classes with small teacher margin.
4.3 Relation to existing work
Previous works varied distillation supervision across examples towards improving average accuracy. Proposals included weighting samples based on the ratio (Tang et al., 2019; Zhou et al., 2021), and difference (Zhang et al., 2020) between student and teacher score. Similarly, Zhou et al. (2020b) proposed to only apply distillation on samples the teacher gets correct. In our experiments, we compare against the baseline from Zhou et al. (2021), weighting examples based on teacher and student scores.
10

---

Published in Transactions on Machine Learning Research (11/2022)
Table Summary of student's average accuracy using one-hot and distilled labels. Worst is denotes accuracy over the worst & classos. Global and adaptive temperaturos ay selected using held out dev set. The proposed AdaAlpha technique improves worst class accuracy over vanilla distillation. For Adagn on CIFAR-100 LT we observed divergence during training, presumably due to this method being sensitive to the selection of hyperparaneters, which in turn are estimated on very small number of examples per class.
Dataset
Method
Per-class accuracy statieties Worst-1
Dataset
Method
Por-class accurscy statisties
Mean
Wurst-10
Mean
Worst-1
Worst-10
Worst-100
CIFAR-100 One-hot
73.31 + 0.10
45.67 + 0.94
02.12 + 0.47
CIFAR-TOO LT
One-hot
43.22 + 0.05
0.00 + 0.00
2.33 + 0.40
N/A
Dissillation
76.24 0.10
49:00 1.63
54.66
0.79
Distillation
45.17 + 0.15
0.00 + 0.00
1.80 0.83
N/A
AdaAiplaa
75.43 0:31
49.33 a 0.94
56.42 0.88
AdaAlpha
48.57 0.11
0.67 0.47
4.20 =.24
N/A
AdaMargin
76.15 +: 0.30
61.33=00
56.62 0.35
AdaMangin
Training divergen
ImageNNt
One-hot
75.94 + 0.05
12.00 +: 1.63
21.88 +: 0.76
ImageNes LT
One-bot
45.71 + 0.14
0.00 + 0.00
0.00 0.00
0.68 + 0.12
Distillation
76.14 0:30
14:00 1.41
21.31 0.58
Distillaaton
45.78 + 0.08
0.00 + 0.00
0.00 0.00
0.58 0.02
AdaAlphas
76.15
2.18
23.07 = 0.96
AdaAlpha
45,90 1: 0.18
0.00 1: 0,00
0.00 0.00
0.83 0.09
AdoMargin
76.08 + 0.17
15.00 + 1.00
22.42 + 1.06
AdaMargin
45.98 0.225
0.00 0.00
0.00 0.00
0.85 + 0.11
Other techniques modifying distillation towards improved average accuracy include: going beyoud logit matching and distilling intermediate layer representations (Li et al., 2019; Sun et al., 2020), introducing auxiliary teachers of intermediate capacity for bridging the gap between the teacher and the student (Mirzadeh et al., 2020), distilling using a large unlabel dataset annotated by the teacher (Cotter et al., 2021), or training the student and the teacher jointly (Zhou et al., 2017). It will be of interest for future work to understand the impact of these techniques on the worst-class performance of the student model.
In the special case where subgroups are defined by classes, and classes have an inherent skew, methods from the class-imbalance or long-tail learning literature are also relevant. Prominent strategies to address this problem include techniques that modify the sampling distribution (Kubat & Matwin, 1997; Wallace et al., 2011; Mikolov et al., 2013; Mahajan et al., 2018; Yin et al., 2018; Iscen al., 2021), adjust the decision threshold post-hoc (Fawcett & Provost, 1996; Provost, 2000; Maloof, 2003; King & Zeng, 2001; Collell et al., 2016), adjust the classifier weights post-hoc (Kim & Kim, 2019; Kang et al., 2020), modify the loss function (Xie & Manski, 1989; Morik et al., 1999; Cui et al., 2019; Zhang et al., 2017; Cao et al., 2019; Tan et al., 2020; Ren et al., 2020; Menon et al., 2020; Wang et al., 2021), perform data-augmentattin (Chawla et al., 2002; Müller et al., 2019; Chu et al., 2020; Temraz & Keane, 2022; Zada et al., 2022), ensemble the base classifier (Fan et al., 1999; Chawla ct al., 2003; Galar et al., 2012; Sharma et al., 2020; Zhou et al., 2020a), perform reinforcementtlearriin (Fan et al., 2021), and attach exits for weighting examples by how confidently they are predicted by the intermediate layers (Duggal et al,, 2020). We refer the reader to Zhang et al. (2021) for more comprehensive survey of recent works.
While the above techniques are typically devised in a non-distillation setup, one may nonetheless consider adapting them when training the student model; e.g., one may ensemble many distilled student models in hopes of reducing any degradation on tail classes. We emphasise however that the long-tail learning setup is only one instance of the problem considered in this work, wherein the data comprises subgroups that may have perfectly balanced frequencies (e.g., the standard CIFAR and ImageNet datasets considered above). Many of the above strategics are not immediately applicable in such cases (c.g., strategics that require adjusting the sampling distribution based on frequency would correspond to standard uniform sampling).
5 Results for adaptive distillation methods
We now present results that further corroborate the palential non-uniform gains of distillation, and the ability to mitigate this with the techniques of the previous section. We emphasise here that our goal is expressly not to improve over the stato-of-the-art in distillation techniques; rather, we wish to verify the key principles identified in the preceding study, which considers distillation from novel angle (i.e., in terms of subgroup rather than average performance).
11

---

Published in Transactions on Machine Learning Research (11/2022)
Table 4:
Summary of student's accuracy using one-hot and distilled labels across teacher and student architectures the CIFAR-100 LT dataset. W-k denotes accuracy over the worst & classes. We find that the proposed methods improve worst class performance in all cases.
Per-class accuracy statistics
Tuacher
Student
Method
Mean
W-10
W-20
RosNet-56
RosNet-14
One-hot
39.95
0.80
3.30
Distillation
39.53
1.30
3.35
AdaAlpha
40.34
2,00
4.00
AdaMargin
40.80
2.40
4.80
RosNet-32 ResNNet-14 14
One-hot
39.95
0.80
3.30
Distillation
39.84
1.20
3.00
AdaAlpha
40.21
1.70
3.95
AdaMargin
40.16
2.20
4.80
RosNet-14 RosNet-14
One-hot
39.95
0.80
3.30
Distillation
40.51
0.00
2.05
AdaAlpha
40.79
0.80
2.45
AdaMargin
41.26
1.20
3.70
Table 5: Comparing the Ada* approaches against Zhou et al. (2021): a method weighting examples in distillation based teacher and student scores. Worst-k denotes accuracy over the worst k classes. We find that Zhou et al. (2021) is competitive in terms of improving the mean accuracy, but, contrary to the Ada* methods, does not mitigate the issue of distillation harming accuracy over the worst classes.
Per-class accuracy statistics
Dataset
Method
Mean
Worst-1
Worst-10
Worst=100
CIFAR-100
One-hot
73.31 + 0.10
45.67 = 0.94
52.12 0.47
N/A
Distillation
75.24 1 0.10
49.00 : 1.63
54.65 1 0.79
N/A
Zhou al. (2021)
75.74 0.20
50.33 = 0.47
54.93 0.34
N/A
AdaAlpha
75.43 0.31
49.33
0.94
56.42 0.88
N/A
AdaMargin
75.15 1 0.30
51.33 1 0.47
56.62 0.35
N/A
ImageNet LTT
One-hot
45.71 +:: 0.14
0.00 + 0.00
0.00 + 0.00
0.68 + 0.12
Distillation
45.78 1 0.08
0.00 1 0.00
0.00 0.00
0.58 1 0.02
Zhou et al. (2021)
46.02 = 0.13
0.00 0.00
0.00
0.56 + 0.00
AdaAlpha
45.90 + 0.18
0.00 + 0.00
0.00 0.00
0.83 + 0.09
AdaMargin
45.98 0.25
0.00 1 0.00
0.00 0.00
0.85 0.11
Setup. We report results on the datasets used in $3: CIFAR-100, ImageNet; and long-tailed (LT) versions of the same. For brevity, we report results under self-distillation. (For results with varying architectures, see the Appendix.) Thus, for each dataset, we train one-hot teacher ResNet model, which is distilled to student. ResNet of the same depth. We use ResNet-56 models for CIFAR, and ResNet-50 models for all other datasets. We employ the same typer-parameters as used in $3, except we use non-early stopped teachers for consistency across datascts; see the Appendix for details.
We compare: (i) one-hot training of the student (ii) standard distillation, i.e., minimising Equation (iii) AdaAlpha, our proposed distillation objective with adaptive mixing between one-hot and teacher labels Equation 3, and per Equation 4 (iv) AdaMargin, our proposed distillation objective with adaptive margins (Equation 6), and Pay' a,
oy For each method, we report: (i) the standard meon accuracy
12

---

Published in Transactions on Machine Learning Research (11/2022)
over all classes; (ii) the accuracy over the worst-1 class; and (iii) the mean accuracy over worst-10 (and worst-100 for the LT datasets) classes.
For the Ada-* methods, per $4, creating the label-dependent ay requires estimating the teacher's generalisation performance. To do this, we create a random holdout split of the training set. For non-LT datasets, we randomly split into 80% (new train) 20% (dev). For LT datasets, for each class we hold out & examples into the dev set (k: 50 for Imagenet-LT, k 20 for CIFAR-100-LL)) or half of examples for a class if the total number of per class examples is - 2k. We train initial teacher the new train slice of data, and estimate its per-class performance the holdout dev slice. We then estimate ay as per, e.g., Equation 4.
Table 3 summarises the results for all methods. We make the following observations.
AdaAlpha improves mean accuracy over distillation. The proposed AdaAlpha method consistently improves standard mean accuracy over vanilla distillation. Thus, AdaAlpha does not sacrifice distillation' 's gains on average class performance, which is desirable. We note that while the improvement not significant for most scenarios, our main goal is in improving the worst-accuracy; thus, even a slight improvement in mean accuracy is additional desirable outcome. Other techniques sometimes perform slightly worse than standard distillation on this metric; however, as we now see, this is compensated by gains on other important dimensions.
AdaAlpha improves worst-accuracy over distillation. The proposed method consistently improves the worst-class accuracy compared to standard distillation: thus, the technique largely fulfills their design goal of improving performance on "hard" classes. Contrasting this observation, notice on ImageNet-LT dataset how Zhou al. (2021) can improve average accuracy, obtaining the best mean accuracy, however at the cost of worsened worst-classes accuracy, as shown in Table 5. Finally, we notice that the improvement on the worst classes is much more significant than on average for AdaAlpha. This is expected and desirable, as we set out to target the former in this work. Significantly improving both the worst and average class accuracies would be a desirable future work.
Comparison of AdaAlpha and AdaMargin. In the Appendix, we report per class statistics for CIFAR- 100 LT. AdaMargin flattens both the margin and log-loss distributions, reducing confidence on the poorly classified, tail classes. AdaAlpha consistently increases log-loss across classes, and improves margins buckets, leading to a positive margin one bucket where all other methods give negalive margins. Intuitively, AdaMargin tries to more aggressively control the worst-class accuracy; when this succeeds, there is a large payoff, but there is also greater risk of overfitting, In Appendix C.6, we confirm that the success of AdaAlpha is not immediately replicated by simpler baselines such as shuffling temperatures and removing distillation from the hardest labels.
few
Varying teacher and student architectures In Table 4 we vary the teacher and student architectures in the distillation experiments on the CIFAR-100 LT dataset. We find that the Ada* methods improve the worst class performance in all cases. Particularly the AdaMargin method proves effective, and improves both the average and the worst-10 accuracy by the highest margin.
6 Discussion and future work
Our goal of ensuring equitable performance across classes can be seen encouraging fairness across subgroups defined by the classes. This is subtly different to the classical fairness literature (Calders & Verwer, 2010; Dwork et al., 2012; Hardt et al., 2016), wherein the subgroups are defined by certain sensitive attributes. Broadly, fairness techniques attempt to learn models that predict the target label accuralely, but the subgroup label poorly; these are inadmissible for our setting, wherein the two labels exactly coincide. Ensuring fairness across subgroups defined by the classes has been studied Mohri al. (2019); Williamson & Menon (2019); Sagawa et al. (2020a). Adapting such algorithms to the distillation setting is of interest for future work. More broadly, furthering the understanding of when distillation can hurt under-represented subgroups is of interest.
13

---

Published in Transactions on Machine Learning Research (11/2022)
References
Allen-Zhu, Z. and Li, Y. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. CoRR, abs/2012..00 2020. URL https://arxiv, org/orgg//a....o.oo..o httpaa/...//..///////./
Anil, R., Pereyra, G., Passos, A., Ormandi, R., Dahl, G. E., and Hinton, G. E. Large scale distributed neural network training through online distillation. In International Conference on Leorning Representations, 2018.
Beyer, L., Zhai, X., Royer, A., Markeeva, L., Anil, R., and Kolesnikov, A. Knowledge distillation: A good teacher is pationt and consistent, 2021.
Bucilä, C.,, Caruana, R., and Niculescu-Mizil, A. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '06, pp. 535-541, New York, NY, USA, 2006. ACM.
Buda, M., Maki, A., and Mazurowski, M. A. systematic study of the class imbalance problem convolutional neural networks. arXic:1710. 0.3381 [cs, statj, October 2017.
Buolamwini, J. and Gebru, T. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Friedler, S. A. and Wilson, C. (eds.), Conference on Fairness, Accountability, and Transparency, volume 81 of Proceedings of Machine Learning Research, pp. 77-91, New York, NY, USA, 23-24 24 Feb 2018 PMLR.
Calders, T. and Verwer, S. Three Naive Bayes approaches for discrimination-fres classification. Data Mining and Knowledge Discovery, 21 (22)2777-2 2010.
Cao, K., Wei, C., Gaidon, A., Aréchiga, N., and Ma, T. Learning imbalanced datasets with label-distri aware margin loss. In Advances in Neural Information Processing Systems 32, pp. 1565-1576, 2019.
Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P. SMOTE: Syuthetic minority over-sampling technique. Journal of Artificial Intelligence Research (JAIR), 16:321-357, 2002.
Chawla, N. V., Lazarevic, A., Hall, L. O., and Bowyer, K. W. Smoteboost:
Improving prediction of the minority class in boosting. In Lavrac, N., Gamberger, D., Blockeel, H., and Todorovski, T.. (eds.), Knowledge Discovery in Dotabases: PKDD 2003, 7th European Conference Principles and Practice of Knowledge Dobtat-Dubroovikt; Croatia, Seplember 22-26, 2003, Proceedings, volume 2838 of Lecture Notes in Computer Science, pp. 107-119. Springer, 2003. doi: 10.1007/9788 1977/9778----- 5-40-3904---44 12. URL https://dok: http::::/dd...or///// org/10 0001100777777 1-007/97-335444 50555400000022-11
Discovery in Dalabases,
Cho, J. H. and Hariharan, B. On the efficacy of knowledge distillation. In 2019 IEEE/CVF International Conference Computer Vision (ICCV), pp. 4793-4801. 2019.
Chu, P., Bian, X., Liu, S., and Ling, H. Feature space augmentation for long-tailed data. In Computer Vision - ECCV 2020: 16th Europeon Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX, pp. 694-710, Berlin, Heidelberg, 2020. Springer-Verlag. ISBN 978-3-030-58525-0 doi: 10.1007 978-3-030-58526-6 41. URL httppp///oo..ogo/.0..00///910000
6_41.
Chuanguang Yang, Zhulin An, L. C. Y. x. Hierarchical self-supervieed augmented knowledge distillation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI), pp. 1213, 2021.
Chzhen, E., Denis, C., Hebiri, M., Oneto, L.,, and Pontil, M. Leveraging labeled and unlabeled data. for consistent fair binary classification. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32, pp. 12760-12770. Curran Associates, Inc., 2019.
Collell, G., Prelec, D., and Patil, K. R. Reviving threshold-move a simple plug-in bagging ensemble for binary and multiclass imbalanced data. CoRR, abs/1600 (666600668,,.. 2016.
14

---

Published in Transactions on Machine Learning Research (11/2022)
Cotter, A., Menon, A. K., Narasimhan, H., Rawat, A. S., Reddi, S. J., and Zhou, Y. Distilling double descent, 2021. URL. https://arxiv...
Cui, Y., Jia, M., Lin, T.-Y., Song, Y., and Belongie, S. Class-balanced loss based on effective mumber of samples. In CVPR, 2019.
Dao, T., Kamath, G. M., Syrgkanis, V., and Mackey, L. Knowledge distillation as semiparametric inference. In International Conference Learning Representations, 2021. URL id-m4UCf24r0Y.
Duggal, R.,, Freitas, S., Dhamnani, S., Chau, D. H., and Sun, J. Elf: An early-exiting framework for long-tailed classification, 2020, URL aittps///arxiv orggabs/2006666 11979.
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. Fairness through awareness. In Innovations in Theoretical Computer Science Conference (ITCS), pp. 214-226, 2012.
Fan, S., Zhang, X., and Song, Z. Reinforced knowledge distillation: Multi-class imbalanced classifier based on 2021. ISSN 0925-2312. doi: neucom 2021. 08.1.08.000 040.
policy gradient reinforcement learning. Neurocomput., 463(C))422-436 10.1016/j. www.eee.............. 2021.08.040 URL attps://dol org/ 10. 1016/ www.o/////..c................oo.
Fan, W., Stolfo, S. J., Zhang, J., and Chan, P. K. Adacost: Misclassification cost-sensitive boosting. In Proceedings of the Sisteenth International Conference on Machine Learning, ICML '99, pp. 97-105, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1558606122.
Fawcelt, T. and Provost, F. Combining data mining and machine learning for effective user profiling. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 8-13. AAAI Press, 1996.
Furlanello, T., Lipton, Z. C.,, Tschannen, M., Itti, L.,, and Anandkumar, A. Born-again neural networks. Tn Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässann Stockholm, Sweden, July 10-15, 2018, pp. 1602-1611, 2018.
Galar, M., Fernandez, A., Barrenechea, E., Bustince, H., and Herrera, F. A review on ensembles for the class imbalance problem: Bagging-, boosting-, and hybrid-based approaches. IEEE Transactions Systems, Man, and Cybernetics, Part C (Applications and Reviews), 42(4):463 484, 2012. doi: 10.1109/TSMCC. 2011.2161285.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, pp. 1321 1330, 2017.
Hardt. M., Price, E.,, and Srebro, N. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems (NIPS), December 2016.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., and Choi, J. Y. A comprehensive overhaul of feature distillation. CoRR, abs/1904.01866, 2019. URL http://arxivv org/ hbap//ba///a aatp//////11/....1 01866.
Hinton, G. E., Vinyals, O., and Dean, J. Distilling the knowledge in neural network. CoRR, abs/ hbs//503.02531, 2015.
Hooker, S., Courville, A. C., Dauphin, Y. N., and Frome, A. Selective brain damage: Measuring the disparate impact of model pruning. CoRR, abss/191.05524 2019. URL http://arxiv.. http:://axxi..or/abbs.... htrg/abs/1990 1991 05248.
Iscen, A., Araujo, A., Gong, B., and Schmid, C. Class-balanced distillation for long-tailed visual recognition. 2021. URL https://arzit wrt. ///.//114...00
CoRR,
Ji, G. and Zhu, Z. Knowledge distillation wide neural networks: Risk bound, data efficiency and imperfect teacher. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, 2020.
15

---

Published in Transactions on Machine Learning Research (11/2022)
Jones, E., Sagawa, S., Koh, P. W., Kumar, A., and Liang P. Selective classification can magnify disparities across groups. In International Conference on Learning Representalions, 2021.
Kang, B., Xie, S., Rohrbach, M., Yan, Z., Gordo, A., Feng, J., and Kalantidis, Y. Decoupling representation and classifier for long-tailed recognition. In Eighth International Conference on Learning Representations (ICLR), 2020.
Khan, S. H., Hayat, M., Bennamoun, M., Sobel, F. A., and Togneri, R. Cost-sensitive learning of deep feature representations from imbalanced data. IEEE Transactions on Newral Networks and Learning Systems, 29 (8):3573 -57837773587 2018. doi: 10.1109/TNNLS. TNNLS. 2017..22221.
Kim, B. and Kim, Adjusting decision boundary for class imbalanced learning, 2019.
King, G. and Zeng, Logistic regression in rare events data. Political Analysis, 92):13771633, 2001.
Koltchinskii, V. and Panchenko, D. Empirical margin distributions and bounding the generalization error of combined classifiers. Ann. Stalist., 30(1):1-50 50, 02 2002.
Kubat, M. and Matwin, S. Addressing the curse of imbalanced training sets: One-sided selection. In Proceedings of the International Conference on Machine Learning (ICML), 1997.
Li, H.-T,, Lin, S.-C., Chen, C.-Y., and Chiang, C.-K. Layer-level knowledge distillation for deep neural network learning. Applied Sciences, (10):1996,, 2019.
Liu, Z., Miao, 7.,, Zhan, X., Wang, J., Gong, B., and Yu, S. X. Large-scale long-tailed recognition an open world. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 2537-2546. Computer Vision Foundation IEEE, 2019.
Lopez-Paz, D., Schölkopf, B., Bottou, I.., and Vapnik, V. Unilying distillation and privileged information. International Conference on Learning Representations (ICLR), November 2016.
Loshchilov, and Hutter, F. Sgdr: Stochastic gradicnt descent with warm restarts, 2017.
Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., and van der Maaten, Exploring the limits of weakly supervised pretraining. Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (eds.), Computer Vision ECCV 2018, pp. 185 201, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01216-8.
Maloof, M. A. Learning when data sets imbalanced and when costs unequal and unknown. In TCML 2003 Workshop Learning from Imbalanced Datasets, 2003.
Menon, A. K., Narasimhan, H., Agarwal, S., and Chawla, S. On the statistical consistency of algorithms for binary classification under class imbalance. In Proceedings of the 30th International Conference Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, pp. 603-611, 2013.
Menon, A. K., Jayasumana, S., Rawat, A. S., Jain, H., Veit, A., and Kumar, S. Long-tail learning via logit adjustment, 2020.
Menon, A. K., Rawal, A. S., Reddi, S., Kim, S., and Kumar, S. A statistical perspective on distillation. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7632-7642. PMLR, 18-24 Jul 2021. URL https: h/aps://preae http///pceeenn..ceee/ mlr, www.spressiiii h0t11/1mmmmm11111 html.
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems, NIPS'13, pp. 3111-3119, Red Hook, NY, USA, 2013. Curran Associates Inc.
16

---

Published in Transactions on Machine Learning Research (11/2022)
Mirzadch, S. I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., and Ghasemzadch, H. Improved knowledge distillation via teacher assistant. Proceedliggs of the A AATT Conference Artificial Intelligence, 34(04): 5191 5198, Apr. 2020. doi: 10.1609/ 11669//aai.vaa veai..34104.5963. URL https://ojs. aaai org/ index. php/ AAAT/ atticle/vies/5950
Mobahi, H., Farajtabar, M., and Bartlett, P. L. Self-distillation amplifies regularization in Hilbert space. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, Н. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Mohri, M., Sivek, G., and Suresh, A. T. Agnostic federated learning. In International Conference on Machine Learning, 2019.
Morik, K., Brockhausen, P., and Joachims, T. Combining statistical learning with a knowledge-basse approach
case study in intensive care monitoring. In Proceedings of the Sirleenth International Conference Machine Learning (ICML), pp. 268 277, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1-55860-612-2.
Miller, R., Kornblith, S., and Hinton, G. E. When does label smoothing help? In Advances in Neural Information Processing Systems 82, pp. 4705, 2019.
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, Deep double descent: Where bigger models and more data hurt. In International Conference on Learning Representations, 2020.
Neyshabur, B., Sodghi, H., and Zhang, C. What is being transferred in transfer learning? In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 512-502 1233.2 Curran Associates, Inc., 2020.
Provost, F. Machine learning from imbalanced data sets 101. In Proceedings of the AAAT-2000 Workshop Imbalanced Dala Sels, 2000.
Radosavovie, I., Dollár, P., Girshick, R. B., Gkioxari, G., and He, K. Data distillation: Towards omni- supervised learning, In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18- 18-22, 2018, pp. 4119 4128, 2018.
Ren, J., Yu, C., sheng, 8., Ma, X., Zhao, H., Yi, S., and Li, h. Balanced meta-softımax for Laiged visual recognition. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 4175-4186. Curran Associates, Inc., 2020.
Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations (ICI.R), 2020a.
Sagawa, S., Raghunathan, A., Koh, P. W., and Liang, P. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning (ICML), 2020b.
Samadi, S., Tantipongpipat, U., Morgenstern, J. H., Singh, M., and Vempala, S. The price of fair pea: One extra dimension. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa- Miin-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31, pp. 10976 10987. Curran Associates, Inc., 2018.
Sharma, S., Yu, N., Fritz, M., and Schiele, B. Long-tailed recognition using class-balanen balanced experts. CoRR, abs/2004.03776, 2020. URL. https://arx/: //triv..///vvv//////// org/ aaa///bs/2000// 2004. 03706.
Sohoni, N., Dunnmon, J., Angus, G., Gu, A., and Ré, C. No subclass left behind: Fino-grained robustness in coarso-grained classification problems. In To appear Conference on Neural Information Processing Systems (NewrIPS), 2020.
17

---

Published in Transactions on Machine Learning Research (11/2022)
Sun, Z., Yu, H., Song, X., Liu, R.,, Yang, Y., and Zhou, D. MobileBERT: compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Associalion for Computational Linguistics, pp. 2158 2170, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/vo /2l//200ccccmmaaii 195. URL
org/2020. acl- mma...n... 195.
Tan, J., Wang, C., Li, B., Li, Q. Ouyang, W., Yin, C., and Yan, J. Equalization loss for long-tailed object recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 11659-11668, 2020.
Tan, M. and Le, Q V. Ffficientnetv2: Smaller models and faster training. 2021. doi: 10445550/ARXIV221004 00298. URL https:// arxiv. org/org///....oooo.... aaaaabb/221000 0tt.//214.00022900 00298.
Tang, J., Shivanna, R., Zhao, Z., Lin, D., Singh, A., Chi, E. H., and Jain, S. Understanding and improving knowledge distillation. CoRR, 23s/2000.00533,, 2020.
Tang, S., Feng, I.., Shao, W., Kuang, Z., Zhang, W., and Lu, Z. Learning efficient detector with semi- supervised adaptive distillation. In 30th British Machine Vision Conference 2019, BMVC 2019, Cardiff, UK, Seplember 9-12, 2019, pp. 215. BMVA Press, 2019. URL papers/0145-papar pdf.
Temraz, M. and Keane, M. T. Solving the class imbalance problem using a counterfactual method for data augmentation. Machine Learning with Applications, 9:100375, 2022. ISSN 2666-8270. doi: hitps:
//ddd..org/10.1111/...bbb m02.2.0002..00375. URL https: http:://wwww..ceecieeeeee.. http://wcccciieeee..eooeeeeearaa com/sciienc/aartice///ipii $2666882002000655..
Van Horn, G. and Perona, P. The devil is in the tails: Fine-grained classification in the wild. arXiv preprinl 1rXii:1709.000 01450, 2017.
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. The inaturalist species classification and detection dataset. In 2018 IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8769 8778, United States, December 2018. Institute of Electrical and Electronics Engineers (IEEE). ISBN 978-1-5396-6421-6 doi: 10.1109/CVPR.20188 409400000..... URL http:// /cvp:2//vvp2208.. thecvf. com/ 2018 IEEE/ /EEE/CVF /VV Conference on Computer Vision and Pattern Recognition, CVPR 2018 Conference date: 18-06-2018 Through 22-06-2018.
Wallace, B., K.Small, Brodley, C., and Trikalinos, T. Class imbalance, redux. In Proc. ICDM, 2011.
Wang, J., Zhang, W., Zang. Y., Cao, Y., Pang, J., Gong, T., Chen, K., Liu, Z., Loy, C. C., and Lin, D. Seesaw loss for loigetailed instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, 2021.
Williamson, R. C. and Menon, A. K. Fairness risk measures. In Proceedings of the 36th International Conference on Machine Learning, 1CML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 6786- 6797, 2019.
Xie, Y. and Manski, C. F. The logit model and response- basoonse-based samples. Sociological Methods & Research, 17 (3):283-300) 302, 1989.
Xu, G., Liu, Z., Li, X., and Loy, C. C. Knowledge distillation meets self-supervision. CoRR, abs/2006.0711 /211100000...1,, 2020. URL
org/ absaaass//aa 20000 07114.
Yang, C., An, 7.,, Cai, I.., and Xu, Y. Knowledge distillation using hierarchical self-supervision augmented distribution. CoRR, Abs/2109.03077 2021. URL https////xxvv..rr//aaa 200.0...55..000........
Yin, X., Yu, X., Sohn, K., Liu, X., and Chandraker, M. Feature transfer learning for deep face recognition with loig-tail data. CoRR, abs/1803.009 2018.
18

---

Published in Transactions on Machine Learning Research (11/2022)
Zada, S., Benou, I., and Irani, M. Pure noise to the rescue of insufficient data: Improving imbalanced classification by training on random noise images. In Chaudhuri, K., Jegelka, S., Song, T.., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 25817-25833. PMLR, 17-23 Jul 2022. URL https: bttps://proceedingg mlr. www.w...eeeseeeee. www.seresssvveeeee v 162 htt///adaa222aaa.a. html.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, Frence, April 24-26, 2017, Conference Track Proceedings. OpenReview.net .ect, 2017.
Zhang, Y., Lan, Z., Dai, Y., Zeng, F., Bai, Y., Chang, J., and Wei, Y. Primo-aware adaptive distillation. In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J.-M. (eds.), Computer Vision ECCV 2020, pp. 658 674. Springer International Publishing, 2020. ISBN 978- 778-3-030-58522-7.
Zhang, Y., Kang, B., Hooi, B.; Yan, S., and Feng, J. Deep long-tailed learning: A survey. CoRR, /rxivv.vvvv////////////////aatv/
abs abs/2110.04596, 2021. URL
04596.
Zhang, Z. and Sabuncu, M. R. Self-distillation as instance-specific label smoothing. In Larochelle, H., Ranzato, M., Hadsell, R.,, Balcan, M., and Lin, H. (eds.), Advances Neural Information Processing Systems 39: Anmual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.
Zhao, B., Xiao, X., Gan, G., Zhang, B., and Xia, S. Maintaining discrimination and fairness class incremental learning. In 2020 IEEE/CVE Conference on Computer Vision and Pollern Recognition, pp- 13205 13214. IEEE, 2020.
Zhou, B., Cui, Q. Wei, X.-.,, and Chen, Z.-M. BBN: Bilateral-branch network with cumulative learning for long-tailed visual recognition. pp. 1-8, 2020a.
Zhou, G., Fan, Y., Cui, R., Bian, W., Zhu, X., and Gai, K. Rocket launching: A universal and efficient framework for training well-performing light net, 2017. URL https:// org/ abs/ /1t.///0.... htt///108.0000000 .44066.
Zhou, H., Song, L., Chen, J., Zhou, Y., Wang, G., Yuan, J., and Zhang, Q. Rethinking soft labels for knowledge distillation: A bias variance tradeoff perspective. In International Conference on Learning Representations, 2021.
Zhou, Z., Zhuge, C., Guan, X., and Liu, W. Channel distillation: Channel-wise attention for knowledge distillation. CoRR, abs/2006.01688 2020b. URL btttps://arxiv.or//aaa 01683.
19

---

Published in Transactions on Machine Learning Research (11/2022)
A Proofs
Proof of Lemma 2. We may write
1 Σ Oyy (x)-
= log
ag
4'
log
1
e
baa - ma fl(()))-f(())
v'zy
= log
1 +
y'zy
efix)
log
Eyeryy ex
= (z) + In ay. The population loss is where =
Raia (f) Ex
= Er (L)('(()) [xx)]
where (x) & exp(fy(x)). Thus, at optimality we must have p(x) = (x)), or F)(x) = (xg By
definition of f, thus that fz(x) log (x)) log Bt log p)(x)
]
R,
B Details of experiments
B.1 Architecture
We use ResNet with batch norm (He et al., 2016) for all our experiments with the following configurations. For CIFAR, we experiment with ResNet-56 and ResNet-32. For ImageNet, we use ResNet-50. We list the architecture configurations in terms (Mayer: Datter, strido) corresponding to cach RosNet block in Table 6.
Architecture
Configuration: [Miayarr Nälter, stride)]
CIFAR ResNet-32
[55, 16, 1), (5, 32, 2), (5, 64, 2)]
CIEAR ResNet-56
[(9, 16, 1), (9, 32, 2), (9, 64, 2)]
ImageNet ResNet-11
[(2, 64, 1), (2, 128, 2), (2, 256, 2), (2, 512, 2)]
ImageNet ResNet-34
[33, 64, 1), (4, 128, 2), (6, 256, 2), (3, 512, 2)]
ImageNet ResNet-50
[3, 64, 1), (4, 128, 2), (6, 256, 2), (3, 512, 2
Table ResNet Architecture configurations used in our experiments (He et al., 2016). [*] Note that ImageNet ResNet-50 uses larger blocks with convolutional layers per residual block compared to ResNet-18 and 34. We refer to He et al. (2016) for more details.
B.2 Training set
For all datasets, we train using SGD and weight decay 10-4 for CIFAR, and 0.5 x 10-4 for Imagenet datasets. We have the following dataset specific settings.
CIFAR-100. We train for 450 epochs with an initial learning rate of 1.0, with a lincar warmup the first 15 epochs, and an annealed learning rate schedule. We drop the learning rate by a factor of 10 at epochs number: 200, 300 and 400. We mini-batch size of 1024. We SGD with Nesterov momentum of 0.9.
20

---

Published in Transactions on Machine Learning Research (11/2022)
Log loss Bottom 10%
0.4
2.0
Margin
0.0
Accuracy
0.2
1.0
dista student one hot
-0.1
distill
student one hot
0.5
student one hot
teacher (early
teacher (early
teacher learly stop)
class
bucket
Tall bucket
(a) Accuracy (tail classes).
(b) Log-loss (tail classes).
(c) Margin (tail classes).
Figure Logit statistics for ResNct-50 self-distillation on ImageNet, for the (early-stopped) teacher, self- distilled student, and one-hot (non-distilled) student. Per Figure we first create 10 class buckets, We zoom in the "tail" bucket (comprising the 100 "hardest" classes), and further split them into 10 "tail sub-buckets". As in Figure 4, the teacher is seen to confidently mispredict most samples on the last few buckets, with such misplaced confidence being transferred to the student.
For our distillation experiments we train only with the cross-entropy objective against the teacher's logits. For each method we find the best lemperature from the list values: {1, 2, 3, 4,5).
ImageNett. We train for 90 epochs with an initial learning rate of 0.8, with a linear warmup in the first 5 epochs, and annealed learning rate schedule. We drop the learning rate by factor of 10 at epochs number: 30, 60 and 80. We use a mini-batch size of 1024.
For our distillation experiments we train with the distillation objective as defined in Equation 1 setting a = 0.2. For each method we fix the temperature to 0.9.
Long-tail (LT) datasets. We follow setup as the non-long tail version, except for the learning rate schedule, which change follow the cosine schedule (Loshchilov & Hutter, 2017).
C Additional Experiments
We present additional experiments to those in the body.
C.1 Further varying datasets and model architectures
On Imagenet, we summarisc statistics for three models: the carly-stopped teacher, distilled student, and the one-hot (non-distilled) student. As with CIFAR-100-LT, we sort classes by leacher accuracy, and bucket them into 10 groups. Owing to the larger number of labels, we further zoom into the "tail" bucket (comprising the 100 "hardest" classes), and split them into 10 sub-buckets. From Figure 5, the distilled student performs worse than its one-hot counterpart on the last bucket; this is in keeping with our results Table 3.1.
Figure 6 shows logit statistics for additional settings to considered in the body, On ImageNet-LTT, e.g., we see again that the margin of the teacher model systematically worsens and becomes negative the hardest. classes.
In Table 7 we report results from the inherently tailed iNaturalist 2018 dataset (Van Horn & Perona, 2017). Our observations made for other considered datasets hold: adaptive margin method improves over both one hot and plain distillation terms the worst class accuracy. We also observe, how the average accuracy improves.
C.2 Logit plots under Ada-* methods
Figure shows the logit statistics under the proposed AdaMargin and AdaAlpha methods on CIFAR-100 LT. We see that AdaMargin can generally improve the student margin and accuracy the hardest classes, while
21

---

Published in Transactions on Machine Learning Research (11/2022)
CIFAR-1 100 res56
CIFAR-100 rec56
2.5
student
studintt student one hot teacher
student one hot teacher
Margn
2.5
0.4
0.5
5
9
8
Class bucket
Class bucket
Imagenet res50
0.9]
Imagenet
0.9)
2.0
Student distil
Student one-hot
0.8
student distil
student cre-hot
1.5
0.6
0.4
0.5
0.2
0.0
3
6
Class bucket
Class bucket
Imagenet-LLT res50 res18 IT 0.9]
Imagenet-LT res50
resl 8 IT 0.91
studient ditill
student distil
student
Log 100-
0.2
0.0
-0.4
bucket
Figure 6: Logit statistics for the teacher, student with onc-hot labels, and student with distilled labels across: datasets and architectures.
1.0
Distilled accuracy
Distilled accuracy
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.2
0.8 One-hot accuracy
0.4
0.6
1.0
0,2
0.4
0.6
0.8
1.0
One-hot accuracy
Figure 7: Per-class accuracies for one-hot and self-distileed ResNet-18 (left) and ResNet-34 (right) on Netg The diagonal denotes classes where both models achieve the same accuracy. Distillation tends to worsen performance on "hard" classes for the one-hot model, ie., those with low accuracy (red rectangle).
also reducing the log-loss. This confirms that the gains of the method come from improving behaviour of the scores on these hard classes.
C.3 Results on Adult dataset
We report the results of an experiment on the UCT Adult datasct. This data compriscs 48K examples, with the target being a binary label denoting whether or not an individual has income 2 50K. The data is mildly imbalanced, with 24% of samples being positive.
22

---

Published in Transactions on Machine Learning Research (11/2022)
Per-class accuracy statistics
Method
Mcan
Worst 20
Top 20%
A20
One-hot
53.00
5.00
100.00
48.00
Distill
52.67
5.00
100.00
47.67
AdaMargin
53.33
8.33
98.33
45.00
AdaAlpha avg
54.67
13.33
100.00
41.33
Table 7: Self-distillation experiments (from Resnct-50 to Resnet-50) on the iNaturalist dataset Van Horn et al. (2018) with student's average accuracy using one-hot and distilled labels. Worst 20 denotes accuracy averaged over worst 20 classes A20 denotes the difference between the mean accuracy and the worst 20 classified classes. The proposed AdaMargin technique improves mean and worst-class accuracy over both one-hot training and standard distillation.
3.5
Medel
ibudent diet. AdaA atudent clet AdsMargn
itudent cleit Aden
2,5
shuterr distil Ae
studenn distll Acladihh
atudent clat standard teacher
0.30
atudent dual standars
atudent dlst AdaMargin
Accumicy
chiteer di-iil stanns
5.25
teacher
Beacher
-0.30
-0.75
Class bucket
Class backet
Class burkee
(a) Accuracy.
(b) Log-loss.
(c) Margin).
Logit statistics for ResNet-56 ceff-distillation on CIFAR-100 LT, for the teacher, self-distileed student, and our adaptive methods. Per Figure 4, we create 10 class buckets. AdaMargin flattens both the margin and log-loss distributions. AdaAlpha increases log loss across classes, while improving margins on few buckets, including flipping the bucket. 5 to have positive margin.
Figure 8:
Inspired by Dao et al. (2021), ) we consider a random forest based distillation setup: we use a teacher model that is a random forest classifier comprising 500 trees with a maximum depth of 20, and a student model that is a random forest reyressor comprising tree with a maximun depth 20. The teacher model achieves test (balanced) accuracy of 81.8%.
We perform distillation by feeding the student model the teacher's prediction scores, mixed in with the binary training labels with a weight a 0.9. Distillation improves the student's overall (balanced) accuracy significantly, from 76.2% to 79.3%. However, this gain is not distributed uniformly: using per-label subgroups, we find that distillation helps the positive class by +7.4%, but hurts the negative class by - -1.2% While by itself suggestive of asymmetry in distillation performance, the data admits an arguably more natural subgroup creation, based on available sex and sex features. For example, we find that amongst low-income males, distillation hurts by -2.2%; further restricting to those who are Asian Pacific-Islander, the degradation is -5.9%. This confirms that in scenarios where fairness may be consideration, naïve application of distillation may be inadmissible.
C.4 Analysis of regularisation samples
Recently, Zhou et al. (2021) proposed the notion of regularisation samples to understand how distillation's 's performance can be improved. In brief, such samples correspond lo cases where the teacher's prediction on the training label is less than the distilled student's prediction on this label; these may be shown to correspond to cases where a certain notion of "variance reduction" dominates a notion of "bias reduction". Given our analysis above of the asymmetric effects of distillation on certain subgroups, not these relate to the presence of regularisation samples in these groups
natural to consider whether
23

---

Published in Transactions on Machine Learning Research (11/2022)
Table 8: Difference between distillation and one-hot performance on Adult dataset. Here, subgroups are defined by the sex and label. A refers to the difference between the distilled and onc-hot student's 's accuracy the subgroup.
SCX
label
Male
-2.222
Female
0
0.393
Female
2.373
Male
8.384
Table 9: Difference between distillation and one-hot performance Adult dataset. Here, subgroups are defined by the sex, race, and label. A refers to the difference between the distilled and one-hot student's accuracy on the subgroup.
race
SCX
label
A
Amer-Indann Essiimo
Female
-6.6667
Saaa--ac-Tslandeee
Male
-5.941
Other
Male
-4.347
Black
Female
-2.381
White
Male
-2.248
Black
Male
-1.639
Black
Female
-0.140
Saian-Pac-sslader
Female
0.000
White
Female
0.388
Other
Female
2.439
White
Female
2.724
Female
6.349
Male
6.493
Asian-Pac Pac-Islander
Female
7.692
White
Male
7.796
Black
Male
9.489
Other
Male
15.000
Asian-Pac-Islandee
Male
19.626
Other
Female
20.000
Aner-Indian-Eskimee
Male
25.000
Figure (()) visualises the distribution of regularisation samples inside subgroups defined by 10 label buckets. where the labels are sorted in descending order of label frequency. Here, we compare the predicted probabilities of the teacher and final distilled student models on training samples (as was done the analysis of Zhou et al. (2021)). Interestingly, we see that the tail buckets tend have very few regularisation samples; i.e. for rare labels, the teacher prodiction on the training label generally higher than that of the distilled student. model. We confirm this in Figure 9(b).
While the analysis of Zhou et al. (2021) regularisation samples was to mitigate their influence during training - we may also identify the breakdown of such samples on test data. Figure 10(a) shows that, compared to the training set, there are in absolute terms more such samples across nearly every label bucket; however, there is again no clear correlation between the label bucket and the fraction of such samples. In particular, the tail bucket again the one with the fewest regularisation samples. This is corroborated by the probability scores of the teacher and student in Figure 10(b).
primarily for training samples since the aim in identifying
24

---

Published in Transactions on Machine Learning Research (11/2022)
50
10
(a) Distribution of regularisation samples per label bucket.
(b) Teacher versus distilled student probail-
ities on training label.
Figure 9: Study of regularisation samples on training set, CIFAR-100 LT.
variable
Teachor
Studen:
0.5
0.2
10
Label
Label
(a) Distribution regularisation samples per label bucket.
(b) Teacher versus distilled student probail- itles on test lahel.
Figure 10: Study of regularisation samples on test set, CIFAR-100 LT.
Per-class accuracy statistics
Method
Mean
Worst 10
Top 10%
A10
One-hot
44.16
3.00
87.70
41.16
Distillation 1x
45.49
0.90
88.10
44.59
Distillation 2x
45.22
0.00
88.40
45.22
Distillation 3x
44.80
0.00
87.60
44.80
Table 10: Results of repeated distillation on CIFAR-100 LT. Using a distilled student teacher for a subsequent round of distillation is seen to further hurt worst-class accuracy.
Overall, this results suggest that the existing notion of regularisation samples may not, by themselves, be sufficient to predict the poor performance of distillation on certain subgroups defined by labels.
C.5 Impact of repeated distillation
In the body, we showed that performing distillation once can harm worst-class accuracy. However, what is the effect of repeating this process, and distilling using the resulting student as a new teacher? Does the worst-class accuracy get further harmed?
Table 10 shows that on CIFAR- LT, repeating distillation can indoed harm worst-class performance, even though average performance remains roughly similar. This further highlights the potential tension between average and worst-case performance under distillation.
25

---

Published in Transactions on Machine Learning Research (11/2022)
Table 11:
Ablations design choices the proposed methods: 1) remove distillation signal from the bottom 10% of classes, according confidence; 2) randomly shuffie per-class a values; 3) weight distillation basod on student and teacher confidence Zhou et al. (2021).
Per-class accuracy statistics
Datasct
Method
Mean
Worst-1
Worst-10
CIFAR-100
AdaAlpha
75.52 0.10
49.33 + 3.09
56.59 + 0.44
remove hardest 10%
75.40 0.04
48.33 1.89
55.79 + 1.19
shuffle temperatures
74.56 0.93
46.00 + 1.91
53.10 = 1.22
Zhou et al. (2021)
75.74 + 0.20
50.33 + 0.47
54.93 + 0.34
C.6 Additional ablations
We confirm that the success of AdaAlpha is not immediately replicated by simpler baselines: (i) remove hardest 10%, which removes the distillation loss component on bottom 10% labels according to the per class margins found using Equation 4. Tt helps analyze whether there is any additional gain beyond simply removing teacher's supervision where it is arguably wrong. (ii) shuffle temperatures, which randomly shuffles the per-class Oy values used in AdaAlpha. This determines whether the precise choice of which labels to up- or down-weight is important; (iit) the adaptive distillation scheme of Zhou et al. (2021), where distillation is weighted differently across examples depending on the teacher and student scores.
In Table 11, we find that the first two methods work worse than the proposed AdaAlpha method, indicating that the precise choice of which labels to up- or down-weight important, and that it does not suffice merely ignore the teacher entire subgroups.
D Theoretical analysis
Consider the kernel ridge regression problem:
N min E
(5w((x)) - vnn) + Al|w||2
(7)
where H = (fw: x x Txx) w E eJ. Let the kernel function k(x,, x') +(x) (0xx)xT<(>> ((x), and let E NxN be the kernel matrix with K,j k(,,,,)) with f; (x,x). Then f" = K(K + IN)- where y is the vector of targets with Yn Yu-
x,). Let I* denote the minimizer of equation 2, and f be the vector
Now suppose K admits an cigendocompasition VTD,, where V is an orthonormal and D a diagonal matrix. Consequently,
=- D'DAAII Vy VT
where A = D + A Iw] From Mobahi et al. (2020), we know that using the predictions of f* for distillation results a solution
pdist = V A
(9)
Recall that we consider the following data distribution for our analysis. Let o, E [L] be L orthonormal
vectors in R corresponding to the input representation x; from different subgroups L. Let each subgroup have d, number of examples indicating the subgroup frequency. In this setting the kernel matrix (Kij x x,) has block diagonal structure, with blocks of ones size d; x d:
Claim 3. Let or, [Z.] be L orthonormal vectors in RDD Let di,... .d be positive integers and N Dt di- Let the dotaset (x)) consist of d, samples of vs, Vi. Then the eigenvolues of the N x N kernel matrix K. fwhere Kij- x x;); (di)s
26

---

Published in Transactions on Machine Learning Research (11/2022)
Proof of Theorem 1. Observe that the regression error is
ff -ll V AVy -yll
= V AVy V VTly| + V yyl?
= =IV (A - (A-l) + +
Here, V is the basis of the subspace orthogonal to V such that V = corresponds to the error in subspace orthogonal to the kernel. Hence, this error is same for all the models.
Note that IV i y
Let M2 denote the subset of data points from subgroup i. The mean error component in the subspace of examples from subgroup i is
1
Σ 1yyyl
1
dg
-1)
1)²
2
yk
di
d x
KEM;
K
2
yt.
d; +
Here 8? = म. EKEM, 2 is the mean subgroup label. The above equations follow from noting that the cigenvectors of examples from subgroup (Vi) are non-zero 1 only for the indices corresponding to examples in subgroup and zero everywhere else.
Recall that the predictions after one round of self distillation are pdist V A Hence, we can similarly compute the mean error of subgroup i after distillation as follows:
-1) 2 y2
Σ - de KEM,
1
1
Σ
=
di KEM:
(d: + X)2
( 12 2Ad,
y
(di X)2
Hence,
A
2
X2 + 2Xd;
C: G
dist
82
8
di + X
(d. + A)2
(2 A)2 (1² 2Ad,)
(d: + A)4
X X²d? + 22
(x + +422d + 4x
87
(d: + 2)²
3Xdd. - 2x
y?
(d, + X)4
We can show that if X < dddd then
3X2d. 2x'd))
dist
A11
3X2d3 2X3d4)
(d, +
(j dist
27


Reviews:
[{"review_id": "AbLLYJrN8D", "reviewer": "Reviewer_zieG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n1. This paper provides a new perspective on KD about fairness and minority.   The new direction seems important. \n\n2. The authors find an interesting observation KD can hurt minorities and tries to explain the reasons for the observation with experiments and a theorem.\n\n3. This paper is well-written and easy to follow.\n\nWeakness\n\n1. The experiment part needs to be improved. KD has some important hyper-parameters like the temperature and the mixing weight. The authors should provide more results with various settings and show some trends. Moreover, NN architectures of the teacher and the student are very important factors for the performance. The authors should consider various scenarios.\n\n2. The two proposed adaptive distillation methods are interesting ideas. However, I think we could think of many different mixing weight functions and per-class margin functions that function almost the same way. It would be very nice if the authors could provide more discussions on that.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the effectiveness of Knowledge Distillation (KD), especially for the case where the label distribution is highly non-uniform. From experiments, the authors find that KD can hurt subgroup performance since, under a long-tail label distribution, the teacher might not have the correct information for subgroup data. To resolve the subgroup performance issue, the authors propose two adaptive methods. The first one introduces adaptive mixing weights that control the weight in front of the KD loss between the teacher and the student so that well-trained classes have high weights while incorrectly trained classes have low weights. The second approach uses a per-class margin loss which enforces minor classes having more margin. The experimental results show the two proposed approaches outperform the vanilla KD. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. More experiment results with various different hyper-parameters and teacher and student architectures are needed. \n\n2. More discussions on mixing weight and adaptive margin functions might be necessary.\n\n3. In Section 3.3, when we sub-sample the classes, the number of data points is also changed. I think the number of data points is a very important factor for the KD performance. So, the authors require to check the performance when the numbers of data points are the same, but the numbers of classes are different. ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ADW0IMHtgm", "reviewer": "Reviewer_UU7c", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1)\tThis work identifies a previously underexplored issue with knowledge distillation. Distillation is a widely applied technique for improving model accuracy while preserving the model size. This systematic study raises the awareness of fairness between subgroups in a distillation setting, and helps future research to improve the fairness when using the distillation technique in various applications.\n\n2)\tThis work has conducted a thorough evaluation of the subgroup bias issue in multiple settings and benchmarks. This study demonstrates that a) this bias issue in distillation can happen in various datasets and model architectures, and b) distillation can be biased due to the large number of classes, the class imbalance, and other factors related to fairness.\n\n3)\tIn addition to empirical results, this work also takes a theoretical perspective to understand the subgroup bias in distillation, based on the analysis of a ridge regression problem.\n\n4)\tThe writing is clear and easy to follow.\n\nWeaknesses:\n\nThe major weaknesses are related to the proposed two solutions, AdaAlpha and AdaMargin.\n\n1)\tLimited improvement: From Table 3, the relative improvement from standard distillation seems insignificant. For example, for the worst class in ImageNet, the standard distillation reaches 14.00±1.41 accuracy, and the proposed two solutions can only improve that to 15.50±2.18 and 15.00±1.00. For the long-tailed (LT) versions of CIFAR-100 and ImageNet, the accuracy of the worst classes is still close to zero. In addition, AdaMargin may not converge in some cases.\n\n2)\tBoth solutions require a holdout split to quantify the teacher model’s influence for different subgroups, which may be an undesired necessity: a) Sampling the holdout split introduces additional variance and complexity. The authors also consider that the divergence of AdaMargin may be related to the holdout split. b) In a long-tailed or few-shot learning setting where some classes only contain a handful of examples, splitting out a holdout set may be unaffordable and hurt final performance.\n\n3)\tThe solutions are only examined in the self-distillation setting. It is unclear whether they are helpful when the teacher and student architectures are different.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work systematically studies a previously under-explored issue with knowledge distillation: though distillation usually improves the overall accuracy of the student model, the accuracy on some rare subgroups may be deteriorated by distillation. Empirical results and theoretical analysis help to identify that a large number of classes and a skewed class distribution can contribute to this observed phenomenon. Two solutions, AdaAlpha and AdaMargin, are proposed to mitigate this unbalanced subgroup improvement issue in knowledge distillation, by limiting the teacher influence for the subgroups where the teacher is making confident but incorrect predictions.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1)\tAs pointed out in the Weaknesses, the major concerns are with the proposed two solutions. More evaluation on different distillation settings, and explanation about the relative improvements over the baselines would be appreciated.\n\n2)\tSection 3: The empirical study of the bias in distillation is based on a fixed value of alpha (a weight for balancing the ground-truth labels and teacher predictions), set to 1.0 for CIFAR-100 and 0.2 for ImageNet. In fact, this weight parameter leads to different behaviors regarding the teacher model’s influence. It would be great to show how the distillation bias between subgroups changes with alpha.\n\n3)\tFigure 1: Before reading Section 3, a reader may have the following questions about how the “worst subgroups” are selected: 1) Are they the subgroups with the worst performance improvement? If so, why is the performance improvement for the worst subgroup in Figure 1(b) - Adaptive Margin larger than average? 2) How are the “worst subgroups” related to the “rarest classes” described in the caption? The frequency of samples may not be always consistent with performance or performance improvement in an unbalanced dataset. A bit more explanation about the “worst subgroups” in the caption could be helpful.\n\n4)\tSection 3.2: EfficientNets (Tan & Le, 2021) are a family of architectures with varying performance and model sizes. It would be better to point out the specific architecture (B0-B7) used in the experiments.\n", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "izZuuRLwWM", "reviewer": "Reviewer_PJm1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Pros : \n- The problem is well defined and crucial. \n- The pilot experiment is complete and rigorous, giving a variety of ablations with different variables. \n- The theoretical analysis is correct and strictly derived. \n- The proposed methods are simple but effective. \n\n## Cons : \n- Literature survey is not sufficient, which lacks some necessary approaches and therefore cannot summarize the background in a complete way. \n- The baselines are not strong enough. Some traditional or recent methods for overcoming the class imbalance issue are not presented. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In the field of knowledge distillation, the issue of error amplification on subgroups with few samples is a problem that has ample space for exploration and has great practical value in both academia and industry. This issue is well defined for the first time in this paper. Moreover, it analyzes and explains this issue through theoretical and empirical lenses. This paper has also proposed two approaches to correct the traditional distillation method, which are both simple but effective. Nevertheless, the experiment is incomplete and therefore insufficient to prove the superiority of these two methods. To my knowledge, the most commonly used methods to solve the class imbalance problem are data augmentation, ensemble learning, etc. At least, some tricks may also help, for example, only distill on the subgroups that the teacher predicts more correctly. The comparison with stronger baselines can better prove the practical value of the proposed method.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Take a complete literature review and categorize the popular methods. e.g.: \n  - Sampling-based: Class-Balanced Distillation for Long-Tailed Visual Recognition \n  - Reinforcement-learning-based: Reinforced knowledge distillation: Multi-class imbalanced classifier based on policy gradient reinforcement learning \n  - Data-augmentation-based: Solving the Class Imbalance Problem Using a Counterfactual Method for Data Augmentation \n  - Ensemble-learning-based: A Review on Ensembles for the Class Imbalance Problem: Bagging-, Boosting-, and Hybrid-Based Approaches \n- Choose some well-known ones as additional baselines such as data augmentation, ensemble learning, or more tricks.\n- Give more theoretical analysis and prove that the existing methods do not work theoretically (and experimentally). ", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hs9hyJlFRj", "reviewer": "Reviewer_fUxL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe proposed method is well-written and the motivation is clear and easy to understand.\n2.\tThe problem is interesting and novel. Improving subgroup performance of KD seems a practical topic.\n3.\tThe proposed method is simple but seems effective in subgroup performance. The theoretical proof looks reasonable.\n4.\tThe preliminary experiments make sense to su\n5.\tpport the motivation.\n\nWeakness:\n1.\tThe experiments are not sufficient. This paper does not compare some related works for logit-based KD methods. And the experimental benchmark is only image classification task but does not transfer to other recognition tasks. \n2.\tThe improved performance using the proposed method on overall accuracy seems very marginal, which may limit the usefulness of the proposed method in practice.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper observes that distillation can harm performance on certain subgroups, e.g., classes with few associated samples, compared to the vanilla student trained using the one-hot labels. Some empirical studies are verified across many network architectures, datasets and class-imbalanced scenarios. This paper proposes to soften the teacher influence for subgroups where it is less reliable. Experiments on several image classification benchmarks show the effectiveness of the proposed method in subgroup performance.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1.Writing issue: in Line 3 of page 7, the definition of margin seems confused. Should the formulation of $\\max_{y^{‘} \\ne y}p^{‘}_{y}(x)$ be $\\max_{y^{‘} \\ne y}p_{y^{‘}}(x)$?\n2.Some results of related logit-based KD methods [1,2,3] should be shown in Table 3, such as your referred works in page 9 (the section of Relation to existing work). Moreover, I also want to know whether the previous logit-based KD method would solve the subgroup performance degradation problem or not.\n3.Some references of self-supervised KD methods [4,5] in third paragraph of page 3 are missing.\n4.Open issue: beyond the image classification, could the proposed method apply to pixel-wise classification problem, for example, Hinton’KD in semantic segmentation [6,7]?\n\nReference:\n[1] Rethinking soft labels for knowledge distillation: A bias–variance tradeoff perspective. ICLR2021.\n[2] Prime-aware adaptive distillation. ECCV2020.\n[3] Channel distillation: Channel-wise attention for knowledge distillation. CoRR, abs/2006.01683, 2020.\n[4] Hierarchical Self-supervised Augmented Knowledge Distillation. IJCAI2021.\n[5] Knowledge Distillation Using Hierarchical Self-Supervision Augmented Distribution. TNNLS-2022.\n[6] Structured knowledge distillation for semantic segmentation. CVPR2019.\n[7] Cross-Image Relational Knowledge Distillation for Semantic Segmentation. CVPR2021.\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper contributes two interesting methods for mitigating KD biases. All the reviewers acknowledge the novelty and clarity of the papers. One of the reviewers questioned the literature review and experiment completeness and the authors partially addressed them, maybe that's the reason that the reviewer is leaning towards rejection. AE read the paper, reviews, authors' responses, and the revision, and indeed feels that the authors may want to further strengthen the survey part and the motivation of the experimental design. Overall, the merits of the paper clearly outweigh the demerits. AE recommends accept with further minor changes.
Decision: Accept (poster)