TEACHER’S PET: UNDERSTANDING AND MITIGATING BIASES IN DISTILLATION
Year: 2021

Abstract:
Knowledge distillation is widely used as a means of improving the performance of a relatively simple “student” model using the predictions from a complex “teacher” model. Several works have shown that distillation significantly boosts the student’s overall performance; however, are these gains uniform across all data subgroups? In this paper, we show that distillation can harm performance on certain subgroups, e.g., classes with few associated samples, compared to the vanilla student trained using the one-hot labels. We trace this behaviour to errors made by the teacher distribution being transferred to and amplified by the student model. To mitigate this problem, we present techniques which soften the teacher influence for subgroups where it is less reliable. Experiments on several image classification benchmarks show that these modifications of distillation maintain boost in overall accuracy, while additionally ensuring improvement in subgroup performance.

text: Knowledge distillation is widely used as a means of improving the performance of a relatively simple “student” model using the predictions from a complex “teacher” model. Several works have shown that distillation significantly boosts the student’s overall performance; however, are these gains uniform across all data subgroups? In this paper, we show that distillation can harm performance on certain subgroups, e.g., classes with few associated samples, compared to the vanilla student trained using the one-hot labels. We trace this behaviour to errors made by the teacher distribution being transferred to and amplified by the student model. To mitigate this problem, we present techniques which soften the teacher influence for subgroups where it is less reliable. Experiments on several image classification benchmarks show that these modifications of distillation maintain boost in overall accuracy, while additionally ensuring improvement in subgroup performance.

heading: 1 INTRODUCTION
text: Knowledge distillation is a technique for improving the performance of a “student” model using the predictions from a “teacher” model. At its core, distillation involves replacing the one-hot training labels with the teacher’s predicted label distribution. Empirically, distillation has proven successful for model compression (Bucilǎ et al., 2006; Hinton et al., 2015), improving the performance of a fixed model architecture (Anil et al., 2018; Furlanello et al., 2018), and semi-supervised learning (Radosavovic et al., 2018). Theoretically, several works (Lopez-Paz et al., 2016; Mobahi et al., 2020; Tang et al., 2020; Menon et al., 2020; Zhang & Sabuncu, 2020; Ji & Zhu, 2020; Allen-Zhu & Li, 2020; Zhou et al., 2021; Dao et al., 2021) have studied how distillation affects learning. Put together, both strands of work further the understanding of when and why distillation helps.
In this paper, we are similarly motivated to better understand the mechanics of distillation, but pose a slightly different question: does distillation help all data subgroups uniformly? Or, do its overall gains come at the expense of degradation of performance on certain subgroups? To our knowledge, there has been no systematic study (empirical or otherwise) of this question. This consideration is topical given the study of fairness of machine learning algorithms on under-represented subgroups (Hardt et al., 2016; Buolamwini & Gebru, 2018; Chzhen et al., 2019; Sagawa et al., 2020a).
Our first finding is that even in standard settings — e.g., on image classification benchmarks such as CIFAR — distillation can disproportionately harm performance on subgroups defined by the individual classes (see Figure 1). To discern the source of this behaviour, we ablate the teacher and student architectures (§3.2), dataset complexity (§3.3), label frequencies (§3.4). These point to the potential harms of distillation when the teacher makes confident mispredictions on a subgroup. We emphasise that this phenomenon is not simply the teacher’s worst class accuracy translating to student; instead, distillation amplifies worst class imbalance much more than the teacher.
Having identified a potential limitation of distillation, we present two simple techniques to remedy it. These apply per-subgroup mixing weights between the teacher and one-hot labels, and per-subgroup margins respectively (§4). Intuitively, these limit the influence of teacher predictions on subgroups it models poorly. Experiments on image classification benchmarks show that these methods typically maintain a boost in overall accuracy, while ensuring a more equitable improvement across subgroups.
In sum, this work provides novel insights into distillation performance, with three main contributions:
(i) we identify a hitherto unexplored issue with distillation, namely, that its improvements in overall accuracy may come at the expense of harming accuracy on certain subgroups (§3.1). Such a finding is topical given the widespread practical use of generic learning paradigms such as distillation, and the increasing societal applications of learning systems more broadly.
(ii) we ablate potential sources for the above phenomenon (§3.2, §3.3, §3.4), and in the process identify certain characteristics of data (e.g., skewed label distributions) where it can manifest. This systematic empirical analysis aims at understanding a non-trivial phenomenon, rather than empirically comparing different learning methods.
(iii) we propose two simple modifications of distillation that mitigate the above problem, based on applying per-subgroup mixing weights and margins (§4); these perform well empirically (§5).
While the paper’s analysis is empirical rather than theoretical, we note that our focus is on systematic analysis aiming at understanding a non-trivial phenomenon, rather than merely empirical comparison: Section §3 is devoted to carefully understanding the extent of, and causes for, the non-uniform gains of distillation. This is in line with works which empirically analyse neural network phenomena, e.g. Zhang et al. (2017); Müller et al. (2019); Nakkiran et al. (2020); Neyshabur et al. (2020).

heading: 2 BACKGROUND AND RELATED WORK
text: Knowledge distillation. Consider a multi-class classification problem over instances X and labels Y = [L]
. = {1, . . . , L}. Given a training set S = {(xn, yn)}Nn=1 drawn from some distribution P, we
seek a classifier h : X → Y that minimises the misclassification error Eavg(h) . = P(h(x) 6= y). In practice, one may learn logits f : X→ RL to minimise R̂(f) = 1N ∑N n=1 `(yn, f(xn)), where ` is a loss function such as the softmax cross-entropy, which for softmax probabilities py(x) ∝ exp(fy(x)) is `(y, f(x)) .= − log py(x). One may then classify the sample via h(x) = arg maxy∈[L] fy(x).
Knowledge distillation (Bucilǎ et al., 2006; Hinton et al., 2015) employs the logits f t : X→ RL of a “teacher” model to train a “student” model. The latter learns logits f s : X→ RL to minimise:
R̂dist(f) = 1
N N∑ n=1 [ (1− α) · `(yn, f(xn)) + α · ∑ y′∈[L] pty′(xn) · `(y′, f(xn)) ] , (1)
where α ∈ [0, 1]. Here, one converts the teacher logits to probabilities pt : X→ ∆L for simplex ∆, e.g. via a softmax transformation pty′(x) ∝ exp(f ty′(x)). The second term smooths the student labels
based on the teacher’s confidence that they explain the sample. The first term includes the original label, so as to prevent incorrect teacher predictions from overwhelming the student. One further important trick is temperature scaling of the teacher logits, so that pty′(x) ∝ exp(T−1 · f ty′(x)). Setting T 0 makes pt more uniform, thus preventing overconfident predictions (Guo et al., 2017). Average versus subgroup performance. The above exposition treats the misclassification error Eavg(h) as the fundamental performance measure of interest. However, suppose the data contains subgroups G = {1, . . . , G}. Defining the per-subgroup errors errg(h) . = P(h(x) 6= y | g), we have
Eavg(h) = ∑ g∈G P(g) · errg(h), which may mask errors on samples with P(g) ∼ 0 (Sagawa et al., 2020a;b; Sohoni et al., 2020). To this end, one may instead measure the balanced error (Menon et al., 2013) Ebal(h) . = ∑ g∈G 1 |G| · errg(h) which treats the subgroup distribution as uniform, or the worst-subgroup error (Sagawa et al., 2020a;b; Sohoni et al., 2020) Emax(h) . = maxg∈G errg(h), which focusses on the worst-performing subgroup. An intermediary is the average of the k worstperforming subgroups (Williamson & Menon, 2019): for ith largest per-subgroup error err[i](h), Etop−k(h) . = 1k ∑k i=1 err [i](h).
The definition of subgroups is a domain-specific consideration. One special case is where each label defines a subgroup (i.e., G = Y), and P(y) is skewed. In such long-tail settings (Buda et al., 2017), classifiers with good average performance can perform poorly on “tail” labels where P(y) ∼ 0. Related work. There is limited prior study that dissects distillation’s overall gains per subgroup. Zhao et al. (2020) showed that in incremental learning settings, distillation can be biased towards recently observed classes. We show that even in offline settings, distillation can harm certain classes. Recently, Zhou et al. (2021) studied the standard aggregate performance Eavg of distillation, which was tied to a certain subset of “regularisation samples”. By contrast, our primary concern is to understand the subgroup performance of distillation.
Study of the fairness of machine learning algorithms on under-represented data subgroups has received recent attention (Dwork et al., 2012; Hardt et al., 2016; Buolamwini & Gebru, 2018; Chzhen et al., 2019). This has prompted dissection of the performance of techniquess such as dimensionality reduction (Samadi et al., 2018), increasing model capacity (Sagawa et al., 2020a), and selective classification (Jones et al., 2021). We follow the general spirit of such works, studying a more delicate setting involving two separate models (the student and teacher), each with their own inductive biases. We present more discussion of related directions in §6. In a related effort, Hooker et al. (2019) explore how model compression may harm subgroup performance compared to the original model.

heading: 3 ARE DISTILLATION’S GAINS UNIFORM?
text: We demonstrate that the gains of distillation are not uniform across subgroups: specifically, considering subgroups defined by classes, distillation can harm the student’s performance on the “hardest” few classes (§3.1). To understand the genesis of this problem, we perform ablations (cf. Table 1) that establish its existence in settings where there are insufficient samples to model certain classes, either due to the number of classes being large (§3.3), or the class distribution being skewed (§3.4). We then identify that the student may amplify the teacher’s errors (§3.6). Finally, we corroborate these results for a more general notion of subgroup in a fairness dataset (§3.5).

heading: 3.1 DISTILLATION HURTS SUBGROUP PERFORMANCE
text: To begin, we consider the effect of distillation on a standard image classification benchmark, namely, ImageNet. We employ a self-distillation (Furlanello et al., 2018) setup, with ResNet-34 teacher and student models, trained with standard hyperparameter choices (see Appendix B). Following Cho & Hariharan (2019), we use early stopping on the teacher model. We now ask: what is the impact of distillation on the student’s overall and per-class performance? The first question has an expected answer: distillation improves the student’s average accuracy by +0.4% (see the Baseline setting in Table 1). Judged by this conventional metric, distillation is thus a success.
A more nuanced picture emerges when we break down the source of the above improvement. We compute the per-class accuracies for the one-hot and distillation models, to understand how the overall gains of distillation are distributed. Figure 2 shows that these gains are non-uniform: distillation
in fact hurts the worst-k class performance for k ≤ 40. (See Appendix C for a detailed per-class breakdown.) Thus, distillation may harm the student on classes that it already finds difficult. At the same time we note that distillation does improve many classes, such as the classes with relatively high accuracies from the one-hot student.
Given that average accuracy improves, it is worth asking whether the above is a cause for concern: does it matter that performance on subgroups corresponding to the “hardest” classes suffers? While ultimately a domain-specific consideration, in general exacerbating subgroup errors may lead to issues from the fairness perspective. Indeed, we shall see that distillation can also harm in settings where the subgroups correspond to sensitive variables; we discuss this further in §3.5.
At this stage, it is apposite to ask whether the above is an isolated finding, or indicative of a deeper issue. We thus study each of the following in turn: (i) does the finding hold in settings beyond self-distillation? (ii) does the finding hold for other datasets, or is it simply due to the idiosyncrasies of ImageNet? (iii) what are some general characteristics of settings where the problem is manifest?

heading: 3.2 IS DISTILLATION BIASED BY THE MODEL SIZE OR ARCHITECTURE?
text: Having begun with a self-distillation setup, we now demonstrate that similar findings hold when the student and teacher architectures differ. Continuing with the ImageNet dataset, in the second row in Table 1 we report statistics for the overall average accuracy and average accuracy over the worst 10 classes when distilled ResNet-50 student from a stronger teacher: Efficient-Net (Tan & Le, 2021) teacher achieving 85.7% accuracy. Again, we see improved average accuracy and harmed Worst subgroup accuracy, composed of the 10 classes with the lowest accuracy according to teacher’s performance. In Table 2, we report statistics when varying teacher and student architectures. The detrimental effect of distillation on hard class performance holds across all scenarios: thus, our earlier results were not specific to self-distillation.
For self-distillation settings, smaller models appear to incur greater losses on the worst-class error. When distilling between different architectures (e.g., from ResNet-50 to ResNet-18), we observe that even average accuracy may not improve, as noted in Cho & Hariharan (2019). There is however no
clear trend between the difference in architectures and drop in worst class performance. Finally, we note that there is little change in the teacher’s and student’s worst-k classe; see Figure 6 (Appendix).

heading: 3.3 IS DISTILLATION BIASED BY A LARGE NUMBER OF CLASSES?
text: Having seen that ImageNet consistently demonstrates a performance degradation on certain classes, we now repeat the same analysis on a smaller image classification benchmark. We return to the selfdistillation setup, using ResNet-56 models on CIFAR-100. On this dataset, Table 1 shows a (perhaps more expected) result: distillation boosts both the average and worst-1 class performance. This indicates that, at a minimum, the behaviour of distillation’s performance gains are problem-specific; on CIFAR, distillation appears a complete win for both the average and subgroup accuracy.
One plausible hypothesis is that the tension between average and subgroup performance only manifests on problems with many labels, which might informally be considered “harder”. To confirm this further while fixing the dataset, we randomly select 10% of classes from the ImageNet dataset, and only keep examples corresponding to those classes across the train and validation sets. Consistent with the results for CIFAR-100, we again find that the worst-10 class accuracy is not harmed under distillation (see the right side of Table 1 where we report results on ResNet-34 self-distillation).
The above indicates that for problems with a few, balanced labels, there may not be a tension between average and worst-subgroup performance under distillation. However, we now show that even for problems with relatively few labels, one may harm subgroup performance if there is label imbalance.

heading: 3.4 IS DISTILLATION BIASED BY CLASS IMBALANCE?
text: We now consider a long-tail setting, where the training label distribution P(y) is highly non-uniform. Following the long-tail learning literature (Cui et al., 2019; Cao et al., 2019; Kang et al., 2020), we construct “long-tailed” (LT) versions of the above datasets, wherein the training set is down-sampled so as to achieve a particular label skew. For ImageNet, we use the long-tailed version from Liu et al. (2019). For other datasets, we down-sample labels to follow P(y = i) ∝ 1µi for constant µ and i ∈ [L] (Cui et al., 2019). The ratio of the most to least frequent class is set to 100. From the Long tail setting in Table 1, we note that on both CIFAR-100-LT and ImageNet-LT, accuracy over the hardest classes drops (we report results from self-distillation using ResNet-56 for CIFAR-100 and ResNet-50 ImageNet). The former is particularly interesting, given that the standard CIFAR-100 shows gains amongst the hardest classes. This provides evidence that for some “harder” problems — e.g., where there are insufficiently many samples from which to model a particular class — there may be a tension between the average and subgroup performance.

heading: 3.5 BEYOND CLASSES: OTHER CHOICES OF SUBGROUPS
text: Our analysis thus far has focused on subgroups defined by classes. This choice is natural for longtailed problems, where it is important to ensure good model performance on rare classes (Kang et al., 2020). In other problems, different choices of subgroups may be appropriate. For example, in problems arising in fairness, one may define subgroups based on certain sensitive attributes (e.g., sex, race). In such settings, does one similarly see varying gains from distillation across subgroups?
We confirm this can indeed hold on the UCI Adult dataset using random forest models (details in Appendix C.3). This data involves the task of predicting if an individual’s income is ≥ 50K or not, and possesses subgroups defined by the individual’s race and sex. Akin to the preceding results, we find that distillation can significantly improve overall accuracy, at the expense of degrading accuracy on certain rare subgroups, e.g., Black women; see Table 1, and Table 8 (Appendix). This further corroborates our basic observation on the non-uniform distribution of distillation’s gains.
A distinct notion of subgroup was recently considered in Zhou et al. (2021), who identified the impact of certain “regularisation samples” on distillation. These are a subset of training samples which were seen to degrade the overall performance of distillation. It is of interest whether such a subgroup relates to our previously studied subgroups of “hard” classes; e.g., is there an abundance of regularisation samples in such subgroups, which might explain the poor performance of distillation? In Appendix C.4, we study the relationship between regularisation samples, and the per-label subgroups
from our analysis; we find that, in general, these may be complementary notions. We further analyze the effect of the technique proposed in Zhou et al. (2021) on average and subgroup accuracies in §5.

heading: 3.6 WHY DOES DISTILLATION HURT CERTAIN SUBGROUPS?
text: The above has established that in a range of scenarios, distillation can hurt performance on subgroups defined by individual classes. However, a firm understanding of why this happens remains elusive. To study this, we consider ResNet-56 self-distillation on CIFAR-100-LT — which showed a stark gap between the average and subgroup (i.e., worst-1 class) performance — and dissect the logits of the teacher and distilled student. (See the Appendix for plots where the teacher and student architectures differ.) Across classes, we seek to understand: (i) how aligned are the student and teacher accuracies? (ii) how reliable are the models’ probability estimates? (iii) how do the models’ confidences behave?
For a test1 example (x, y) and predicted label distribution p(x) ∈ ∆L, we thus compute each models’ accuracy, log-loss `log(y, p(x)) = − log py(x), and margin (Koltchinskii & Panchenko, 2002) `marg(y, p(x)) = py(x)−maxy′ 6=y p′y(x). Note that the latter may be negative if the model predicts the incorrect label for the example. Figure 3 shows these metrics on 10 class buckets: these are created by sorting the 100 classes according to the teacher accuracy, and then creating 10 buckets of classes. Within each bucket, we compute the average of the metric specified above.
Remarkably, for 5 out of 10 class buckets, average margins are negative, suggesting that the teacher is often wrong yet confident in predicting these classes. On these buckets, the student accuracy generally worsens compared to the teacher. Further, log-loss increases across all buckets (including those where accuracy improves), indicating reduced confidence in the true class of the distilled student. This points at a potential source of the poorer performance on the worst-1 accuracy. Recall that the distilled student’s aim is to mimic the teacher’s logits on the training samples. This is a proxy to the student’s true goal, which is mimicking these logits on test samples, so as to attain similar generalisation performance as the teacher. When such generalisation happens, the student can thus be expected to roughly inherit the teacher’s per-class performance; in settings like the above, this unfortunately implies it will perform poorly on those classes with negative teacher margin.

heading: 4 IMPROVING SUBGROUP PERFORMANCE UNDER DISTILLATION
text: We now study simple means of correcting distillation to prevent the degradation of subgroup performance identified above. These leverage the insight that the behaviour is potentially a result of
1The choice of test, rather than train, example is crucial: an overparameterised teacher will likely correctly predict all training samples, thus rendering the above statistics of limited use. To leverage the insights from the above analysis in practice, we shall use a holdout set that can be carved out from the training set.
the teacher confidently mispredicting on some subgroups. In the following, for concreteness and simplicity, we focus on subgroups that are given by the individual classes.

heading: 4.1 DISTILLATION WITH ADAPTIVE MIXING WEIGHTS
text: In §3.6, we saw that distillation can hurt on classes where the teacher is inherently inaccurate. Such inaccuracy may in fact be amplified by the student, which is hardly desirable. An intuitive fix is to simply rely less on the teacher for classes where it performs poorly, or is otherwise not confident; instead, the student can simply fall back onto the one-hot training labels themselves. Formally, for per-class mixing weights (α1, . . . , αL) ∈ [0, 1]L, the student can minimise
R̄dist(f) = 1
N N∑ n=1 [ (1− αyn) · `(yn, f(xn)) + αyn · ∑ y′∈[L] pty′(x) · `(y′, f(xn)) ] . (2)
This objective introduces a mixing weight αy per-class, which allows us to weigh between teacher predictions and one-hot labels for each class independently. By contrast, in the standard distillation setup equation 1 we only have a single weight α that is common for all classes.
How do we choose the weights αy? In the standard distillation objective equation 1, one only needs to tune a single scalar α, which is amenable to, e.g., cross-validation. By contrast, equation 2 involves a single scalar for each label, which makes any attempt at grid search infeasible. Following the observations in §3.6, we propose the following intuitive setting of αy given teacher predictions pt:
αy = max ( 0,Ex|y [ γavg(y, p t(x)) ]) γavg(y, p t(x)) . = pty(x)− 1 L− 1 ∑ y′ 6=y pty′(x). (3)
In words, Equation 3 places greater faith in the teacher model for those classes which it predicts correctly with confidence, i.e., with large average margin γavg. When this margin is negative — so that the teacher is incorrect on average, which can occur on classes that are rare in the training set — we set αy = 0, and completely ignore the teacher predictions. The above requires estimating the expectation Ex|y [·], which requires access to a labelled sample. This may be done using a holdout set; we shall follow this in our subsequent experiments.

heading: 4.2 DISTILLATION WITH PER-CLASS MARGINS
text: Our second approach for improving distillation on harder classes is to leverage recent developments in long-tail learning, where the goal is to improve performance on rare classes. Specifically, Khan et al. (2018); Cao et al. (2019); Tang et al. (2020); Ren et al. (2020); Menon et al. (2020); Wang et al. (2021) proposed a variant of the softmax cross-entropy with margins ρyy′ between label pairs:
`(y, f(x)) = log 1 + ∑ y′ 6=y ρyy′ · efy′ (x)−fy(x)  . (4)
Intuitively, this penalises predicting label y′ instead of y when ρyy′ is large. For training label distribution π, Cao et al. (2019) proposed to set ρyy′ ∝ exp(π−1/4y ), so that rare labels receive a higher weight when misclassified. Khan et al. (2018); Ren et al. (2020); Menon et al. (2020); Wang et al. (2021) showed gains with ρyy′ ∝ πy′
πy , so that rare labels are not confused with common ones.
We adapt such techniques to our setting, with the intuition that we ought to increase the student penalty for misclassifying those “hard” classes that the teacher has difficulty modeling. We thus choose ρyy′ = αy′
αy , where αy is the adaptive per-class mixing weight from the previous section. This
discourages the model from confusing “hard” labels y with “easy” labels y′, when αy′ > αy. To avoid a division by 0 issue, we add a small offset to αy when it becomes 0.
We may understand the effect of Equation 4 by studying how it impacts the Bayes-optimal student model predictions, i.e., the optimal predictions in the infinite sample limit, and without a model capacity restriction. We have the following. Lemma 1. Let ` be the loss of Equation 4, with ρyy′ . = αy′
αy . Let f∗ be the Bayes-optimal student
scores that minimise Rdist(f) . = Ex(pt(x))>`(f(x)). Then, ∀x ∈ X, y ∈ [L], f∗y (x) = log
pty(x)
αy .
Lemma 1 illustrates that using per-class margins encourages the student to mimic the teacher predictions pt(x), but with an important modification: we up-weight the probabilities for classes that the teacher does poorly on (αy ∼ 0). Intuitively, this makes it easier for the student to improve performance on classes with small teacher margin.
Relation to existing work. Previous works varied distillation supervision across examples towards improving average accuracy. Proposals included weighting samples based on the ratio (Tang et al., 2019; Zhou et al., 2021), and difference (Zhang et al., 2020) between student and teacher score. Similarly, Zhou et al. (2020) proposed to only apply distillation on samples the teacher gets correct.

heading: 5 RESULTS FOR ADAPTIVE DISTILLATION METHODS
text: We now present results that further corroborate the potential non-uniform gains of distillation, and the ability to mitigate this with the techniques of the previous section. We emphasise here that our goal is expressly not to improve over the state-of-the-art in distillation techniques; rather, we wish to verify the key principles identified in the preceding study, which considers distillation from a novel angle (i.e., in terms of subgroup rather than average performance).
Setup. We report results on the datasets used in §3: CIFAR-100, ImageNet; and long-tailed (LT) versions of the same. For brevity, we report results under self-distillation. (For results with varying architectures, see the Appendix.) Thus, for each dataset, we train a one-hot teacher ResNet model, which is distilled to a student ResNet of the same depth. We use ResNet-56 models for CIFAR, and ResNet-50 models for all other datasets. We employ the same hyper-parameters as used in §3, except we use non-early stopped teachers for consistency across datasets; see the Appendix for details.
We compare: (i) one-hot training of the student (ii) standard distillation, i.e., minimising Equation 1 (iii) AdaAlpha, our proposed distillation objective with adaptive mixing between one-hot and teacher labels Equation 2, and α as per Equation 3 (iv) AdaMargin, our proposed distillation objective with adaptive margins (Equation 4), and ρyy′ = αy′
αy . We summarise each method by reporting: (i) the
standard mean accuracy over all classes; (ii) the accuracy over the worst-1 class; and (iii) the mean accuracy over worst-10 (and worst-100 for the LT datasets) classes.
For the Ada-* methods, per §4, creating the label-dependent αy requires estimating the generalisation performance of the teacher. To do this, we create a random holdout split of the training set. For non-LT datasets, we randomly split into 80% (new train) – 20% (dev). For LT datasets, for each class we hold out k examples into the dev set (k = 50 for Imagenet-LT, k = 20 for CIFAR-100-LT), or half of examples for a class if the total number of per class examples is at most 2k. We train an initial teacher on the new train slice of data, and estimate its per-class performance on the holdout dev slice. These are used to estimate αy as per, e.g., Equation 3.
Table 3 summarises the results for all methods. We make the following observations.
AdaAlpha improves mean accuracy over vanilla distillation. The proposed AdaAlpha method consistently and significantly improves standard mean accuracy over vanilla distillation. Thus, AdaAlpha does not sacrifice the gains offered by distillation on average class performance, which is desirable. Other techniques sometimes perform slightly worse than standard distillation on this metric; however, as we now see, this is compensated by gains on other important dimensions.
AdaAlpha improves worst-accuracy over distillation. The proposed method consistently improves the worst-class accuracy compared to standard distillation: Thus, the technique largely fulfil their design goal of improving performance on “hard” classes, while not overly sacrificing average-case performance. In most cases, these improve both the average and worst-class accuracy, indicating that softening the teacher influence can be broadly beneficial.
Comparison of AdaAlpha and AdaMargin. In the Appendix, we report per class statistics for CIFAR-100 LT. AdaMargin flattens both the margin and log-loss distributions, reducing confidence on the poorly classified, tail classes. AdaAlpha consistently increases log-loss across classes, and improves margins on few buckets, leading to a positive margin on one bucket where all other methods give negative margins. Intuitively, AdaMargin tries to more aggressively control the worst-class accuracy; when this succeeds, there is a large payoff, but there is also greater risk of overfitting.
Additional ablations. We confirm that the success of AdaAlpha is not immediately replicated by simpler baselines: (i) remove hardest 10%, which removes the distillation loss component on bottom 10% labels according to the per class margins found using Equation 3. It helps analyze whether there is any additional gain beyond simply removing teacher’s supervision where it is arguably wrong. (ii) shuffle temperatures, which randomly shuffles the per-class αy values used in AdaAlpha. This determines whether the precise choice of which labels to up- or down-weight is important; (iii) the adaptive distillation scheme of Zhou et al. (2021), where distillation is weighted differently across examples depending on the teacher and student scores.
In Table 4, we find that the first two methods work worse than the proposed AdaAlpha method, indicating that the precise choice of which labels to up- or down-weight is important, and that it does not suffice to merely ignore the teacher on entire subgroups. The adaptive distillation scheme Zhou et al. (2021) is also not as effective as AdaAlpha; see Appendix for more such results.

heading: 6 DISCUSSION AND OTHER APPROACHES
text: Our goal of ensuring equitable performance across classes can be seen as encouraging fairness across subgroups defined by the classes. This is subtly different to the classical fairness literature (Calders & Verwer, 2010; Dwork et al., 2012; Hardt et al., 2016), wherein the subgroups are defined by certain sensitive attributes. Broadly, fairness techniques attempt to learn models that predict the target label accurately, but the subgroup label poorly; these are inadmissible for our setting, wherein the two labels exactly coincide. Ensuring fairness across subgroups defined by the classes has been studied in Mohri et al. (2019); Williamson & Menon (2019); Sagawa et al. (2020a), who proposed algorithms to explicitly minimise the worst-class (as opposed to the average) loss. Adapting such algorithms to the distillation setting is of interest for future work. More broadly, the intent of the analysis in this paper is to better understand settings where distillation can implicitly hurt certain under-represented subgroups.

heading: ETHICS STATEMENT
text: This work studies settings wherein distillation can harm performance on subgroups. The aim of such a study is to help identify and mitigate potential adverse effects from distillation in practical applications where fairness considerations are important. Further, our proposed techniques were shown to improve the subgroup performance of distillation, and thus could be useful for preventing such adverse behaviour. As a limitation, we considered natural definitions of such subgroups, but an important practical consideration is settings where subgroup membership is unknown.

heading: REPRODUCIBILITY STATEMENT
text: For all experiments, we used the default hyperparameters from previous works (see Appendix B for details).

heading: A PROOFS
text: Proof of Lemma 1. We may write
`(y, f(x)) = log 1 + ∑ y′ 6=y αy′ αy · efy′ (x)−fy(x)  = log
1 + ∑ y′ 6=y elnαy′−lnαy · efy′ (x)−fy(x) 
= log 1 + ∑ y′ 6=y ef̄y′ (x)−f̄y(x)  = − log e
f̄y(x)∑ y′∈[L] e f̄y′ (x) ,
where f̄y(x) . = fy(x) + lnαy . The population loss is
Rdist(f) = Ex [ (pt(x))>`(f(x)) ] = Ex [ KL(pt(x) ‖ p̄s(x)) ] ,
where p̄sy(x) ∝ exp(f̄y(x)). Thus, at optimality we must have p̄s(x) = pt(x), or f̄y(x) = log pty(x). By definition of f̄ , we thus see that fy(x) = log pty(x)− logαy = log pty(x) αy .

heading: B DETAILS OF EXPERIMENTS

heading: B.1 ARCHITECTURE
text: We use ResNet with batch norm (He et al., 2016) for all our experiments with the following configurations. For CIFAR, we experiment with ResNet-56 and ResNet-32. For ImageNet, we use ResNet-50. We list the architecture configurations in terms of (nlayer, nfilter, stride) corresponding to each ResNet block in Table 5.

heading: B.2 TRAINING SET
text: For all datasets, we train using SGD and weight decay 10−4 for CIFAR, and 0.5× 10−4 for Imagenet datasets. We have the following dataset specific settings.
CIFAR-100. We train for 450 epochs with an initial learning rate of 1.0, with a linear warmup in the first 15 epochs, and an annealed learning rate schedule. We drop the learning rate by a factor of 10 at epochs number: 200, 300 and 400. We use a mini-batch size of 1024. We use SGD with Nesterov momentum of 0.9.
For our distillation experiments we train only with the cross-entropy objective against the teacher’s logits. For each method we find the best temperature from the list of values: {1, 2, 3, 4, 5}. ImageNet. We train for 90 epochs with an initial learning rate of 0.8, with a linear warmup in the first 5 epochs, and an annealed learning rate schedule. We drop the learning rate by a factor of 10 at epochs number: 30, 60 and 80. We use a mini-batch size of 1024.
For our distillation experiments we train with the distillation objective as defined in Equation 1 setting α = 0.2. For each method we fix the temperature to 0.9.
Long-tail (LT) datasets. We follow setup as in the non-long tail version, except for the learning rate schedule, which we change to follow the cosine schedule (Loshchilov & Hutter, 2017).

heading: C ADDITIONAL EXPERIMENTS
text: We present additional experiments to those in the body.

heading: C.1 FURTHER VARYING DATASETS AND MODEL ARCHITECTURES
text: On Imagenet, we summarise statistics for three models: the early-stopped teacher, distilled student, and the one-hot (non-distilled) student. As with CIFAR-100-LT, we sort classes by teacher accuracy, and bucket them into 10 groups. Owing to the larger number of labels, we further zoom into the “tail” bucket (comprising the 100 “hardest” classes), and split them into 10 sub-buckets. From Figure 4, the distilled student performs worse than its one-hot counterpart on the last bucket; this is in keeping with our results in Table 1.
Figure 5 shows logit statistics for additional settings to considered in the body. On ImageNet-LT, e.g., we see again that the margin of the teacher model systematically worsens and becomes negative on the hardest classes.
In Table 6 we report results from the inherently long-tailed iNaturalist 2018 dataset (Van Horn & Perona, 2017). Our observations made for other considered datasets hold: adaptive margin method improves over both one hot and plain distillation in terms of the worst class accuracy. We also observe, how the average accuracy improves.

heading: C.2 LOGIT PLOTS UNDER ADA-* METHODS
text: Figure 7 shows the logit statistics under the proposed AdaMargin and AdaAlpha methods on CIFAR100 LT. We see that AdaMargin can generally improve the student margin and accuracy on the hardest classes, while also reducing the log-loss. This confirms that the gains of the method come from improving behaviour of the scores on these hard classes.

heading: C.3 RESULTS ON ADULT DATASET
text: We report the results of an experiment on the UCI Adult dataset. This data comprises ∼ 48K examples, with the target being a binary label denoting whether or not an individual has income ≥ 50K. The data is mildly imbalanced, with 24% of samples being positive. Inspired by Dao et al. (2021), we consider a random forest based distillation setup: we use a teacher model that is a random forest classifier comprising 500 trees with a maximum depth of 20, and a student model that is a random forest regressor comprising 1 tree with a maximum depth of 20. The teacher model achieves a test (balanced) accuracy of 81.8%.
We perform distillation by feeding the student model the teacher’s prediction scores, mixed in with the binary training labels with a weight α = 0.9. Distillation improves the student’s overall (balanced) accuracy significantly, from 76.2% to 79.3%. However, this gain is not distributed uniformly: using per-label subgroups, we find that distillation helps the positive class by +7.4%, but hurts the negative class by −1.2%. While by itself suggestive of asymmetry in distillation performance, the data admits an arguably more natural subgroup creation, based on available sex and sex features. For example, we find that amongst low-income males, distillation hurts by −2.2%; further restricting to those who are Asian Pacific-Islander, the degradation is −5.9%. This confirms that in scenarios where fairness may be a consideration, a naïve application of distillation may be inadmissible.

heading: C.4 ANALYSIS OF REGULARISATION SAMPLES
text: Recently, Zhou et al. (2021) proposed the notion of regularisation samples to understand how distillation’s performance can be improved. In brief, such samples correspond to cases where the teacher’s prediction on the training label is less than the distilled student’s prediction on this label; these may be shown to correspond to cases where a certain notion of “variance reduction” dominates
a notion of “bias reduction”. Given our analysis above of the asymmetric effects of distillation on certain subgroups, it is natural to consider whether or not these relate to the presence of regularisation samples in these groups.
Figure 8(a) visualises the distribution of regularisation samples inside subgroups defined by 10 label buckets. where the labels are sorted in descending order of label frequency. Here, we compare the predicted probabilities of the teacher and final distilled student models on all training samples (as was done in the analysis of Zhou et al. (2021)). Interestingly, we see that the tail buckets tend to have very few regularisation samples; i.e., for rare labels, the teacher prediction on the training label is generally higher than that of the distilled student model. We confirm this in Figure 8(b).
While the analysis of Zhou et al. (2021) was primarily for training samples — since the aim in identifying regularisation samples was to mitigate their influence during training — we may also identify the breakdown of such samples on test data. Figure 9(a) shows that, compared to the training set, there are in absolute terms more such samples across nearly every label bucket; however, there is again no clear correlation between the label bucket and the fraction of such samples. In particular, the tail bucket is again the one with the fewest regularisation samples. This is corroborated by the probability scores of the teacher and student in Figure 9(b).
Overall, this results suggest that the existing notion of regularisation samples may not, by themselves, be sufficient to predict the poor performance of distillation on certain subgroups defined by labels.
C.5 IMPACT OF REPEATED DISTILLATION
In the body, we showed that performing distillation once can harm worst-class accuracy. However, what is the effect of repeating this process, and distilling using the resulting student as a new teacher? Does the worst-class accuracy get further harmed?
Table 9 shows that on CIFAR-100 LT, repeating distillation can indeed harm worst-class performance, even though average performance remains roughly similar. This further highlights the potential tension between average and worst-case performance under distillation.

Review summaries:
- This paper is established by a discovered phenomenon that in distillation, not all the classes’ performance is improved although the student is improved significantly.   - Thus, the authors explore a method that focuses on promoting the performance of the poorly-performed labels, which also lead to the overall promotion while distillation.  - The authors propose two methods to achieve it, one is the distillation with adaptive mixing weights and the other is the distillation with per-class margins. 

This paper identified an underexplored problem with distillation, fairness of distillation on different subgroups. This paper further proposed two methods to tackle this challenge. Experiments are conducted on CIFAR-100, ImageNet, and their long-tailed variants.

The paper proposes to have a closer look at (logit matching) knowledge distillation, and explores whether the gains are uniform across all portions of the test data or if instead there are gains on some portions of the data but damaged performance on others. This is justified by the potential of KD to amplify biases. The authors quickly show that indeed the gains are not uniform, some classes being affected negatively by KD so that the bad performance of the teacher on some classes is amplified on the student. Then the authors conduct a number of experiments to study the impact of different aspects of training (e.g. architectures involved and aspects of data like class imbalance) over this amplification of class-specific poor performance. They are then able to identify what classes are likely to be negatively affected and follow that with the proposal of two modifications of the basic KD formulation with the aim to improve KD effect on the worse-performing classes. Experiments focus on CIFAR-100 and ImageNet, including long-tail variants.

In this work the authors examine the impact of distillation on certain sub-groups of the data. More specifically, they demonstrate that even though distillation can increase the overall accuracy of a model, it can harm the accuracy on certain subclasses. This includes both self-distillation, as well as regular distillation setups. To overcome this, the authors propose using class-specific weights, that have been tuned according to a holdout set, to tune the importance of each class during distillation, as well as using softmax cross-entropy with margins tuned according to the difficulty of each class. Furthermore, it is also demonstrated that these phenomena can also impact the fairness of the model.