Robust Active Distillation
Authors: Cenk Baykal
Khoa Trinh
Fotis Iliopoulos
Gaurav Menghani
Erik Vee
Venue: ICLR 2023 poster (2023)
Keywords: knowledge distillation
active learning
semi-supervised learning
model compression

Abstract: Distilling knowledge from a large teacher model to a lightweight one is a widely successful approach for generating compact, powerful models in the semi-supervised learning setting where a limited amount of labeled data is available. In large-scale applications, however, the teacher tends to provide a large number of incorrect soft-labels that impairs student performance. The sheer size of the teacher additionally constrains the number of soft-labels that can be queried due to prohibitive computational and/or financial costs. The difficulty in achieving simultaneous \emph{efficiency} (i.e., minimizing soft-label queries) and \emph{robustness} (i.e., avoiding student inaccuracies due to incorrect labels) hurts the widespread application of knowledge distillation to many modern tasks. In this paper, we present a parameter-free approach with provable guarantees to query the soft-labels of points that are simultaneously informative and correctly labeled by the teacher. At the core of our work lies a game-theoretic formulation that explicitly considers the inherent trade-off between the informativeness and correctness of input instances. We establish bounds on the expected performance of our approach that hold even in worst-case distillation instances. We present empirical evaluations on popular benchmarks that demonstrate the improved distillation performance enabled by our work relative to that of state-of-the-art active learning and active distillation methods.

Published as a conference paper at ICLR 2023
ROBUST ACTIVE DISTILLATION
Cenk Baykal, Khoa Trinh, Fotis Iliopoulo,, Gauray Menghani, Erik Vee Google Research
(baykalc, choatrinh, fotisi, amenghani,eriken e @google.com
ABSTRACT
Distilling knowledge from large teacher model to lightweight one is a widely suc- cessful approach for generating compact. powerful models in the semi-supervised learning setting where a limited amount of labeled data is available. In large-scale applications, however, the teacher tends to provide a large number of incorrect soft-labels that impairs student performance. The sheer size of the teacher addi- tionally constrains the number of soft-labels that can be queried due to prohibitive computational and/or financial costs. The difficulty in achieving simultaneous efficiency (i.e., minimizing soft-label queries) and rubustness (i.e., avoiding student inaccuracies due to incorrect labels) hurts the widespread application of knowl- edge distillation to many modern tasks. In this paper, we present a parameter free approach with provable guarantees query the soft-labels of points that simul- taneously informative and correctly labeled by the teacher. At the core of our work lies game- theoreti formulation that explicitly considers the inherent trade-off between the informativeness and correctness of input instances. We establish bounds on the expected performance of our approach that hold even in worst-case distillation instances. We present empirical evaluations popular benchmarks that demonstrate the improved distillation performance enabled by our work relative that of state-of-the-an active learning and active distillation methods.
INTRODUCTION
Deep neural network models have been unprecedentedly successful in many high-impact application areas such as Natural Language Processing (Ramesh et al., 2021; Brown et al., 2020) and Computer Vision (Ramesh et al., 2021; Niemeyer & Geiger, 2021). However, this has come at the cost of using increasingly large labeled data sets and high-capacity network models that tend to contain billions parameters (Devlin et al., 2018). These models are often prohibitively costly to use for inference and require millions of dollars in compute to train (Patterson et al., 2021). Their sheer size also precludes their use in time-critical applications where fast decisions have to be made, e.g., autonomous driving, and deployment to small embedded systems (Baykal et al., 2022). To alleviate these issues, a vast amount of recent work in machine learning has focused on methods to generate compact, powerful network models without the need for massive labeled data sets.
platforms, e.g., mobile phones and
Knowledge Distillation (KD) (Buciluă et al., 2006; Hinton et al., 2015; Gou et al., 2021; Beyer et al., 2021) is a general purpose approach that has shown promise in generating lightweight powerful models even when limited amount of labeled data is available (Chen et al., 2020). The key idea is to use large teacher model trained on labeled examples to train a compact student model so that its predictions imitate those of the teacher. The premise that even small student is capable enough to represent complicated solutions, even though it may lack the inductive biases to appropriately learn representations from limited data on its own (Stanton et al., 2021; Menon et al., 2020). In practice, KD often leads to significantly more predictive models than otherwise possible with training in isolation (Chen et al., 2020; Xie et al., 2020; Gou et al., 2021; Cho & Hariharan, 2019).
Knowledge Distillation has recently been used to obtain state-of-the-ar results in the semi-supervised setting where a small number of labeled and a large number of unlabeled examples are available (Chen al., 2020; Pham et al., 2021; Xie et al., 2020). supervupee KD entails training teacher model on the labeled set and using its soft labels on the unlabeled data to train the student. The teacher is often pre-trained model and may also be generic large model such GPT-3 (Brown al., 2020)

---

Published as a conference paper at ICLR 2023
or PaLM (Chowdhery al., 2022). The premise is that large teacher model can more aptly extract knowledge and learn from a labeled data set, which can subsequently be distilled into a small student.
Despite its widespread success, KD generally suffers from various degrees of confirmation bias and inefficiency in modern applications to cemi-supervised learning. Confirmation bias (Pham et al., 2021; Liu & Tan, 2021; Arazo et al., 2020; Beyer et al., 2021) is the phenomenon where the student exhibits poor performance due to training on noisy or inaccurate teacher soft-labels. Here, inaceuracy refers the inconsistency between the teacher's predictions for the unlabeled inputs and their groundtruth labels. Feeding the student inaccurate soft-labels leads to increased confidence in incorrect predictions, which consequently produces a model that tends to resist new changes and perform poorly overall (Liu & Tan, 2021; Arazo et al., 2020). At the same time, sarge-scale applications often require the teacher's predictions for billions of unlabeled points. For instance, consider distilling knowledge from GPT-3 to train a powerful student model. As of this writing, OpenAl charges 6e per 1 token predictions (OpenAI, 2022). Assuming just 1M examples to label and average of 100 tokens per example leads to a total cost of $6M. Hence, it is highly desirable to acquire the most helpful - i.c., informative and correct - soft-labels subject to a labeling budget (GPT-3 API calls) to obtain the most powerful student model for the target application.
Thus, it has become increasingly important to develop KD methods that are both query query-efficient and robust to labeling inaccuracies. Prior work in this realm is limited to tackling either distillation efficiency (Liang et al., 2022; Xu et al., 2020), by combining mix-up (Zhang et al., 2017) and uncertainty-based sampling (Roth& & Small, 2006), robustness (Pham al., 2021; Liu & Tan, 2021; Arazo et al., 2020; Zheng et al., 2021; Zhang et al., 2020), through clever training and weighting strategies, but not both of these objectives implement method that finds a sweet spot and improves over standard techniques. Relatedly, there has been prior work learning under label noise (see Song et al. (2022) for survey), however, these works generally assume that the noisy labels are available (i.e., no active learning component) or impose assumptions on the type of label noise (Younesian et al., 2021). ). In contrast, we assume that the label noise can be fully adversarial and that we do not have full access to even the noisy labels.
the same time. In this paper, we present a simple-to-
To the best of our knowledge, this work is the first to consider the problem of importance sampling for simultaneous efficiency and robustness in knowledge distillation. To bridge this research gap, we present an efficient algorithm with provable guarantees to identify unlabeled points with soft-labels that tend to be simultancously informative and accurate. Our approach is parameter-fred imposes no assumptions on the problem setting, and can be widely applied to any network architecture and data set. At its core lies the formulation of an optimization problem that simultaneously captures the objectives of efficiency and robustness in an appropriate way. In particular, this paper contributes:
1. A mathematical problem formulation that captures the joint objective of training on infor- mative soft-labels that are accurately labeled by the teacher in a query-efficient way
2. A near linear time, parameter-free algorithm to optimally solve it
3. Empirical results on benchmark data sets and architectures with varying configurations that demonstrate the improved effectiveness of our approach relative to the state-of-the-art
4. Extensive empirical evaluations that support the widespread applicability and robustness of our approach to varying scenarios and practitioner-imposed constraints.
2 PROBLEM STATEMENT
We consider the semi-supervised classification setting where we are given a small labeled set X typically tens or hundreds of thousands of examples together with large unlabeled set Ny. typically on the order of millions or billions. The goal is to leverage both the labeled and unlabeled sets to efficiently and reliably train a compact, powerful model Astudent- To do so, we use knowledge distillation (Xie et al., 2020; Liang et al., 2020) where the labeled points are used to train a larger, (often pre-trained) teacher model that can then be used to educate a small model (the student). We emphasize that the teacher may be a pre-trained model, however, it is not trained on the unlabeled set Xy. The distillation process entails using the soft-labels of the teacher for the unlabeled points. The student is then trained on these soft-labeled points along with the original labeled data set. The key insight is that the large, pre-trained teacher model can more aptly learn representations from the limited data, which can then be imitated by the student.
2

---

Published as a conference paper at ICLR 2023
Somewhat more formally, we given two input sets Ny independently and randomly drawn from the imput space X' S R We assume that we have access to the hard labels 3'2 E {0, 1) for the instances in Xy, but not those in Xe and that |xx,] < |x Consistent with modern ML applications, we assume that a validation data set of labeled points is available. We will use a slight abuse in notation and refer to the set of labeled data points as (1.)
denote the set of (x, y) labeled pairs. We assume large-scale applications of KD where the teacher is exceedingly large to the extent that querying the teacher soft-label fareateer (x) [0, 1]th for an unlabeled point x E Xy is costly and soft-labeling all of Xy is infeasible. In the following, we introduce and motivate robust active distillation to conduct this process efficiently and reliably.
2.1 ACTIVE DISTILLATION
The objective of active distillation is to query the minimum number of teacher soft-labels for points in Xe in order to train a efficient way. This process is shown in Alg. 1. Here, we conduct T active distillation iterations after training the student and the teacher models on the training set P, which initially only includes the set of hard-labeled points. On Line 8 and throughout, fo(x)) [0, 1] denotes the softmax output of a neural network model 0 with respect to input æ. At each iteration (Lines 5-10, Alg. 1), we use a given querying algorithm, SELECT, to identify the most helpful 6 unlabeled points to soft-label by the teacher based on the most up-to-date student model 0 (Line 6). The selected points are then soft- by the teacher and added the (expanding) training set P. Subsequently, the student is trained using the Kullback-Leibler (KL) Divergence (Hinton et al., 2015) as the loss function on the training set which includes both the hard-labeled points (K,,, Vr. and the accumulated soft-labeled ones. We follow the standard convention in active learning (Ren et al., 2021) and efficient distillation (Liang et al., 2020; Xu et al., 2020) and train the student model from scratch on Line
student model @ttudent in computationally and financially-
Algorithm 1 ACTIVEDISTILLATION
Input: a set of labeled points (X2, Jzz). set of unlabeled points Ny. the number of points to soft-label per iteration E 4,, and selection algorithm SELECT(X", 6, b) that selects a sample of size b from X' 1: P + (LL))))); (Training set thus far; initially only hard-labeled points)
2: freacher f- TRAIN(P, .readeer) ); (Train teacher on labeled data starting from random initialization)
3: do TRAIN(P, orandom);; (Train student on labeled data starting from random initialization)
4: + 8; [Set of inputs that have heen soft-labeled)
5: for E (1.....77 T} do
6: S: + SELECT((Xv \ \S (,,0:-1,b b) (Select b points to be soft-labeled by @teacher }
7:
S < SUS. [Add new points so we do not sample them again)
8:
P PU [(I, Peraccher (x)) x E S.) [Soft-label points and add them the training set]
9: O. + TRAIN(P, Orandam) [Train network with the additional soft-labeled points from scratch)
10: end for
11:
return 0T
The active distillation problem is deeply related to the problem of active learning, where the objective is to query the labels of only the most informative points in order to minimize labeling costs. To this end, prior approaches in efficient KD (Xu et al., 2020; Liang methods inspired by margin-based sampling (Balcan et al., 2007; Roth & Small, 2006), a popular and widely used active learning algorithm (Ren et al., 2021). Margin-basee based sampling one example of uncertainty-baat based sampling, other examples are clustering- based selection (Sener & Savarese, 2017; Ash et al., 2019), model uncertainty (Gal et al., 2017), and adversarial proximity (Ducoffe & Precioso, 2018) (see (Ren et al., 2021) for survey). In the following, we consider margin-based sampling due to its simplicity and prior application to efficient distillation by related work (Liang et al., 2020; Xu et al., 2020). Margin-based sampling for KD is an intuitive and simple-to-implemen idea where the teacher predictions for inputs that the student is most uncertain about are queried. For an input x and prediction i* argmax;e feandem (z)s, the uncertainty is measured in terms of the margin between the top-2 highest probability entries, i.e., ma) =oosuto (x);-- max S0 (x)i-
al., 2020) have proposed
2.2 RESEARCH GAP
Despite the widespread success of margin- margin-based sampling in active learning, we claim that it is generally ill-suited for knowledge distillation due to its tendency to amplify confirmation bias, leading
3

---

Published as a conference paper at ICLR 2023
Probability teacher is correct
66
0.8
|n/ |w/ MMMMAA/MM/ Ml Mrm Ay
Test Accuracy (%)
64
62
Robust Distilation (Oracle)
0.6
Standard Margin
60
58
0.4
56
0.0
0.2
0.4
0.6
0.8
1.0
5000
10000
15000
20000
Student margin score
Number of soft-labeled points
Figure 1: Left: teacher's accuracy (overall 60.7% relative the student's (overall 49.4%) margin score; points with lower margin tend to he incorrectly classified by the teacher. Plot was generated by averaging teacher's accuracy over 100 masest points. Right: Performance of robust distillation (red), which picks the lowest-margin points among those correctly labeled by the teacher, compared to that of margin (bluc).
to poor student performance. To observe this, note that the objective of margin-base sampling and more generally, other uncertainty-basee query methods which the student is most uncertain about ("hard" instances). However, hard instances for the student are often hard to predict correctly by the teacher. Hence, the soft-labels for these points are more likely to be incorrect with respect to the groundtruth labels, leading to misleading student training.
query the soft-labels of inputs for
Fig. 1 shows an instance of this phenomenon for CIFARIO with ResNet stucent-teacher architectures of varying depth. As the figure depicts, the teacher tends to predict incorrect labels for points with low student margin (hard instances), and conversely, tends to be highly accurate on points with high margin (easy instances). This suggests that there is inherent trade-off between efficiency (minimizing queries) and robustness (mitigating confirmation bias) that needs to be considered. That is, would like to pick informative points for the student for efficiency, but these informative points tend to be incorrectly classified by the student which leads to misguided training and poor performance. Is it possible to simultaneously achieve both in principled way? We label this problem Robust Active Distillation and propose method to solve it in the following section.
3 ROBUST ACTIVE DISTILLATION (RAD)
3.1 BACKGROUND
The margin algorithm (Liang al., 2020; Roth & Small, 2006) selects the b points with the lowest margin scores margin(z), where b is our soft-label budget. Let margin; be shorthand for the margin of each unlabeled input margin(1;) and observe that its gain informativeness algorithm corresponds to
be quantified
as = 1 margin,. Given a budget 6, note that the margin-samplln the optimal solution of the following optimization problem where the objective is generate probability distribution that maximizes the expected sum of gains,
maxped, E S
[Eies 9i where A, (p E [0,
E
Eielal
Ps = b}.
As discussed previously, this formulation solely focuses on the informativeness of the points and does not consider the increased likelihood of mislabeling by the teacher.
Robust Distillation To extend (1) so that it is robust to possible teacher mislabeling, consider the masks e 11 (teahhe labels point correctly) for each i [x]] where 1 (xx) = x is true and otherwise. Equipped with this additional variable, one way to explicitly mitigate confirmation bias and simultancously pick informative samples is to reward points that are correctly labeled by the teacher by assigning gains as before, but penalize those that incorrectly labeled via losses. This can be done by using the modified gains in the context of (1)
if teacher labels point correctly otherwise
gi = g - (1 (1-4))
Vi e [n].
4

---

Published as a conference paper at ICLR 2023
In words, this means that if the point i is correctly labeled by the teacher assign the standard margin-basee based gain 9: = (1 (1- margin,) as before; otherwise, we penalize the selection by assigning -l; for some loss & 2 0. This leads to the following general problem of robust distillation
maXpeds Eup [g:c; - - [))c)
The optimal solution to problem (2) corresponds to picking the most informative (highest gain) points among those that are predicted correctly by the teacher, i.e., those points i E [n] with highest Ii subject to C; 1. This approach is shown as Robust Distillation (Oracle) Fig. (right). Fig. exemplifies the effect of inaccurate examples student training (see also (Pham et al., 2021)). If we had knowledge of (c:) (c=)se[m) then we could optimally solve (2) to obtain significant improvements over the standard margin algorithm. Unfortunately, perfect knowledge of whether the teacher labels each point correctly or not, i... (Ci)iE(n) is not possible the cemi-supervised setting.
3.2 OUR APPROACH
We consider general and robust approach that simultaneously leverages instance-specific knowledge without having to know the masks (cs)sE[n individually. Suppose that we only know that the teacher mislabels m points out of the n unlabeled inputs X'B instead. Can we generate a sampling distribution so that no matter which of the m points are labeled incorrectly by the teacher, our expected gain is
high? Assuming b 1 for simplicity, we arrive at the extension of the formulation (2)
maXpEA, min E sup Ig (1 - c.),,. where 4 {g [0, > ai == (3)
Problem (3) has the following game theoretic interpretation. We go first and pick sampling distribution p over the points. In response, an adversary decides which points are misclassified (i.e, = 0) by the teacher subject to the constraint that it can set c; 0 for at most m S of them since e E An-m- Given the linear structure of the problem, turns out that we can invoke von Neumann's Minimax Theorem (Neumann, 1928) which states that the equilibrium point is the same regardless of whether we go first and pick the probability distribution p E A or the adversary goes first and picks (ci)EE(n- By exploiting this connection, obtain closed form solution formalized below.
Theorem 1. Suppose 91 92 2
gn. > O, and define GR Eisk Eisk9i//e 9i/(:: + f.) and H₁ Eisk 1/(9 (s). For G. m, optimal solution p" E Ai to (3) is given by
G - m
P H (gs + 2.)
ifi K- and p 0 otherwise, where k" = argmax
H
The distribution can be computed in linear time (assuming sorted g) and achieves an objective value of OPT(k")
Gy- m
HA*
We sketch the proof here. The full proof can be found in the Appendix (Sec. C).
Proof sketch. Let
(B)
Σ Pilg,c (1 c))...
<E(M)
Substituting p" from Thm. and considering minimizing value for c An-m it is possible to show that p³ E A and
OPT(k")=
min cEA.-m
OBJ(p", e) S max min (B)(p.c). PEAi cEAn-m
On the other hand, let of min (1, O With little more work, using the fact that ge- 2 OPT(k*) 2 9k*41, we can show similarly that c* Ap-m and
get6,
OPT(k") = max OBJ(p, (") 2 min max (p. cEA,-m PEA;
With these inequalities in hand, we apply the Minimax Theorem (Neumann, 1928), which yields OPT(k") < max min yEA c
OBJ(P, c) min max OBJ(p, c) S OPT(k") cE PEA:
Hence, p" does indeed obtain the optimal value, OPT(K*)
5

---

Published as a conference paper at ICLR 2023
RAD Loss Equipped with Theorem all that remains is to specify the losses in (3). Prior work on confirmation bias has shown that even a small number misguided soft-labels can derail the student's s performance and significantly impact its predictive capability (Liu & Tan, 2021). Additionally, the harm of an incorrectly labeled point may be even more pronounced when the student is uncertain about that point. To model this, we consider instantiating our general problem formulation (3) with losses that are relative to the gain f; = -wyg -gg; for each i E [n]] where w E [0, 1] is a weight parameter that controls the magnitude of the penalization. This formulation purposefully leads to higher penalties for misclassified points that the student is already unsure about (high gain) to mitigate confirmation bias, and leads to the following optimization problem which is the focus of this paper
maXpe41 min E sup [g.0 - - (1-4) w.j
(4)
Invoking Thm. 1 with the relative gain losses as described above immediately leads to the following, which, along with the choice of w below, describes the algorithm RAD that we propose in this paper.
Corollary 1. An optimal solution p" E Ai to (4) has k non-zero entries In- S [rx] corresponding to the k- indices of the largest entries of g. with
1
k: - + wim
pt =
Vi Ix- where =-= argmaxke(M)
EjEr, 9j" The distribution p" can be computed in @(n n) time, with OPT(k*) = (k' (k-+(1+))m)/ EESIz 87
9: EjEI, 93 II
Choice of w Although RAD can be applied with any user-specified choice of w. we use the theoretically-mont motivvaed weight of w = (1 - m/n) as the relative penalization constant in our experi- ments. This choice of w guarantees that the optimal value (expected gain) of (4) (see Corollary (1)) is non-negative - if the expected gain were negative, we would be better off not sampling at all. This default value for makes RAD parameter-free. Extensive empirical evaluations with varying values of w are presented in Sec. D.5 of the Appendix.
0.0006
teacher acc: 50%
teacher acc: 60%
0.0006
teacher acc: 99%
teacher soc: 60%
teacher 89%
teacher 59%
teacher 89%
teacher 59%
0.0005
teacher acc: 79%
teacher 50%
0.0005
teacher acc: 79%
teacher acd: 50%
0.0004
0.3004
Probsallity
Probebility
0.0003
0.0003
0.0002
0.0002
0.0001
0.0001
0.0000
0.0000
0.0
0.2
0.4
0.5
0.8
1.0
2000
4000
6000
8000
10000
Gain
Point index (sorted by descending gain)
Figure 2: The optimal solution to (4) given by Corollary for various values for teacher accuracy (varying m). Even when most of the samples are labeled correctly by the teacher, our method is purposefully cautious and does allocate all of the probability (of sampling) mass the highest gain points.
We observe several favorable properties of RAD'S sampling distribution in Fig. 2, which depicts the computed distribution a synthetic scenario with gains drawn uniformly random from [0, 1]. For one, the sampling distribution tends to allocate less probability mass to the highest gain items. As prior work has shown, this is desirable because the hardest (highest gain) examples tend to be outliers or points with noisy labels (Mindermann et al., 2022; Ren et al., 2018). In fact, robust learning approaches typically downweight hard examples for this reason (Kumar al., 2010), analogous to RAD's 's sampling behavior. At the same time. Paul et al. (2021) show that the easiest (lowest gain) examples tend to be truly uninformative and the best strategy is to ignore a certain fraction of the highest and lowest gain points. This strategy parallels RAD'S computed distribution, where a number of low gain points are ignored and the probability peaks around a region inbetween (see Fig. 2). A prominent benefit of RAD is that this region is computed in a fully automated way as function of the teacher's accuracy (i.e., amount of label noise). If the teacher is highly accurate, the distribution accordingly concentrates on the highest gain points (blue, Fig. 2); otherwise, it spreads out the probabilities over larger portion of the points and purposefully assigns lower sampling probability the highest gain points which are likely to be noisy (e.g., brown, Fig. 2).
6

---

Published as a conference paper at ICLR 2023
3.3 IMPLEMENTATION DETAILS
We conclude this section by outlining the practical details of RAD. We follow the setting of this section and set g = margin, to define the gains of each point (see Sec. D.6 of the Appendix for evaluations with differing gain definition). In practice, we use the distribution g = min 1) when sampling a set of 6 points, where p is from Corollary 1. This is optimal as long as the probabilities from Corollary 1 (or more generally, Theorem 1) are not heavily concentrated on a few points (i.e., maX Pi I 1/6). As exemplified in Fig. 2 and experimentally verified in Sec. 4 in all of the found this to virtually always be the case. Alternatively, the acquisition size b
evaluated scenarios, can be adjusted after the single-sample probability distribution is computed so that b S minls]m] 1/p, Since the number of mistakes the teacher makes on Xn, m, is not known us in the semi-supervised setting, we approximate this quantity by first taking the sample mean inaccuracy of a small uniform random sample of buniform (see Appendix, Sec. D.1) points as way to approximate m and bootstrap our approach. We then use our approach with R' = b- - Duniform as described above. By Bernstein's 's inequality (Bernstein, 1924), this weighted estimate tightly concentrates around the mean, which in turn implies a high-quality approximation of m. We theoretically analyze the effect of an approximate m on the quality of the optimal solution in Sec. of the Appendix (see Lemmas and 5).
4 RESULTS
We apply our sample selection algorithm, RAD, benchmark vision data sets and evaluate its performance in generating high- nigh-performance student models on a diverse set of knowledge distillation scenarios. We compare the performance of RAD to the following: (i) MARGIN (Balcan et al., 2007; Roth & Small, 2006) as described in Sec. 3, (ii) UNIFORM, (ii) CLUSTER MARGIN (labeled CM), state-of-the-arr active learning technique (Citovsky et al., 2021), (iv) CORESET, a popular clustering- based active learning algorithm (Sener & Savarese, 2017), (v) ENTROPY, a greedy approach that picks the points with highest student prediction entropy (Holub et al., 2008), and (vi) UNIXKD (Xu et al., 2020), a -fate-of-the-- art active distillation approach based mix-up (Zhang et al., 2017). We implemented all algorithms in Python and used the TensorFlow (Abadi et al., 2015) deep learning library. We used the hyperparameters specified in the respective papers for all of the compared approaches. For RAD, we use the hicoretically-derivee setting of w = 1 m/n as specified in Sec. 3 and emphasize that this makes RAD fully parameter-free.
In Sec. D of the Appendix, we present: the full set of prperr-prrameters and experimental details (Sec. D.1); additional evaluations that report statisties beyond test accuracy (Sec. D.3); applications of RAD to the standard active learning setting and comparisons to SOTA approaches (Sec. D.4); experiments with varying w and gain definition to evaluate the robustness of RAD (Sec. D.5 and D.6, respectively); comparisons on diverse set of knowledge distillation configurations (Sec. D.7). Overall, our empirical evaluations show that RAD uniformly improves on state-of-the-ee baselines and demonstrate its off-the-shelff effecttiveness without the need to tune change any hyperparameters.
4.1 CI CIFAR100, & SVHN
Setup We use ResNet (He et al., 2015), ResNetV2-(11,, 20, 29} (He et al., 2016), or Mo- bileNet (Howard et al., 2017) with depth multiplier of as the student and ResNet-50, V2 (56, 110), or MobileNet with a depth multiplier of 2 as the teacher model. We considered the CIFARIO/CIFARI
(Krizhevsky et al.. 2009), SVHN (Netzer et al., 2011), and ImageNet (Deng et al., 2009) data sets. Unless otherwise specified, we use the Adam optimizer (Kingma & Ba, 2014) with batch size of 128 with data set specific learning rate schedules. We follow the active distillation setting shown in Alg. with various configurations. We use 64 Cloud TPU v4s each with two cores. The full set of hyper-parameters and experimental details can be found in Sec. D of the Appendix.
Configurations We experimented with a diverse set of configurations for the knowledge distillation task. We reported the specific configuration for each plot as part of the plot's title (e.g., see Fig, 3). In context of the variables in the plot heading, we varied the number of epochs that the student is trained for (denoted as e), the size of the initial set of labeled points (lAA), the number of soft-labels to query per iteration (b), and the teacher model (f); resnet in the configuration refers to ResNetV2-20 as the student and Net-5 as the teacher unless otherwise specified. All results were averaged over 10
7

---

Published as a conference paper at ICLR 2023
trials unless otherwise stated. For each trial, we reshuffled the entire data set and picked random portion (of size |A) to be the labeled data set Xz, and considered the rest to be the unlabeled set Xy.
recmettt tteache $1460) Ala00CO 1=000
nusnettf on 100 tteacher acc 22,4%0 14--00000 2e0000
mobi on Bleacher acc: 26.3%2 E4 200000000000000000000000
mecolimmtt on che "0 baacher woc: 43c%% 2000000000000000000000
Oan
Xoeo 20003
1010000))11000000000000 1000 rombbreeeeee
(e-100 113000 d0000 m000
250
lowinn 14 w1oo0 oo 1000 cenearantt))
recoecr0 on owtn thecher aedi 91 199
repnetie on olart Beacher ous 51 (%
100 -500L 10000-11000 recnetsst
1000 3000
Numberaf actt-aaele
5000 6000 200d
8000
20000
30:00
humber of gedie
4000
4003
000
2000
3030
1000
recnection on 103 tfeachee age 10,250)
recnactl tne tao (eecher ane 17.855)
orr 25.0%0
ondefest o boatther anm 61250 2000
260
(o-200) (A)-500 bo 20000-0000000000
1o-200 5000 by (o0teecseettt
Jemboo je Jewooodoo be 2000 teneereeeett
Mays
harbe sith pi
humbbar tath Meelrd points
Murker of sch wublllbeee point:
mob lonct cnciiaan teaahher 1199))
(ue M4e0000 be U0S fermitatanett
mobil otteiio thaacher aces 16.09)
10: 200 10000.0 (0 mob lonelt
onarcittoonal beasher
9000 by (-reoze15))
(conet29
10-00000 1000
24.51
10.
4200
8030
3000 3007 4000 5000 1000
1000 ADDD
9000
1000 (000) 2000
40.00
500000
rearct20 10f lseacher acz 8%5)
reanot22 dleer 100 (oaacher ace: 63.5
dil 100 teache
mnd leteer riterino Owecher wer
Je:100 (M0-20000000000000 300000000000000000000000 resteeeet
Je 10000000000000000000000000 te0t:op teseeeetiin
(n-300, 10100
10-2200 14 -00000 p00. enct!
1aader of sat Maiiddieee
Figure 3: Evaluations of RAD, state-of-the active learning, and active distillation strategies on a diverse set of distillation configurations with varying data sets and network architectures. RAD consistently outperforms competing approaches. Shaded regions correspond to values within one standard deviation of the mean.
In the first set of experiments, we evaluate the effectiveness of each method in generating high- accuracy student models subject to a soft-labeling budget on CIFARIO, CIFARIOO 100, and SVHN data sets with ResNet(y2) and MobileNet architectures of varying sizes. CIFARIO contains 50, 000 images of size 32 x 32 with 10 categories, CIFAR100 has 50. 000 32 x 32 images with 1000 labels. and SVHN consists of 73, 257 real-world images (32 x 32) taken from Google Street View. Fig. 3 depicts the results of our evaluations on a diverse set of knowledge distillation scenarios with varying configurations. We observe consistent and marked improvement in the student model's predictive performance when our approach is used to actively select the points to be soft-labeled by the teacher. This improvement is often present from the first labeling iteration and persists continuously over the active distillation iterations.
We observe that RAD performs particularly well relative to baselines regardless of the teacher's accuracy. For instance, we see significant improvements with RAD when distilling from a MobileNet teacher CIFAR 100, which has relatively low accuracy (see corresponding plots in Fig. 3). This
8

---

Published as a conference paper at ICLR 2023
observation suggests that the explicit consideration of possible teacher inaceuracy is indeed helpful when distilling from a teacher that may be prone to making mistakes. At the same time, we observe that RAD outperforms state-of- the-art art active learning algorithms such as Cluster Margin (CM) and others (MARGIN, ENTROPY) - which do not explicitly consider label noise in the form of incorrect teacher soft-labels even in instances where the teacher accuracy is as high as 92% (see SVHN plots in Fig. 3). These observations support RAD'S ability to automatically adapt its sampling distribution the applied scenario based on the approximated teacher inaccuracy.
resnet'8 on ImageNet (e-100, [A|=/0/10, t-nesnet50)
resnet18 on ImageNet (e-100, |A|-II/10, t-resnet50)
resnet'8 on ImageNet (e-100, [A|-I//10, t=resne150)
Text Acsuniny 260
Text Accuracy
Marg
Margin
-10000
Margh
Unioor
Uiilorm
51
CM
Entropy
50
CM
-20000
Intropy
QM
50
-30000
10
40
50
60
70
80
90
100
Number of soft rerr--abeeedddd points (as 25 0
spoch
Number soft-ateee aber-d points (as % or
Figure 4: The classification accuracy and gain on the ImageNet data set with ResNets.
4.2 IMAGENET EXPERIMENTS
Here, we report the results of our evaluations with ResNet architectures trained on the ImageNet data set which contains nearly 1.3 million images spanning 1000 categories (Deng et al., 2009). We exclude the UNIXKD and CORESET methods due to resource constraints and the fact that CM supersedes CORESET (Citovsky et al., 2021) and UNIXKD consistently performed poorly on the configurations in the previous subsection. This highlights an additional advantage of our approach: it merely requires a sort (O(n log n) total time). This is in contrast to computation- and memory- expensive methods like CORESET and CM that require clustering (see Sec. D.2 for details). Fig. 4 depicts the results of our evaluations, where our method consistently improves upon the compared approaches across all training epochs. Here the teacher is ResNet50 model and the student is Re model. The teacher is trained on the initial labeled dataset with ||| = 10%|/]. /|. For each choice of the budget b E (3% / 5% 5 7, ,9%[41, 11% we run trials. In the rightmost plot we observe that the gain of our approach (w.r.. = 1 margin,) is higher than that of the competing approaches, and that the gain correlates well with the test accuracy of the student, which reaffirms the practical validity of our formulation (Sec. 3).
CONCLUSION
In this paper, we considered the problem of efficient and robust knowledge distillation in the semi- supervised learning setting with a limited amount of labeled data and a large amount of unlabeled data. We formulated the problem robust active distillation and presented a near linear-time algorithm with provable guarantees to solve it optimally. To the best of our knowledge, our work is the first to consider importance sampling for informative and correctly labeled soft-labels to enable efficiency and robustness in large-scale knowledge distillation tasks. Our method is parameter-free and simple- to-implement. Our experiments on popular benchmark data sets with a diverse set of configurations showed a consistent and notable improvement in the test accuracy of the generated student model relative to those generated by state-of-the-a methods.
Limitations and future work In future work, we plan to establish a deeper theoretical under- standing on the trade-offs of the various instantiations of our general framework, (3) in Sec. 3, on the test accuracy of the student model. For example, it is not clear whether defining the gain as gi 1- margin; or g, exp(-margin,) is more appropriate, even though both definitions lead to gains that are monotonically increasing with the uncertainty of the student. Besides considering teacher accuracy in the robust formulation, we plan to also consider other relevant metrics such as student-teacher disagreement to construct more informed distributions. Overall, envision that our approach can be used in high-impact applications to generate powerful student models by efficiently distilling the knowledge of large teachers in the face of limited labeled data.
9

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Our algorithm is fully-specified (Corollary 1), simple-toimmplemnnt and parameter-free (Sec. 3). We provide the full details and hyperparameters required reproduce our results Sec. and Sec. D.I of the Appendix. We specify descriptions of how the competing algorithms were implemented, including the hyperparameter settings. We provide precise theoretical results (Sec. the main body and Sec. C of the Appendix) that clearly specify the assumptions and provide full proofs and additional helper lemmas the Appendix (Sec. C). Our evaluations use publicly available and easily accessible data sets and models.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: /www//wwwoooooo////// tensor flow..rrrwww..o.o......oooooowwo org/ Software available from tensorflow.org. 7
Olah,
Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference
on Neural Networks (IJCNN), pp. 1-8. IEEE, 2020. 2
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv: 190660366,, 2019. 3
Maria-Floring Balcan, Andrei Broder, and Tong Zhang. Margin based active learning. In International Conference on Computational Learning Theory. pp. 35-50. Springer. 2007. 3,7
Cenk Baykal, Lucas Liebenwein, Igor Gilitschenski, Dan Feldman, and Daniela Rus. Sensitivity- informed provable pruning of neural networks. SIAM Journal on Mathematics of Data Science, (1):26-45, 2022. 1
Sergei Bernstein. On modification of chebyshev's 's inequality and of the error formula of laplace. Ann. Sci. Inst. Sav. Ukraine, Sect. Math, 144):38-99 1924. 7, 18
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. arXiv preprint arXiv:2106.05237, 2021. 1,2
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. I
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp- 535-541, 2006.
Chandra Chekuri, Jan Vondrák. and Rico Zenklusen. Dependent randomized rounding for matroid polytopes and applications. arXiv preprint arXiv:0909.4344 2009. 14
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big
self-supervised models are strong semi-supervised learners. Advances in neural information processing systems, 01-22233-22255 2020.
Jang Hyun Cho and Bharath Hariharan. On the efficacy knowledge distillation. In Proceedings of
the IEFE/CVF international conference on computer vision, pp. 4794-4807 2019. 1
10

---

Published as a conference paper at ICLR 2023
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. 2
Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Rostamizadeh, and Sanjiv Kumar. Batch active learning at scale. Advances in Neural Information Processing Systems, 34, 2021. 7, 9,
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. leee, 2009. 7,9
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv 2018.
Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: a margin based approach. arXiv preprint
2018.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data.
In International Conference on Machine Learning, pp. 1183- 192. PMLR, 2017. 3
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision 129(6))1789-188))) 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. corr abs/ 12.03385 (2015), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pp. 630-645 Springer, 2016. 7
Geoffrey Hinton, Oriol Vinyals. Jeff Dean, al. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.025.:1 2(7), 2015, 1,3
Alex Holub, Pietro Perona, and Michael C Burl. Entropy-based active learning for object recogni- tion. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1-8. IEBE, 2008. 7
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint T0KKv:1704.04880 2017. 18
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint
142.::1412.6980, 2014. 7, 18
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
M Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. Advances in neural information processing systems, 23, 2010. 6
Kevin J Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and Lawrence Carin. Mixkd: Towards efficient distillation of large-scale language models. arXiv preprint arXiv:2011.00599, 2020. 3,
Kevin J Liang, Samrudhdhi B Rangrej. Vladan Petrovic, and Tal Hassner. Few-shot learning with noisy labels. arXiv preprint arXiv:2204.0544 2022.
Lu Liu and Robby T Tan. Certainty driven consistency loss on multi-teacher networks for semi- supervised learning. Pattern Recognition, 120:108 1408 2021. 2,
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150. Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http: //www. acl web. www.gg/nnhooogyyyooooo
28
11

---

Published as a conference paper at ICLR 2023
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J Reddi, Seungyeon Kim, and Sanjiv Kumar. Why distillation helps: a statistical perspective. arXiv preprint arXiv:2005.10419, 2020. 1
Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pp. 15630-15649. PMLR, 2022. 6
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011. 7, 22
John von Neumann. Zur theorie der gesellschaftsspiele. Mathematische annalen, 100(1):955-320, 1928. 5, 14
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp.
11453-11466 11464, 2021. 1
OpenAl. Openai pricing. https:/ / 2022. Accessed: 2022-08-01.
David Patterson, Joseph Gonzalez, Quoe Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104. 103550, 2021.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34: 20596-20607, 2021. 19
Mansheej Paul, Brett W Larsen, Surya Ganguli, Jonathan Frankle, and Gintare Karolina Dziugaite. Lottery tickets on data diet: Finding initializations with sparse trainable networks. arXiv preprint 0rXii::2066.01288, 2022. 19
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11557-11568, 2021. 2,
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference machine learning, pp. 4334-4343, PMLR, 2018. 6
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. survey of deep active learning. ACM Computing Surveys (CSUR), 54(9):1-40, 2021.
Dan Roth and Kevin Small. Margin active learning for structured output spaces, In European Conference Machine Learning, pp. 413-424. Springer, 2006. 2, 3, 4, 7
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint 1tXh::170.000889 2017. 3. 7, 19
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions Neural Networks and Learning Systems, 2022.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 34, 2021. 1
12

---

Published as a conference paper at ICLR 2023
Iulia Ture, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv: 2019. 28
Qizhe Xie, Minh-Thie Kinh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10687-10698, 2020. 1.2
Guodong Xu, Ziwei Liu, and Chen Change Loy. Computation-eeficien knowledge distillation via uncertainty- mixup. arXiv preprint arXiv:2012.09413 2020. 3,
Taranch Younesian, Zilong Zhao, Amirmasoud Ghiassi, Robert Birke, and Lydia Y Chen. Qactor: Active learning noisy labels. In Asian Conference Machine Learning, pp. 548 3638. PMLR, 2021. 2
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.0944 09.00...944..... 2017. 2,
Zizhao Zhang, Han Zhang, Sercan Arik, Honglak Lee, and Tomas Pfister. Distilling effective supervision from severe label noise. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 9904,,303, 2020.
Guoqing Zheng, Ahmed Hassan Awadallah, and Susan Dumais. Meta label correction for noisy label learning. AAAI 2021, 2021. 2
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
In this supplementary material, we provide the details of batch sampling (in Sec. B), full proofs of the results in Sec. and additional theoretical results (in Sec. C), and details of experiments in Sec. and additional evaluations (in Sec. D).
B IMPLEMENTATION DETAILS
To supplement our discussion in Sec. we provide additional details regarding the batch sampling procedure. One approach is to iterate over the points E [n] and pick each point with probability q:- EEEj)) 9: = b, then this procedure samples points in expectation. A more principled approach is to use randomized dependent rounding (Chekuri et al., 2009) which samples exactly b points given distribution that sums b. This procedure is shown Alg. 2, and efficient implementation of it runs in (n) time (Chekuri et al., 2009).
Algorithm 2 DEPROUND
Inputs: Probabilities p e [0, 1 such that EiE(n) P: [n] with [Z] =
Output: set indices
while E [n]] such that P: do
2: Pick i, E [n] with * j. 0 < < 1, and < <
3: a - min( (1 Pt.
4: B min
PU)
5: Update p; and Pj
(p: + a. Dj o)
8 with probability a+3
(ps. p,)
(p: B.Ps 3)
with probability
9
a+
6: end while
7: I + (i E [m] p; = 1}
return I
C PROOFS & ADDITIONAL ANALYSIS
C.I PROOF OF THEOREM 1
Theorem 1. Suppose 91 2 92 2 ... 9n > 0, and define Gk 91/ + Es) and
H₁ 1/ss:1/9:1 L.). For Gn
optimal solution p' E Ai to (3) is given by
G - m
p HE. (g. + 2.) si K- and p otherwise, where k* argmax
H
The distribution can be computed in linear time (assuming sorted g) and achieves objective value of (PT((K))
Gy- 1
Proof. The proof relies on the the Minimax Theorem (Neumann, 1928), which yields
min max f(p,c) max min f(p.)). of c AE
PEA, cEAn-m
We will further use two claims, which we'll prove shortly.
Claim 2. Let p². k", and (kk) be defined as in the statement of the lemma. Then p" E A and N (K*) = mingeA.-m (()".c).
Claim 3. Let k" and N NK) be defined in the statement of the lemma. Further, define
NO*)+4
ifi Siih k
c
getti otherwise
14

---

Published as a conference paper at ICLR 2023
Then e* An-m and N(k*) maxpEd: f(p.c*).
Given the claims, f(p" max min f(p,c) c) by Claim 2
N(k) min cEAn-m
peai cE
min max f(p c) by Minimax cEAn-m PEAt
< max f(p,c') = N (k*) by Claim 3 PEA1
That is, p" attains the maximum, which is N(k"), we wanted. We now prove the claims. Before beginning. we will first simplify { c) somewhat, finding
f(p.c) Σ gipic L Eupil1 c))
Ec + + - l,)p:
Proof of Claim 2. We first show p" E 11. We have
Σ
Σ p.
Σ
Hs*
1
HE- (gi + e,)
H
iSn
iSk
And since pt 2 for all it immediately follows that pī S as well.
We now show N(k") = -minceAn-.. f(p',c). c). To this end,
f(p" c) E(cele + +() - E.)p:
inn
Glg9 + (.) 2.)-1;
Hy (gi + (i)
Ggg + +b)
(gi + +()
Σ
Σ
Hy (gi + +4)
Hy (gi + +()
Hy- (9i E.)
isk
iSk
k"
G
4
+
H
H
Hy.
Clearly, for c* An-m this expression is minimized when & for i > k- and Eisk C4 k* So have
(6*=m) k- Gk-
min f(1,,c)= = H
oE GEAn-m
H H
G+- N(k"))
H
claimed.
We now prove our second claim.
Proof of Claim 3. We first show that N(k*) 2 9: for > k=! Observe that if then & > for non-nennenee neg-aevtivve values (and b 0, 0). Notice
b+d
G N(K) N(k* +1)
Gk- gk- +1 (g/ +1 + Lg- +1)
H
H + 1/ +1 + lx++1)
In the border that k² n,, add dummy item with ga+t and the claim follows trivially.
15

---

Published as a conference paper at ICLR 2023
By our observation,
/ + +1) N(+)) 1/ +1 +x++1)
91+ 2 9s for i > k²
Similarly, we show [(+)*) - gi for i S k", using the observation that 8- S g. then & S g for
non-negative values (and b and > 0). Notice
N - 1) Gk ga (gg. + (x.) m
Gk. m N(k*)
H 1/ 1/ + (x+)
H
And again, by our observation,
N(k)) & 9h- /gkk (xx)) 1/(gk* + (x-)
gb S 9: for S K.
(5)
Now we're ready prove the claim.
We first show e* E An-m- Since ((+))) S 9 for i 1 ke, we see of N(k")+G, og E [0, 1]. Further,
- for i
1 k. So
g.t
N(k)) +
Σ
Σ
9i +4
SS
k M'sish
N(k*) gi
9i lg
Σ
+
- li
isk
igk
=N(K") -Gp +n
=G m Gk. + = n m
That is, c¹ E An-m-
Finally, we show N(k*) = m, f(p, c²). Note that ci (.) - 44 = N(k))
for for S k", while
(9: + li) l = gi for i ks We have
f(p,c") (C)(9 + +4) l.)p
E N(K))p Σ
9:Pr
1>k"
From above, (A*)) 2 9, for i > k", So the expression is maximized for E A when = for i ket and Eisk Pt Hence,
-
PEA:
as we wanted.
C.2 EFFECT OF APPROXIMATING
Here, we prove that an approximately optimal solution can be obtained even if an approximate value of m is used (e.g., via validation data set). For sake of simplicity, the following lemma considers the case where the losses are 0 in the context of Theorem however, its generalization to general gains and losses - including the relative error formulation that we study in this paper - follows by rescaling the error parameter appropriately. Our main result is that if we have approximation i E (1+e))m, then we can use this approximate m to obtain an (1 1122mm/eee (2em (1+))) -compeiiteeeeeeeee solution.
Note that this result generalizes to the RAD relative loss with us as discussed in Sec. by considering m' (1 w)m (see Corollary l).
16

---

Published as a conference paper at ICLR 2023
Lemma 4. Let E [n] be the groundtruth value the number of incorrect examples in Xg and
let km be the optimal solution with respect to m. Assume 91 2 ... 2 9n and let a = 9km. and
2i-aa. Suppose that have approximation in of such that
E (1 + E) with E (0, (),
then the solution km with respect m m (1 EE) is (1 2 (2em + (1 + +-)==))ccomoittttt with the optimal solution, iee,, it satisfies
2em
2cm
OBI(K m)
OPT 1-
OBJ(Km:
25m + (1+
22m - (1+
where OBJ(k, m.)
k'-m k-m maxwe([) He Hy
Proof. For sake of notational brevity, let i and k- denote kn and km. respectively. First, observe that by the assumption of the lemma, we have
m (1
and m S m.
1+E
14E
Note that since & 2 by the optimality of is with respect to m, we have (1-elm follows from the optimality condition from the proof of Theorem
> which
[1+8)[-a)
k 2 -h+11 H + m 2 ak: + m
h 2
1
and 2[ Since &s and are integral, we have l: 2 + 1 2 m 1. Finally, by the optimality of ls with respect to m. have
2 k m
H₂ (k- - H
Putting all of the above together,
h: m
m)
1
k: m
m
H₂
H
h- m
(m m mk(k m)
H
(k: m))Hk.
1 m 17
H
m
m m
IV OPT
1
m
m
2mm
OPT
1 2em + (1 (1++))
where the first inequality is by the inequality on the lower bound 1 HH and l 2 m + 1, the second by m m, the third by m 1, and the fourth by
2cm
m - m < 1+
and rearrangment.
C.3 APPROXIMATING m
Next, show how to estimate using validation data set T. To do so, define
err (T. fo)
Σ 1ff(x) + y) (T.T)T
where fo(-) [R] corresponds to the label prediction of network 0. Note here that m = crr(PBs PP
where Pn corresponds to the points in dataset B.
17

---

Published as a conference paper at ICLR 2023
Lemma 5. For any d E (0,1). we validation set T of size l obtain approximation in fo) for m = crr(PB, B. then with probability at least
mm - ml I log(4/8)
1
[P +
p))
[PBl
VIPn vk
where p- = P(a.y) + y) the probability of mistabeling for network fo(.).
Proof. Let T = ((z1),9)) ( B1 ) be set of [ i.i.d. points from the data distribution D
and define X; = (1;) + y for each i E [7]. Letting X = + E. we observe that E [X] = E
(ス,0)D' [L(f((x) wH1
P (æ.y)~>
(fe(z) +
Since we have sum of k: [7| independent random variables each bounded by 1 with variance Var (X)) = p(1 - p), we invoke Bernstein's inequality (Bernstein, 1924) to obtain
-kE
P(IX - eT) S 2 exp
2(p(11 - p) + + ET/3
setting the above to 6/2 and solving for ΣT yields
X - pl ST log(4/0)
2p(1 pp) log(4/0)
k
k
with probability at least 1 5/2. Similarly, we can define the random variables Y,,..., Ypp so that Y; [s(()))) * 3 for each (z,, v:) Pp. Letting [Pl Σ. Y; and observing that R [Y] as before, we invoke Bernstein's inequality again obtain that with probability at least 5/2.
log(4/6) PPAl
2p(1 p) log( (4/6)
Y pl Ep
Pal
The statement follows by the triangle inequality YY bound.
S Y pl + - pl
and the union
D ADDITIONAL EVALUATIONS & EXPERIMENTAL DETAILS
Here, we describe the experimental details and hyperparameters used in our evaluations and provide additional empirical results that supplement the ones presented in the paper. Our additional evaluations support the robustness and widespread applicability of our approach.
D.1 EXPERIMENTAL DETAILS
We conduct our evaluations on 64 TPU v4s each with two cores. We used validation data set of size 0.00 for the CIFARIO, CIFAR100, and SVHN data sets, and used validation data of size 10, 000 for ImageNet, respectively, to estimate The hyper-parameters used with respect to each architecture and corresponding data set(s) are as follows.
MobileNet (CIFAR10, CIFAR100, SVHN For the experiments involving MobileNet (Howard al., 2017), whenever MobileNet was used student architecture it was initialized with width paramater of 1, and whenever it was used teacher, it was initialized with width parameter of 2. We used the Adam optimizer (Kingma & Ba, 2014) with default parameters (learning rate: le-3) and trained for either 100 or 200 epochs depending the experimental configuration. We did not use data augmentation or weight regularization.
ResNets and ResNetv2s (CIFAR1O, CIFAR100, SVHN We used the Adam optimizer (Kingma Ba, 2014) with the default parameters except for the learning rate schedule which was as fol- lows. For given number of epochs Repochs {100, 200), we used le-3 as the learning rate for the first (2/5)Mepochss then used le-4 until (3/5)neeochhs- 1c-5 until (4 (4/5)mepochs> 1c-6 until Repoohe and finally 5e-7 until then end. We used rounded values for the epoch windows that determine the learning rate schedule to integral values whenever necessary. We did not use data augmentation weight regularization.
18

---

Published as a conference paper at ICLR 2023
D.2 IMAGENET
Setup We used a ResNet-18 student and ResNet-50 teacher model for the ImageNet experiments. We train the student model for 100 epochs using SGD with momentum (= 0.9) with batch size 256 and a learn rate schedule as follows. For the first 5 epochs, we linearly increase the learning rate from 0 to 0.1. the next 30 epochs we use a learning rate of 0.1, the next 30 after that, we use a learning rate of 0.01, the next 20 we use learning rate of 0.001, and use a learning rate of 0.0001 for the remaining epochs. We use random horizontal flips as our data augmentation.
Methods The implementations of RAD, MARGIN, ENTROPY, and UNIFORM are the same as in our evaluations of CIFARIO/100 and SVHN in Sec However, the Cluster Margin (CM) (Citovsky et al., 2021) and CORESET (Sener& & Savarese, 2017) algorithms require expensive clustering operations and were inapplicable to ImageNet off-the-shelf due to memory and computational constraints. Nevertheless, we implemented an approximate version of CM for completeness. We choose CM over CORESET because is currently the state-of-the-ar and Citovsky et al. (2021) 1) have already demonstrated that it outperforms CORESET on large-scale settings. For our approximate version of CM, we partition the Rs 1.1M images into buckets of size 10, 000 and run HAC clustering in each bucket. We stop when the number of generated clusters reaches 500. This leads to 56, 000 clusters total, after which we apply the CM algorithm in its usual way.
D.3 BEYOND TEST ACCURACY & COMPARISON TO GREEDY
We investigate the performance of the student model beyond the final reported test accuracy and verify the validity our problem formulation. In particular, we question (i) whether our method also leads improved student accuracy across all epochs during training and (ii) whether it actually achieves a higher gain with respect to the robust distillation formulation ((4), Sec. 3) compared to using the (greedy) STANDARD MARGIN algorithm that simply picks the highest gain points and UNIFORM sampling. To this end, we conduct evaluations on CIFARIO 10, CIFAR 100, and SVHN data sets similar to those in Sec. 4, and additionally report the test accuracy over each epoch for the last knowledge distillation iteration and the realized gain over the active distillation iterations.
Fig. 5 summarizes the results of our experiments for various knowledge distillation configurations. From the figures, we observe that our approach simultaneously achieves a higher final test accuracy (first column) and generally higher test accuracy over the entire training trajectory (second column). This suggests that the improvements we obtain from our approach are consistent and present regardless of when the student training is terminated. In the third column of Fig. 5, we also observe that our approach achieves the highest realized gain among the evaluated methods and that this gain tends to be a good predictor of the method's ' performance. This sheds light into why the greedy variant (STANDARD MARGIN) that simply picks the points with the lowest margin (highest gain) is not consistently successful in practice: the high gain points are often mislabeled by the teacher, further confusing the student. This further motivates our robust formulation in Sec. 3 and supports its practicality.
D.4 APPLYING RAD TO STANDARD ACTIVE LEARNING
Here, we demonstrate the applicability of RAD to standard active learning settings and compare its performance to SOTA strategies. This is motivated by recent work that has demonstrated that selecting the most difficult or informative with respect to a proxy metric samples may in fact hinder training of the model (Paul et al., 2022; 2021). For example, on CIFAR10, choosing the most difficult instances was observed hurt training, and the best strategy was found be one where moderately difficult points were picked (Paul et al., 2022; 2021). This is method of sampling is reminiscent of the sampling probabilities generated by RAD as depicted in Fig. 2. The results of the experiments are shown in Fig. 6. RAD matches or improves on the performance of state-of-the-art techniques.
The results of the active learning experiments are shown in Fig. 6. Since there is no teacher model involved, we instantiate RAD with m = 0.05n for the number of teacher mistakes. This is based on the empirical studies showing that around 5% of the data points are inherently too difficult (Paul et al., 2022; 2021) or outliers which may impair training. Setting the appropriate value for m when applying RAD to the standard active learning setting remains an open question, and is interesting
19

---

Published as a conference paper at ICLR 2023
mobiienet difar100
mobilenet on difer100
mobilenet cifer100
[0=200] 20000,,0 2000
20000
[0-2010 04000220000000,22000000000 loneet
Dan Sandar Vargin
Ours
25-
Stedaer
Test ACC moc
22-
Oct
150
90ft
Number absted
resne129 oifar 10
resnet29 cifeno
resee029 of
30-100 LOO-SAA--SOOO--0 2-00000000000000 (esnent)5))
[e- 1000://A0000..-22000000
(6- 1000:AA-0000000000---00000--00--
$6.0
Cutt
37.5
Fandard Mergh
5700
Toos Rccuuicc
56.5
56.0
Oura
50.5
35
500C
Number libolied
Opochs
Number soft abeiled
resret20 difer100
resret20 difer00
resnet20 cifer00 2:0:::444--20002220000----000000
30-200,
Quy
Text Acturaey 73
Test Foc moy
-100
Oc's
gin
-50
Number soft-labrled
Epoch
Number
solin
resnet on syhn
resret svinn
A0-200.Al-102000 b-Iie0)tenesreet
(e-200.[Al 1000.0-000----00000000000 100..--eenee))
(e-200JA --000,-- t00ntt-resett)
Chas
Nargh
Tean Accuracy (4)
Teer Accurncy
33
25
-42
34
-50
Numberr Intedied
Number bor't pontte
Figure 5: The student's final accuracy (first column), student's test accuracy over the training trajectory (second column), and the realized gains (third column). Our approach generates student models that generally achieve higher test accuracy and gain with respect to the formulation Sec. across the evaluated scenarios.
direction for future work. The results in Fig. 6 show that RAD is competitive with state-of-the-art active learning algorithms in the evaluated scenarios and matches or improves the performance of the berf-performing active learning technique. We emphasize that, in contrast to existing clustering- based approaches such as CM Coreset, RAD achieves this performance in a computationalll--ffice and fully parameter-free way for a given m..
recnet on svltn
mobilenet on cilar10 [e=50]AA-20000.0-5333 5000500000mieeeell
resne129 on cifar10
91
66
Test (8)
62
Teaal Ausouaso (8)
65
90
87
Maige
Unform
68
2000
6030 Sumber of laboled points
1000
8000
1000
10000
14000
20000
3000
4000
1000
6000
Numoer lobllddd points
Number of labeled points
Figure 6: Evaluations in the standard active learning setting.
20

---

Published as a conference paper at ICLR 2023
D.5 ROBUSTNESS TO THE CHOICE OF
In this subsection, we evaluate the robustness of RAD by evaluating the performance of the algorithm with various instantiations for the w parameter CI CIFARIOO, and SVHN datasets and various resnet student-teacher architectures. The results of experiments comparing RAD with the default setting of w
wide range of distillation scenarios spanning
m/n as described in Sec. 3 (Ours) to RAD variants with w E {0.0, 0.1, 0.2, 0.3, 0.5, 0.7, 0.8, 1.0) are shown in Fig. 7. The results were averaged over 5 trials. As we can from the figure, the performance of RAD remains relatively consistent (generally within one standard deviation) over varying choices of w. Moreover, the theoretically-derivee choice of 1 m consistently performs well across the evaluated scenarios it is always within one standard deviation of the best-performing w for each scenario.
rssnest JAA5CCO bw5000 (e Ireeenatdd)
on Roacher Ie-1b0 31-302000 on tensmnttaa)
repreet on off 100 Coacher soc. 18 3Xd
(o-00 Wi 5000 8000
2000
(40 Jeninn
2000
6000
30000
8000
2000
Humber
Number lnteedports
menert ofterid (bacchr wod 250
> IN 5000le 5000 resienti0
resricti cil 103 Neache occ 24 9933
Iow00 -00000 1D00 caceentto
rosnictt or afor100 Blad or acc. 34.5%)
So [A)300000 20000 lecar s9560
recwettion oft inn theacher sen 25:
W. 2000
100 www.
20:00
2.04
60%%
00
3000
10000
1500
20000
1000
2000
3000
1000
1000
2002
5000
4000
4000
5000
8000
1000
Do00
reenert en spter theache soc B9 206)
resnecti on outo (saachr ace (0.1)))
resneet on adtn bescher acc (1.4%))
nennet2 caart 10 (baacnrr ace 52 2
PN-6000 1e0nniit
(0-100 3000000000000000000000000 Ameeneri1
00-100 2000000-00000ee
20.2
Tea
540
10.0
540
36.
14.0
5000
1000
1903
20000
5000
10000
1000
20000
4200
6000
5000
10000
1000
4000
8000
10000 1000
Hemmet//iin osherto thechee wos 72.380
1090iC299 on Cuschht acc. 71.61))
Peetacd ottero braccher 18)
le: 100.4 20000.6 2000 repiictin
(en1001) (4-20000 b43000 tomeraati
X00/Al 3 30000 res/e19
Mittur
Artoondel Zai
Dan
Narka of
Moder Mo
roone225
geitt (4-000000 -000ntit
jes 50 (400000 p 4000 reserwitt))
(pelthh (4 1e pererne
(2. (0-00000 petcon teeerratsl
922
no 52
800
100 --
1D0 2.8
120
2000
Number of woth lcceedpoiess
Nunber of w2th lceeddaee
burber O' LDe
Number of eatt sodedpoott
Figure 7: Comparisons of the performance of RAD with various settings of the hyper-parameter x compared to OURS, which uses the default value of w = 1 - m/n n (i.c., teacher accuracy). The overlapping performance of the various instantiations (within shaded region of one standard deviation) supports the robustness of RAD to the various settings of w E [0, 1].
21

---

Published as a conference paper at ICLR 2023
D.6 ROBUSTNESS TO THE CHOICE OF GAIN
In our empirical evaluations, we had so far only considered a specific definition of gains with respect the student's margin as described in Sec. 3, i.e., Bi margin,. Since RAD can generally be used with any user-specified notion of gain, in this section we investigate the performance of RAD when the entropy of the student's softmax prediction (student(x.) E [0, 1]kh used define the gains, i.c.
g = Entropy (fatudenn = Σ
fatudent (x:), log fstudent j=1
We label this algorithm RAD ENTROPY and compare its performance to our variant that uses the student margins.
Fig, 8 shows the results of our comparisons with varying distillation configurations, architectures, and data sets averaged over trials. Overall, we observe that the change the definition of gain does not lead to significant change (> one standard deviation) in performance.
D.7 ROBUSTNESS TO VARYING CONFIGURATIONS
In this section, we consider the robustness of our algorithm to varying configurations on fixed data In particular, we consider the SVHN (Netzer et al., 2011) data set and consider the performance with varying size of the student model (ResNetv2-(11 20, 29)).. the size of the teacher (ResNetv2-(56, 110)). A| (5000, 10000, 20000), b {1000, 2000, 5000). and number of epochs e {100, 200). Due to resource constraints, we conduct the extensive comparisons against the top-2 best performing algorithms from the main body of the paper (Sec. 4): (STANDARD) MARGIN and UNIFORM. The results of the evaluations show that our method uniformly performs better or at least as well as well as the competing approaches.
22

---

Published as a conference paper at ICLR 2023
mobiinet on clfar10 (teacher ecp: 43 7%)
mobilenet di (teacher 30.3%)
mobienet avhn (teacher acc: 2%))
[o=200_|A|300000 |A|-30000, (0t.---00000000lloo0o....
(e-200, |A--10000 10000..60000.000 b-500000-mobblleeee
46.0
85.5
46.5-
Text Acramery P96
85.0
Acturetyy
84.5
68.
64.3
46.0
Oura
83.5
435
Tearser
Teacher
5000
'0000
8000
20000
6300
5000
2000
14000
5002
10000
1:000
23000
Number soft-ibbeed
Number setellebeled
Namber soft-labeled points
mobilenet on cifer100 (reacher acc: 10.4%)
resnet on oifar10 (teacher acc: 52.3%)
resnet on cifart00 (teacher acc: 41
B0000//A--000.b----- t-mbilenn)))
11-2000MM--500,010 50000
(e-2000|M] 500000.b--0000.eeeeeee
Aogurscy
12.5
Test Accurny
8
Tiae Aecuanayy PR
40
8
11.
Oyls
BAD briweyy
53
Teatter
5000 7500 10003 2500 15000 (7500 20000
6000
10000 12000 54000
Number soft-labaled
Number soff-labeled points
Number softt labll-lbbbledee points
resnet on cifar100 (teacher acc: 17.5%)
resnet on suhn (teacher acc: 91.6%)
resnet on syhn (teacher acc: 93.3%)
11-2000AA--500.0--000 100::0000010011-1-0e0
(010000002000000000000 f-reenet)
focuraccy
19.5
Test Accurmayyy (kg)
Tean Acrureeyy th2
93.0
19.0
SI.D
$2.5
18.5
00.5
180
Curs
Cuts
92.0
1AD Entropy
BAB Erirspy
Entrapo
90.0
17.5
91.
2000
1000
2000
1300
20000
4000
Number of soft-labe points
Number soft-labeled points
Number soft-lablled points
resnet on svhn (teacher acc: 93.5%) 10000AAA-2000.-33000---ee-e
resnet on evhn (teacher soc: 93.4%)
resnet on svhn (tescher ecc: 19.1%) (8-200, 0000000000000000000--oooo0 t-resntt
1-rennee
03.50
93.4
93.25
91.0
Pocurcyyb)
Test Aocumay bia
93.2
Tees Arrusmmmy (s))
93.00
90.3
93.0-
92.76
90.0
22.0
02.25
IAB Entrogn
Oura
92.6
Outs
142 Entrogy
89.
Ours
(AO Entispy Teacher
92.00
Teacher
92.4
89.0
4000
6000
8000
10000
12000
5000
10000
15030
20000
5000
13000
15000
20000
Number of soft-isbbeleddoint
Number of soft-iabeled po nts
Number of soft-labeled points
Figure 8: Comparisons of RAD with the gains defined with respect to the student margins as in Sec. 3.1, OURS, to RAD with gains defined with respect to the entropy of the student predictions, RAD ENTROPY. RAD's performance is robust the alternative notion of uncertainty define the gains.
23

---

Published as a conference paper at ICLR 2023
resnetti on swhn
resnet1l on suhn
resnet on ewhn (c=100|AA=5000b-000000 termmett)d
88
"est-lcourecy (50
Test Aucuuncy th3
Text Aac...nnaa (N3)
89
86-
88
88
Oura
Cura
Cura
04
Manga
Margie
Standerd Margh
Uniform
85
Uniforn
Unform
86
Number
Numbergf soft lebeled
resnethi an suhn
resnetti an suhn
resnet11 on svhn
1-110//A-50000010000reeeeeee-
(e-100 10-55000--22000mmeshetttt00
fecurscy
82
Leen Accuracy (3))
Test Accuracy (K)
90
86
86
84-
Standard Margin
Ours
86
Curt
Standard Margh
Curs
Unifomm
Unifom
lnfturm
1000
10000
13000 15000
Numbor soft-laabld points
Numbor sott--tbeldddd points
Number soft-lsbeled points
Figure 9: ResNetv2-11 architecture with 100 epochs and |A|| = 5000. First row: ResNetv2256 teacher; second
row: ResNetv2-101 101 teacher. Columns correspond to batch size of 6 = 1000, & = 2000, and b 5000, respectively.
resnet1l on
onnnetllonssun suhn
resnetil de-2000M--00003:00000rreeesss
89
Test Acco enc 260
Teat Arm.rerrayy err
Oura
Curs
Cura
Soandard
Margi
Unioomm
Unifor
Unioom
86
20000
Mumber soft points
Number of
lebeled
resmetTi on swhnt
resnet11 on svhn
resnett1 on swhnn
(a-200 1006000000000001--ssee1000111111
Fesst Rocureryyy DM1
Test Acruuaay No1
[en fepuracy
87
20
89
86
Oue
Standar Ma
Curs
Sandor Morgi
38
Cure Standord Margin Unform
Untoomm
Unitom
86
1000
2000
10000 18000 Number of soft-lsbeied points
Numbor woft-otbeedd points
Numberst soft-labclle points
Figure 10: ResNetv2-11 architecture with 200 epochs and |A| 5000. First row: ResNetv2-56 teacher;
second row: ResNetv2-101 teacher. Columns correspond to batch size of b = 1000, b = 2000, and = 5000, respectively.
24

---

Published as a conference paper at ICLR 2023
retneeff on syhn
resnett1 on swhn
Fconet11 11 on svhn
1-2000AA-10000-0-- h:t:::0000000000trrereereeee
112:00AA-100000-55000rreeeaa
92.0
Test Accurecy Oel
Text Aucuvemy (8g)
Test/opurasy
91.0
910
91.5
90.5
91.0
00.0
Dura
Oua
Ous
Standend Mergin
90.0
Stadded Margi
90.5
Standard Merg
89.5
Beform
Unfom
Weloom
Number points
points
soft pointss
resnetti on syhn
regnet'1 svhn
resnett1 on syhn
(a-200, 1000000AA-100000 t0 1000,trmmmne11111)
(e-200, [e-200||A|=10000) h0t0--00000000200-eeeeeeeeeeeee
(e-200 100_:AA1110000000000000aa0e
92.0
92.5
V1.0
Test Accursty PN1
92.0
Text Aecouustry
910
913
90.0
89.5
Stendend Vergin
Ours
90.5
Stardeed Margin
Ours
01.0
-+- Ous
++- Standard Varg
Unfomm
93.0
Unform
Unloom
1000
2000
2000
1000
2003
2000
4000
6000
8000
10000
5000
10000
18000
20000
Number of softt lebe ed points
Number of softt lobeled points
Number of soft: lsseled po int
Figure 11: ResNetv2-11 architecture with 200 epochs and |A| = 10000. First row: ResNetv2-56 teacher; second row: ResNetv2-101 teacher. Columns correspond to batch size of 6 = 1000, 6 2000, and = 5000, respectively.
resnettl on svin
resnettf on swhn
resnet1f swhn
98.00
Test Aeounayy Akt
92.7
Tea Accurstty 281
02.2
92.0
02.25
Ours
92.00
Ous
Cara
Standard Mergin
92.28
Standerd Margin
Unitom
Unllom
Unifomm
9150
92.00
2000
3000
4000
2000
4000
6000
BOGD
10000
5000
10000
15000
20000
Number of soft lateled points
Number soft labeleddddeeeee coints
Number labeled points
resnettl on suhn 21-200000-1100---1ee00
resnettf on svhnn
resnetll on suhnn
20-
20-200/A//-2000 1110000002200.01eee000 t-resnetto)))
(0- 200 2000://A22200....00 b-5000 1-resnett0))
92./5
93.00
Nouuracy 688
02.50
92.28
Accour mapg
92.75
Tel: Rcouu mary
93.00
92.50
92.75
92.00
92.28
92.50
Mangerd
92.25
Stadard Ma
9150
Unform
Unitorm
92.00
2000
2000
6000
points
Figure 12: ResNetv2-11 architecture with 200 epochs and |A| 20000. row: ResNetv2-56 teacher;
second row: ResNNetv2-10 teacher. Columns correspond to batch size of = 1000, b = 2000, and = 5000, respectively.
25

---

Published as a conference paper at ICLR 2023
resnet20 on gynn
reenet20 on synn
resnet20 on synn
L0- 0o=1000/AA--0000--000000 (-resnatt6)
(=-100/A-50000.=-200000 (-resmettt))
[o-1000]AA:::000--40000 b-000-0--0000---tttt-------
91.0
91.5
eett-hcsureyy (5))
40.5
Test Aar.AAccoomno (h3
90.5
Text Arr-AAraann eeraaaeeryy PN3
91.0
90.5
89.5
90.0
Oura
Orra
90.0
Ous
Stardard Mergin
89.5
Stardard Margh
Standard Vargia
Unfoom
Unform
Welomm
005
89.5
Number points
Number points
Number soft lebeled points
resmet20 on syitin
resnet20 on gyhn
resnet20 on suhn
(e-
1e 0=110|/A-500000=5500...rreee
91.5
fccunscy188
Test Acourncy (N6)
Test Acmuracy /BQ
$1.0
90.5
80.0
90.5
90.0
89.5
Ours
Our
Our Standerd Margin llstom
Standard Margin
90.0
Stadard Margh
Instrr
Unfom
4000 Number sot--sffo points
3000
5000
8000
5000
20000 Number soft labllold points
10000
15000
Numoer soft: lsboled points
Figure 13: ResNetvv2220 architecture with 100 epochs and |A| = 5000. First row: ResNetv2-56 teacher;
second row: ResNetv2-101 teacher. Columns correspond to batch size of b = 1000, 6 2000, and = 5000, respectively.
reenet20 on syan
resnat20 on svhn
resnet20 on sunn
(e=200/Al-5000.--100) a00--500000000000rreeeeeeeee
11-2000/A:50000--200000
01.5
91.0
Test Accuu Accurncy 260
Texh Arcuuneyy 24g
91.0
91.0
80.5
90.5
30.5
90.0
Ours
Our
Stendard Margin
Margi
90.0
Stendard Margin
Unloom
90.0
Unform
Wefoom
<000
20000 soft lebeled points
points
Number points
2e on gyinn
resnet20 on syhn
resnet20 on suhn
(6-200.Al-5000 b- (000t1ir0))))0000
(e- A0-200/A1-50000-2200 w0w.:000---000.rreeettt000.
(e=200.
99.0
01.5
91.5
focuracy
91.5
Tea: Acturecyyy (B
91.0
Tesr Accurecy-
91.0
90.5
30.5
Dars
Out
Dun
Standord Margin Unform
Surdar Margin
Standerd Margin
90.0
90.0
Unfom
- Ueform
89.5
4000 Number of sotts-obbeeeee points
3000
6000 Number soft: lobeldd points
5000
20000 Number soft waw.llbblldddo points
10000
Figure 14: ResNetvv-20 architecture with 200 epochs and |A| 5000. First row: ResNetv2-56 teacher; second row: ResNetv2-101 teacher. Columns correspond to batch size of b = 1000, b = 2000, and = 5000, respectively.
26

---

Published as a conference paper at ICLR 2023
resnet20 on Syhn
resnet20 on syhn
re6ne120 on swhn
00-200/A4=103000bbee t-resnett5))
(a=200, A0-10000 (-remettt6))))
05.00
93.2
93.0
92.5
Test Pett-Accoruns Ph3
Text Accountyy (N3)
22.50
92.75
920
9225
22.50
92.
07.00
Our
924
Dus
Margh
Standard Margin
Mergh
9175
Wnfeom
Unlom
-4- Unloom
resnet20 on sunn
resnet20 on synn
resnat20 suhn
93.50
93.25
Teen Kocuracy (g)
93-0
Text Aucornayy 84
V3.25
93.00
92.8
93.00
92.78
9266
8250
Durs
MeendardMeergh
92.4
Standard Margh
Curs
92.50
MtendardhMerth
Our
92.25
Unlcom
922
Unformm
Unlomm
92.25
2000
4000
6000
8C00
12000
20200 Shhaled points
Numbor soft labaled points
softs points
Figure 15: ResNetv2-20 architecture with 200 epochs and |A| = 10000. First row: ResNetv2-56 teacher; second row: ResNetv2-101 teacher. Columns correspond to batch size of b 1000, & 2000, and = 5000, respectively.
resnet29 swhn
resnet29 on suhn
reshet29 on syhn
(--200/4-5000.b-2200 (00eeee
91.5
Ours
Standard Margin
01.5
91.5-
Test Accou www.--couuuyyyy enry 260
Tixh Arcb Accuuneyy PkG
$1.0
90.5
90.5
Our
Margin
90.0
Stendard Margin
Unform
Weloom
90.0
20200
points
Mumber points
soft points
resnet29 on suhn
resn6t29 on suhn
resnet29 on svhn
|e=200|A|
92.0
91.5
91.50
Standard
Accurscy
91.25
Tean Accurecy-
Tea: Acturnyyy B8a
91.5
91.00
20.75
90.5
Ours
Dur
90.50
Surdard Margin
903
Slanderd Margin
90.0
Unfomm
-+-
leform
2000
6000
labiled points
Number soft: points
Number soft loft-laoeeed
Figure 16: ResNettv222 architecture with 200 epochs and AA| 5000. First row: ResNetv2-56 teacher; second row: ResNetv2-100 101 teacher. Columns correspond to batch size of b = 1000, b = 2000, and = 5000, respectively.
27

---

Published as a conference paper at ICLR 2023
D.8 EXPERIMENTS WITH PRE-TRAINED TEACHER MODELS
rasnet20 difar100 (teacher acc: 43,8%) [e-10C, (00/AA/--1000000000000 trrssnettt 6-100;
reanet20 on svtn tteocher ncc: 92.8%)
resnet29 on cifer100 (teecher ecc: (700%
(o-100]AA]- 10t0//100000--000
(e-100 [A] 10000.000000000000 10000000-000.00mm t0eete 1010
Cortec
Vergi
Un fu Toather
Test Rocurasyyy
Teacter
48
25
DM
30
To:
Outa
Coococt onerapy
Mergt
20
20
5000 100CO 1500 20OCO 260CO 90300 36000 Nurbbar of aoft abeled poirts
5000
10000
15030
20003
5000
10000
1500
20000
humbar of soft labelld paints
humber of soft labeled points
resnettl on suhn (teacher acc: 93.9%)
resnet20 on oifar100 (teacher acc: 47.2 2%2
resnet29 on cifar100 (teacher acc: 58.8%)
04-
CG-10D//A--DD8 10000, 12200,0-30000----00 www.eett-e..e100----- 100)
45
Margin
Test Neaurecy
93
80
Teacher
Tost www.roocuaaooo 750
-0
Lefoom Teacher
CM
Ours
Mergin
Corcest
Uniform OM
Brivpy
Teacher
20
5000
10000
15000
20000
25000
5000
10000
waon
23000
5000
100000
150000
20000
25000
Number of sott Lebwlrd points
Number of soft-ibeeed points
Number of sch--teeeedddpeee points
Figure 17: Evaluations on with a ResNet teacher model pre-trained on ImageNet and fine-tuned on the respective data sets. RAD uniformly performs least well as the best comparison method, and often better especially in the low sample regime.
In this section, we present evaluations with ResNet50 and ResNet101 teacher models that are pre- trained on ImageNet and fine-tuned on the labeled data that is available for the academic data sets consider. Fig. 17 depicts the results of our evaluations on CIFAR 100 and SVHN datasets. Consistent with the trend of our results in Sec. 4, RAD uniformly outperforms or matches the performance of the best performing comparison method across all scenarios.
D.9 NLP EVALUATIONS
on imdo reviews (teacher acc: 76.7%)
bert on Imdo_ reviews (teacher aoc: (e=10.A)-10000, www.000.bb000..bbrrrreee.
imolo reviews (teacher acc 79, (at0A/10000000000--ee0000
77.6
80.0
75.0
64
77.6
1850 Heconecyy
22.5
Iee Accuraay
85
CM
75.0
72.8
70.0
Out
Coreeit
Our
67.
Hergh
Intrapy
Magin
brovpyy
65.0
2500
6000 7500 io0co 12500 1o00 Number of soft laooke points
12000
5000
6000
7000
8000
5000
10000
1000
2000
3000
4000
5000
40OC
Number of cof:-bbolcd points
Numbor of soft www.-iooccccppccoooooce points
Figure 18: Evaluations on the IMDB dataset with tiny BERT student model and pre-trained BERT teacher.
We conclude the supplementary results by presenting evaluations on Natural Language Processing (NLP) task on the IMDB Reviews (Maas et al., 2011) 1) dataset with a pretrained BERT teacher model. The IMDB dataset has 25,000 training and 25,000 testing data points, where each data point is a movie review. The task is to classify each review as either positive or negative. We used a pre-trained, 12-layer SmalIBERT (Ture et al., 2019) with hidden dimension 768 as the teacher model and a randomly initialized 2-layer SmallBERT with hidden dimension 128 as the student. From Fig. 18 we that the improved effectiveness of RAD relative to the compared approaches persists on the NLP task, consistent with our evaluations on the vision datasets. RAD is particularly effective in the small sample regime, where the number of labelld points is small relative to the size of the unlabeled dataset.
28


Reviews:
[{"review_id": "BDgQmSOUEn", "reviewer": "Reviewer_fKaY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies robust active distillation that can simultaneously achieve efficiency and robustness. The query rule of the algorithm follows from the solution of a minimax problem that tries to maximize the \"gain\" given a limited number of mislabels. The minimax optimization problem can be solved in nearly linear time (with closed-form solutions provided in Thm 1 and Cor 1).  The authors conduct extensive experiments which empirically confirm the efficacy of their proposed method.", "summary_of_the_review": "Based on the detailed reviews above, I think this paper has proposed some interesting and novel ideas. I also would like to hear the author's rebuttal regarding the weaknesses/questions. \n\n====after rebuttal====\n\nThank you for your response. I would like to keep my current scores. I believe the paper can be made stronger if the authors can provide some analyses for the case with b>1.", "strength_and_weaknesses": "Strength: \n1. The authors proposed an interesting minimax optimization problem to select data points, which ensures both efficiency and robustness. The idea behind this optimization problem seems novel to me.\n2. The authors identify a theoretically-motivated weight parameter such that the expected gain is never negative. As a result, the proposed algorithm can be operated in a parameter-free manner. \n\n\nWeaknesses:\n1. The minimax problem seems to be only solved in the case when b=1 (in Thm 1). When the budget is larger than 1, it seems that the authors simply multiply the solution (obtained when b=1) by the budget. Can we directly solve the minimax optimization problem when the budget is greater than 1?\n2. The choice of m is estimated based on sample inaccuracy, which seems to require hard-labeled data points. Are there ways to get around with this?\n\nQuestions: \n1. In algorithm 1, why both the teacher and the student are trained on the same hard-labeled data points? I thought the teacher should be trained on a much larger dataset in order to provide soft-labels that are better than the ones generated by the student.\n2. Following the above question, can you provide more explanations on why the student can eventually outperform the teacher in your experiments?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8ibJrvIuuiq", "reviewer": "Reviewer_FQ16", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors presented a model distillation method that can actively select the training samples for distillation. The method is efficient and robust - in the sense that the method suffers less from inaccurate soft labels. The proposed method is parameter-free. The authors further verified the efficacy of the proposed method empirically.", "summary_of_the_review": "This paper proposed an intuitive, interesting and effective approach for active distillation. With the theoretical understanding and the extensive empirical results, I think this paper is above the acceptance threshold.", "strength_and_weaknesses": "Strength:\n\n1. The author presented a mathematical formulation that captures the objective of training on informative soft labels that are accurately labeled by the teacher.\n\n2. The solution takes a simple and intuitive game-theoretical viewpoint, which can be obtained in near-linear time without tuning the hyperparameter.\n\n3. The author also presents extensive empirical evaluations that demonstrate the effectiveness of the proposed framework.\n\nQuestions and Weaknesses:\n\n1. At the beginning of page 3, the author state that \"a very limited number of ground truth label queries can be made for the points in $X_i$ throughout the distillation process\". From my understanding, the ground truth label queries are different from querying the teacher model for soft labels. How exactly are those ground truth label queries used?\n\n2. The game-theoretic approach can deal with very bad quality soft-label, which seems somewhat pessimistic. Is there any way we can estimate the quality of the soft-label without using oracle? For instance, what will happen if we use the margin of the teacher-generated soft-label as a proxy for $c_i$ in equation (2) ?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0qxSod-yQ3", "reviewer": "Reviewer_26jz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an active learning-based approach for knowledge distillation. It assumes that the teacher model does not have access to a fully labelled dataset either, and there is a cost associated to every call of the teacher model to produce soft labels for the unlabelled data. The aim is to minimize the total cost of making teacher model inference calls while maximizing the student accuracy learned from the soft labels produced by the teacher model. Its core idea is to select the samples where the teacher model is more confident instead of those where the student model is most unconfident. Experimental results on multiple real datasets confirm the effectiveness of the proposed technique. ", "summary_of_the_review": "The paper considers the problem of importance sampling for simultaneous efficiency and robustness in knowledge distillation which is an interesting idea. The proposed solution is supported by non-trivial theoretical analysis and strong results on real datasets. Overall, the paper presents a solid contribution that has the potential to intrigue follow-up studies. \n\n=== Update after rebuttal ==\n\nThanks for the response. I have no further questions.", "strength_and_weaknesses": "Strengths:\n\n1. The proposal that considers the problem of importance sampling for simultaneous efficiency and robustness in knowledge distillation is novel and interesting.\n\n2. The proposal is supported by non-trivial theoretical analysis. \n\n3. The proposed techniques are shown to be effective on multiple real datasets. \n\nWeakness:\nThe paper motivates with KD using large NLP models such GPT-3. It would be good to incorporate experiments on NLP datasets and tasks. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster