Teacher Guided Training: An Efficient Framework for Knowledge Transfer
Authors: Manzil Zaheer
Ankit Singh Rawat
Seungyeon Kim
Chong You
Himanshu Jain
Andreas Veit
Rob Fergus
Sanjiv Kumar
Venue: ICLR 2023 poster (2023)
Keywords: Distillation
Semisupervised learning
Efficient machine learning
Generalization bounds
knowledge distillation

Abstract: The remarkable performance gains realized by large pretrained models, e.g., GPT-3, hinge on the massive amounts of data they are exposed to during training. Analogously, distilling such large models to compact models for efficient deployment also necessitates a large amount of (labeled or unlabeled) training data. In this paper, we propose the teacher-guided training (TGT) framework for training a high-quality compact model that leverages the knowledge acquired by pretrained generative models, while obviating the need to go through a large volume of data. TGT exploits the fact that the teacher has acquired a good representation of the underlying data domain, which typically corresponds to a much lower dimensional manifold than the input space. Furthermore, we can use the teacher to explore input space more efficiently through sampling or gradient-based methods; thus, making TGT especially attractive for limited data or long-tail settings. We formally capture this benefit of proposed data-domain exploration in our generalization bounds. We find that TGT can improve accuracy on several image classification benchmarks as well as a range of text classification and retrieval tasks.

Published as a conference paper at ICLR 2023
TEACHER GUIDED TRAINING: AN EFFICIENT FRAMEWORK FOR KNOWLEDGE TRANSFER
Manzil Zaheeri, A Ankit Singh Rawat", Seungyeon Kim, Chong You, Himanshu Jain, Andreas Veit, Rob Fergus, Sanjiv Kumar Google Research and DeepMind, New York, USA
{manzi 1 ankitsraww seungyeonk @google..com com (cyou, himj, aveit, robfergus, sanjivk) Egoogle.com
ABSTRACT
The remarkable performance gains realized by large pretrained models, e.g., GPT-3, hinge on the massive amounts of data they are exposed to during training. Analo- gously, distilling such large models compact models for efficient deployment also necessitates large amount of (labeled or unlabeled) training data. In this paper, we propose the teacher- teacher-guided training (TGT) framework for training a high-quality compact model that leverages the knowledge acquired by pretrained generative models, while obviating the need to go through large volume of data. TGT exploits the fact that the teacher has acquired good representation of the underlying data domain, which typically corresponds to a much lower dimensional manifold than the input space. Furthermore, we can the teacher to explore input space more efficiently through sampling or gradient-bass based methods; thus, making TGT especially attractive for limited data or long-tail settings. We formally capture this benefit of proposed data-domain exploration in our generalization bounds. We find that TGT can improve accuracy on several image classification benchmarks well as range of text classification and retrieval tasks.
INTRODUCTION
Recent general purpose machine learning models (e.g.. BERT Devlin et al. 2019), DALL-E (Ramesh) al. 202T), SimCLR (Chen et al. 20000a), Perceiver (Jaegle et al. 2021), GPT-3 (Brown et al. 2020)). ),
trained broad data at scale, have demonstrated adaptability Despite being trained in unsupervised (or so-called superviddr) fashion, these models have been shown to capture highly specialized information in their internal representations such relations between entities Heinzerling & Inui (002)) or object hierarchies from images (Weng et al. 2021).
diverse range of downstream tasks.
Despite their impressive performance, the prohibitively high inference cost of such large models prevents their widespread deployment. A standard approach to reducing the inference cost while preserving performance is to train a compact (student) model via knowledge distillation (Bucilua et al. 2006 Hinton et al.) (0155) from a large (teacher) model. However, existing distillation methods require a large amount of training data (labeled or unlabeled) for knowledge transfer. For each data point, the teacher must be evaluated, making the process computationally expensive (Xie et al.) 2020d He et al. 2021 [Sanh et al. 2019a). This compounded by the need repeat the distillation process separately for every downstream task, each with its own training set. Enabling efficient distillation thus an important challenge. Additionally, minimizing the number of distillation samples would especially benefit low-data downstream tasks, e.g., those with long-tails.
Another inefficiency with standard distillation approaches that within each evaluation of the teacher, only the final layer output (aka logits) is utilized. This ignores potentially useful internal representations which can also be levered for knowledge transfer. Various extensions have been proposed in the literature along these lines (sce, e.g.. (Sun et al., (2000] Aguilar et al. 2020 Li et al.) 2019 et al. 2019) and references therein). However, despite their success, such extensions mostly the teacher model in black-box manner, and do not fully utilize the domain understanding it contains (Cho & Hariharan] 2019; [Stanton et al. [02II. In these approaches, the teacher is used passively as the input sample distribution remains fixed and does not adapt the student
1 Equal contribution.

---

Published as a conference paper at ICLR 2023
model performance. Consequently, these forms of distillation do not lead to faster training of high-performance student model.
In this work, go beyond the passive application of large teacher models for training compact student models, and leverage the domain understanding captured by the teacher to generate new informative training instances that can help the compact model achieve higher accuracy with fewer samples and thus enable reduced training time. In particular, we propose the teacher guided training (TGT) framework for more efficient transfer of knowledge from large models to a compact model. TGT relies on the fact that teacher's internal representation of data often lies in a much smaller dimensional manifold than the input dimension. Furthermore, we can use teacher to help guide training by identifying the directions where the student's 's current decision boundary starts to diverge from that of the teacher, e.g., via backpropagating through the teacher to identify regions of disagreement.
We provide a theoretical justification for the TGT algorithm, showing that leveraging the data repre- sentation of large models ensures better general- ization for the student. Given n instances in a D-
Teacher Encoder
Data X' & k
Teacher Libeler
Losc
dimensional space the generalization gap for learn-
ing a Lipschitz decision boundary of a classification
Latent Space
Teacher Decoder
Student Model
task decays only as O( n +) (Görf et al. 2002). In contrast, provided that the large model learns good data representation in a d-dimensional latent space, the TGT framework realizes generalization gap of (n & +WD, D))). where W(D, Dt) de- notes the Wasserstein distance between the data dis- tribution D and the distribution D' learned by the generative teacher model. Typically d < D, thus TGT ensures much faster convergence whenever use a high-quality generative teacher; making TGT especially attractive for low-data or long-tail regimes.
Search Parth wnlaters space ds
ac
DC
Figure 1: An overview of the proposed teacher guided training (TGT) framework. Given learning task, the framework leverages large teacher with pretrained generator and la- beler that exhibits high performance on the task. In particular, we assume that the generator consists of an encoder and a decoder. TGT performs three key operations during student model training: (1) Given an original training instance, by using the teacher generator, identify a novel task-relevant instance. We search for informative instances in the lower dimensional latent space, where we can propagate the gradi- ent to. (2) Obtain (soft) labels for the original and newly generated training instances from the teacher labeler; and (3) Minimize the student training objective that depends on the original and newly generated instances along with their labels produced by the teacher labeler.
To realize TGT, we take advantage of the fact that most of the unsupervised pretrained models like Transformers, VAE, and GANs have two compo- nents: (1) an encoder that maps data a latent representation, and (2) a decoder that transforms the latent representation back the original data space. We utilize this latent space for the data representations learned by the teacher model effi- ciently search for the regions of mismatch between the teacher and student's 's decision boundaries. This search can take the form of either (i) a zero-order approach involving random perturbation or (ii) a first-order method exploring along the direction of the gradient of suitably defined distance measure between the teacher and student models.
Many pretrained models, particularly in NLP such as T5 (Raffel et al. 2020), can also provide labels for a downstream task and act sole teacher. However, approach sufficiently general to utilize separate pretrained models for generative and discriminative (labeler) functions (cf. Fig. [ e.g., employ a BiGAN generator and EfficientNet as labeler for image classification task.
Our main contributions are summarized as follows:
1. We introduce the TGT framework, conceptually simple and scalable approach distilling knowledge from a large teacher into a smaller student. TGT adaptively changes the distribution of distillation examples, yielding higher performing student models with fewer training examples.
2. We provide theoretical justifications for utilizing the latent space of the teacher generator in the TGT framework, which yields tighter generalization bounds.
3. We empirically show the superiority of TGT to existing of distillation methods on both vision and NLP tasks, unlike most prior work which is specialized to domain.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Our proposed TGT framework can be considered a form of data augmentation where data is dynam- ically added at points of current discrepancy between the teacher and student. Next, we provide a brief overview of how data augmentation has been used in the context of distillation and distinguish our work from these existing efforts.
Using pseudo labels. The earliest line of work involves using consistency regularization (Sajjadi ct al., 2016 Tarvainen & Valpola 2017) to obtain pseudo labels for unlabelled data while a model is expected to make consistent predictions on an unlabeled instance and its augmented versions, cf. (Miyato et al.] [019] Xie et al. 2020a Sohn et al.] 2020; Zhu et al., 202T] inter alia). Another approach is self-truining (Xie et al. 2020d Du et al. 2021) where one learn smaller teacher model on the labeled data which then generates pseudo labels for a large relevant unlabeled set. A large student model is then trained on both labeled and pseudo labeled sets. Label propagation (Iscen et al., 2019) is another direction where unlabeled instances receive pscudo labels based on neighboring labeled instances in suitably constructed similarity graph.
Furthermore, prior work on learning to teach (Fan et al. 2018 Raghu al. 2021 Pham et al. 2021 dynamically updates the teacher so as to provided more valuable pscudo labels based on the student loss. Such an interactive approach presents challenging optimization problem and potentially opens up the door for borrowing techniques from reinforcement learning. In contrast, our work focuses on the setting where high-quality pretrained teacher model is fixed throughout the training. We focus on setting where updating the large teacher model is prohibitively costly or undesirable as such model would potentially be used to distill many student models. Morcover, many large models like GPT-3 may only be available through API access, thus making it infeasible to update the teacher.
Using pretrained models. One can use large pretrained class conditional generative models like BigGAN (Brock et al. 2019) or VQ-VAE2 (Razavi et al., 2019) to generate more data for augmentation. Despite evidence (Webster al., 2019) that GANS are not memorizing training data, using them to simply augment the training dataset has limited utility when training ResNets (Ravuri & Vinyals 2019b a). Lack of diversity (Arora et al.) 2017) in data generated by GANs, especially among high density regions (Arora et al., 2018) is a potential reason for this. In contrast, we use generative models to adaptively explore the local region of disagreement between teacher and student opposed to blindly sampling from the generative model. This way we circumvent the excessive reliance on samples from high density regions which often have low diversity.
Another line of work by Chen et al. (2020b) combines unsuperviseed/sel--upervvii (on unlabeled data) with SimCLR-based approach (Chen et al. 2020a), task-specific finetuning (on labeled data), and distillation (natural loss on labeled and distillation loss on unlabeled data). Our work is very close to this line of work with two key differences: (1) We assume access very high- quality teacher, which is potentially trained on much larger labeled set, to provide pseudo labels; (2) We go beyond utilizing a given relevant unlabeled dataset and explore the dynamic generation of domain-specific unlabeled data by leveraging the representations learned by pretrained models. Additionally, we develop a theoretical framework to establish the utility of unlabeled data instances for student training, specifically the instances generated based teacher learned representations.
pretraining
Using both pseudo labels and pretrained models. The GAL framework (He et al. 1021] previously considered generating training instances by using pretrained generator models along with pseudo- labelers. However, the GAL framework generates these new instances in an offline manner at the beginning of student training. In contrast, our approach (cf. Fig. generates new informative instances in an online fashion to attain high-performance and reduce training time for the student.
Recently, MATE-KD (Rashid et al. 2021 also used generator model to obtain new training instances based the student's current performance (by looking at the divergence between the student and teacher predictions). However, there are two key differences between our TGT approach and the MATE-KD framework: First, their method updates the teacher so as find adversarial examples for the students, which can cause the generator to drift away from the true data distribution. Second, they introduce perturbations in the input space itself and do not leverage the latent space of the teacher, which is the crux of our method. See Appendix A for further details.
Notably KDGAN (Wang et al.) 2018) leverages GAN during distillation. However, it samples examples from a GAN without taking student's 's performance into account. (Heo et al.) (2019);] Dong et al. 2020) search for adversarial examples during distillation. However, their search also does
3

---

Published as a conference paper at ICLR 2023
not depend on student's performance, resulting in wasteful exploration of those regions of the input spaces where the student is already good. Further, unlike TGT, they search examples in the input space which is often inefficient due to the large ambient dimension of the input space.
Finally, data-free KD approaches (Nayak et al. 2019. Yoo et al. 2019, Chen et al. 2019 use only synthetically generated data for knowledge distill ation. Unlike TGT, in such approaches, the synthetic data distribution is updated at each epoch, which causes the student model to lose the information over epochs and experience accuracy degradation (Binici al. 2022). In this framework, Micaelli & Storkey (2019) targeted generating samples that would cause maximum information gain to the student when learned, however, also suffers from similar drawbacks MATE-K noted above.
TEACHER GUIDED TRAINING
We begin by formally introducing setup in Section 3.1
We then describe our proposed TGT framework Section 3.2 and present its theoretical analysis in Section 3.3
3.1 PROBLEM SETUP
In this paper, focus on multiclass classification task where given an instance X the objective is to predict its true label y E y == [K] out of K potential classes. Let D) ;= 'D Dx.y denote the underlying (joint) data distribution over the instance and label spaces for the task. Moreover, we use Dx and xyix=r to denote the marginal distribution over the instance space X and the conditional label distribution for given instance æ, respectively. A classification model f X + R with f(x) = (f ( f(x) x), takes in an input instance x and yields scores for each of the K classes. Finally,
given a (tractable) loss function l : IR k [K] + E which closely approximates model's misclassificatiin error on an example (x. y), e.g., softmax-bar -base cross-entropy loss.
Given i..dd labeled samples glabeled {( ys heein) generated from D and collection of allowable models F, one typically learns a model via empirical risk minimization (ERM):
fn arg min JEF n EEE(m) {(f(x.).)))
1
(1)
In our TGT setup, we further assume access to a high quality teacher model, which has: Teacher generator. A generative component that captures DD well, e.g., transformer, VAE, ALI-GAN. This usually consists of an encoder Enc X + Rd and a decoder Dec IR 1 X.
Teacher labeler. A classification network, denoted by h X R with good performance on the underlying classification task. In general, our framework allows for h to be either a head on top of the teacher generator
independent large teacher classification model.
Given glabeled and such a teacher model, our objective to learn a high-quality compact student (classification) model in F, as assessed by its misclassification error D.
3.2 PROPOSED APPROACH
To train a student model f E F, we propose to minimize:
R. RTTTTgglabell (glabeled]
Σ (e(f(z:). (a(((z)) ve(m)
1 Σ ((((E)))
(2)
m 5E[m]
where la R x R - a loss function that captures the mismatch between two models f and h, and 8,, [ )ie[m] is introduced in subsequent passage. The first term, (x;)), v.). corresponds to standard ERM problem (cf. Eq. (T). The subsequent terms, (a(f(z;), h(z;)) and la(f(E,) h(ā,)) do not make use of labels. In particular, the second term, (a(f(x;), h(x:)), corresponds to the standard knowledge distillation where the teacher h provides supervision for the student f.
We introduce novel third term, (f(f) h( where 8, = {2,} is generated based on 8, {v,). Here, we want to generate informative instances 8.n that will help student learn faster, e.g., points the data manifold where the student disagrees with the teacher. In other words, want to find i as follows:
T E arg max (f(),),, (x)) such that PDx (x) 2 A
#EX
4

---

Published as a conference paper at ICLR 2023
Note that the objective and constraint in Eq. ensure that select an instance where the student and teacher disagree and the instance belongs to a region where true data distribution assigns a non-trivial mass, respectively. Based this, propose two specific approaches to generate the novel samples Sm
1. Isotropically perturb in latent space:
Ø Dec(Enc(()) + 1/) where v ~ N/ o0)))
This can be regarded as a zero-order search in the latent space, which satisfies the constraint of remaining within the data manifold.
2. Gradient-based exploration: Run few iterations of gradient ascent on Eq. (3) in order to find the example that diverges most with teacher. To enforce the constraint, run the gradient ascent the latent space of the teacher generator as opposed to performing gradient ascent in the instance space X, which might move the perturbed point out of the data manifold. For a high-quality teacher generator, the latent space should capture the data manifold well. To implement this need to backprop all the way through the student and teacher-labeler the teacher-decoder, as shown in Fig. Mathematically, it involves the following three operations:
2 := Enc(z);
2
+ 2 La (f (Dec(z)) h(Dec(z)));
f == Dec(z).
This is akin to first-order search in the latent space.
Extension to discrete data. Note that perturbing an instance from discrete domain, e-g., text data, is not as straightforward as in a continuous space. Typically, one has to resort to expensive combinatorial search crude approximations perform such perturbations (Tan et al., 2020 Zang ett al. 2020; Ren et al., 2019). Interestingly, our approach in Eq. (4) provides a simple alternative where one performs the perturbation in the latent space which is continuous. On the other hand, in gradient based exploration, we assume that X is differentiable space in order to calculate necessary quantities such as @f(x) in Eq. 5 This assumption holds for various data such as images and point clouds but not for discrete data like text. We can, however, circumvent this limitation by implementing weight sharing between the output softmax layer of the teacher's decoder Dec and the input embedding layer of the student / (and also to teacher labeler h when an independent model is used). Now, one bypass discrete space during the backward pass, similar to ideas behind VQ-VAE (Hafner et al., 2019). Note that, during forward pass, we still need the discrete representation for decoding, c.g., using beam search.
Finally, we address the superficial resemblance between our approach and adversarial training. For latter, the goal is learn a robust classifier, i.e., to increase margin. Towards this, for any x, one encourages model agreement in its local neighborhood (x),), i... f(x') f(x). Vx E (x)).. One needs carefully choose small enough neighborhood by restricting so as not cross the decision boundary. In contrast, we are not looking for such max-margin training which has its own issues (Nowak- Vila et al. 202T). We simply desire global agreement between the teacher and student, i... f(z') h(x'), Vz'. As result,
explore much bigger regions long remain the data manifold, i... PDx (x) is non-trivially large.
3.3 VALUE OF GENERATING SAMPLES VIA THE LATENT SPACE
Now, we formally show how leveraging the latent space can help learning. For this exposition, we assume X = TR. Furthermore, for directly learning in the input space, we assume that our function class F corresponds to all Lipschitz functions that map R2 to g For any such function f e F. existing generalization bounds take the form (Devroye et al. 2013 Mohri et al. 2018
(D, < (8,,) (9,.)) +0 (V(log(1//8)/)) <0(n 1/P)
where (D)) is true population risk of the classifier, Re,r(Sn) is empirical risk, and (9, F) the Rademacher complexity of the induced function class Ge,F, which known in our case to be 1nnn11/0 (see Appendix B for more details). Note that any reduction in the Rademacher term would imply smaller generalizing gap, which is our goal.
5

---

Published as a conference paper at ICLR 2023
In our TGT framework, we assume availability of teacher that is able to learn good representation for the underlying data distribution. In particular, we assume that, for x E (Dpp(Dx), we have
|Decc Enc(x) - x=|| S
i.c., for æ, applying the decoder Dec on the latent representation of æ, as produced by the encoder Enc, leads to point Dec o Enc(x) E X that approximates z with a small error.
This ability of teacher generator to model the data distribution using latent representation can be used to reduce the complexity of the function class needed. Specifically, in TGT framework, we leverage the teacher decoder to restrict the function class to be a composition of the decoder function Dec and a learnable I. Lipschitz function operating on the latent space RR. Since d < D, this leads to function class with much lower complexity. Next, we formally capture this idea for distillation with both the original samples Sn sampled from Dx as well as the novel samples introduced by the teacher generator. In what follows, we only consider the distillation losses and ignore the first loss term (which depends on true labels). Our analysis can be easily extended to take the latter term into account (e.g.. by using tools from Foster al. (2019)).
We start with the standard distillation in the following result. See Appendix C.I for the details.
Theorem 3.1. Suppose a generative model with Enc and Dec satisfies the approximation guarantee functions, and the distillation loss La satisfies Assumption C.1 Then, with probability at least 1 ò, the following holds for any f F.
in Eq. (6) for Dx. Let Dec and teacher labeler h be Lipschtiz,functions,
Re, SR (Sn) + (oh +0 vlog(1/0)) LE Le+OOVKEDY [[Prix - x(x))l2))
Vn
50(n
where is the effective Lipschitz constant of St,.f = f2 + la(f Dec(z), Dec(z)) : F} an induced function class which maps Rdd (latent space of generator) to R.
Dec
Thus, we can reduce the Rademacher term from (11)) to O(n-l/d), which yields significant reduction in sample complexity. However, as the teacher model is not perfect, a penalty is incurred in terms of reconstruction error L.c and prediction error O (VKEDx [I|'y|x h(X)ll2)):
Thus far, we have not leveraged the fact that we can also use the teacher to generate additional samples. Accounting for using samples 8,, (ef. Section 3.2). one can obtain similar generalization gap for the distillation based on the teacher generated samples:
Theorem 3.2. Let 8n (E(n be i.i.d. samples generated by the the TGT framework, whose distribution be denoted by Dx. Further, let fo E F denote the student model learned via distillation
on 8.. with h as the teacher model and la be the distillation loss satisfying Assumption with probability at least 1 5, have
C. Then,
(V
(1og(1/0) + W(Dx, Dx)
(D) < Rhh Ia
(8m)) + RR (C +0
n
50(n 1/4)
+ O(VKEDx D ((x)112) where Dla is defined in Thm. 3.1
Please see Appendix C.2 for more precise statement and proof of Thm. 3.2 Comparing with the generalization gap for standard distillation (cf. Thm. [.I.p the generalization gap for TGT in Thm. 3.2 does not have the reconstruction error related term Le. Thus, by working with the samples with exact latent representation, TGT avoids this reconstruction error penalty. On the other hand, generalization gap for TGT does have additional term W(Dx, Dx), capturing the mistmatch between the original data distribution and the distribution the samples used by TGT.
As guiding principle, Thm. 3.2 dictates that one should select teacher generator that minimizes W(Dx. Dx). Similarly, the teacher labeler should ensure small prediction error O OVKKEdx Dy|x h(X)ll2] for the underlying classification task.
Motivation for gradient- gradient-based exploration. Our theoretical results so far do not throw light on the particular utility of the gradient-based exploration in Eq. (5). In this regard, we provide variance-based

---

Published as a conference paper at ICLR 2023
Appreach
Architecture
Balanced Accuracy
parameters
FLOPs
Logit adjustment loss Menon 2021b
ResNer-50
50.4
M
4.1
LDAM-DRS-RSG Wang 2021
ResNeXt-50
4.2
DRAGON Bal'T Los' Samuel
2021
ResNet-10
46.5
5.4 M
819 M
ImageNNel
DRAGON Lai'Loss Samuel
ml.
2021
ResNet-50
53.5
26 M
4.1
flabeler)
EfficientNes-bt
79.2
One-hot
MobileNeeV3-0.75
M
Distillation
MobrileeeVV300.55
M
TGT (random)
MobilcNerVV3-0.75
53.2
4.01 M
156 M
TGT (gradient-based)
MobileNeeV3-0075
4.01 M
M
I.WS (Kang al.
2020
ResNeXt:50
33.9
M
4.2
DRAGON Bal
Samuel al. 2021
ResNet-101
36.1
M
7.6
SUN397-LI
flabeler) model
EfficientNet-b3
65.3
MobhileNeVV3.0.55
Distillation
MobileNeeV3-00.55
M
M
(random)
MobiileeetV3-0.75
M
TGT (gradient-based)
MobileNerV3--075
44.7
4.01 M
156 M
Table Performance of TGT and various baselines on long-tail image classification benchmarks (see Appendix E for results on Places-LT). Rows with denote results taken from Menon et al. (2021b) and the rest were taken from (amuell al. (002)) We report top-1 accuracy on balanced eval sets. We also state the number of model parameters and inference cost (in terms of FLOPs) for all the methods. Note that TGT leads to performance improvements over standard distillation on all three datasets, particularly for ImageNet-LT where the teacher generator models the task distribution well. TGT also often outperforms stated baselines that rely on much larger and expensive models.
Such bounds suggest that, besides minimizing the discrepancy W(Dx. x, Dx).. an ideal Dx should reduce the variance of la (f(ā), h(ā)) for newly generated instances. Incidentally, the sampling approach realized by the gradient- based exploration Eq. (5) aims to achieve this: it controls for W(Dx, Dx) by operating in the latent space of good quality teacher generative model and minimizes variance by finding instances with high loss values through gradient ascent, thereby striking a desired balance between the two objectives. See Appendix C.3 for detailed discussion.
generalization bounds [Maurer& & Pontil 2009) in Appendix C.3
EXPERIMENTS
now conduct comprehensive empirical study of our TGT framework in order to establish that TGT (i) leads to high accuracy in transferring knowledge in low data/long-tail regimes (Section 4.1: (ii) effectively increases sample size (Section 4.2h: and (iii) has wide adaptability even to discrete data domains such text classification (Section 4.3) and retrieval (Section 4.4)
4.1 LONG-TAIL IMAGE CLASSIFICATION
Setup. We evaluate TGT by training student models three benchmark long-tail image classification datasets: ImageNet-LT (Liu et al.) 2019c), SUN-LT (Patterson & Hays 2012). Places-LT (Liu et al. 2019cb We employ off-the-shelf teacher models, in particular BigBiGAN (ResNet-50) (Donahue & Simonyan 2019) and EfficientNet-B3 (Xie et al. 2020c) as the teacher generator and teacher labeler models, respectively. We utilize MobileNetV3 (Howard et al.) (219)) as compact student model architecture. The teacher-labele model is self- trafnerained on JFT-300M (Sun et al. (017). and then finetuned on the task-specific long-tail dataset. The teacher generator trained on the unlabelled
full version of ImageNet (Russakovsky et al. 2015).
The results are reported in Table 1 compared with similar sized baselines (we ignored gigantic transformer models). We see that TGT is able to effectively transfer knowledge acquired by the teacher during training with the huge amount of data into a significantly smaller student model, which also has lower inference cost. TGT considerably improves the performance across the board over standard distillation, even on Sun-LT and Places-LT whose data distribution does not exactly match to the distribution that the teacher's generator trained with. That said, the gains from TGT are more pronounced when the mismatch between the task data distribution and the
Results.
Results for Places-I.T and additional baselines for ImageNet-I.T and SUN-LT in Appendix

---

Published as a conference paper at ICLR 2023
Method
Architecture
Amazon-5
IMIDB
MNLI
Yelp-5
2.5k
3M
2.5k
650k
UDA (Random Init) Xie et al
2020b
BERT base
55.8
58.6
UDA (Pretrained) Xic cl 2020b
BERT base
62.9
67.9
Layer-wise Distillation (Son
2020
MobileBERT
93.6
83.3
MATE-KD (Rashid et
DistiBERT
85.8
Teacher (labeler) model
RoBERTa large
67.6
96.2
90.6
72.0
One-hot (Random Init)
DistiBERT
44.3
53.6
50.0
63.0
50.4
58.1
One-hot (Pretrained)
DistiBBRTT
55.9
66.3
93.6
84.1
59.1
67.3
Distillation (Randorn Init)
DistiBERT
56.5
65.3
87.9
77.4
54.8
69.5
Distillation (Pretrained)
DisulBERT
60.2
66.8
94.0
84.5
63.2
71.4
TGT (Random Init)
DistBBERT
61.3
66.6
91.0
79.3
62.0
70.4
TGT (Pretrained)
DistBBRRT
64.6
67.1
95.4
86.0
68.6
71.7
Table 2: Performance of TGT and various baselines from the literature on four text classification benchmarks. For student model training, we show results for task-specific finetuning on both randomly initialized and pretrained DistilBERT models. Note that TGT (Pretrained) - TGT with a pretrained student model outperforms all other methods across the board. Even more interestingly, on Amazon-5 and Yclp-5, TGT with randomly initialized student, i.c., TGT (Random Init), outperforms the standard approach of finetuning pretrained model with one-hot labels, ie.., One-hot (Pretrained).
distribution modeled by the generator is very large, which the case for ImageNet-LT, The fact that TGT (random) (cf. Eq. (4) provides large gains over standard distillation establishes the value of utilizing the latent space, as suggested by our analysis Section 3.3 Note that TGT (gradient-bass bbsedd) brings further gains over TGT (random). particularly on SUN-LT and Places-LT which are extremely long-tail. We believe that gradient-basee where data is extremely sparse or where isotropic random perturbation in the latent space does not produce diverse enough instances. systematic study of this constitutes an interesting avenue for future research. Owing to its computational efficiency, we focus on TGT (random) for rest of paper.
first-order exploration specifically useful for settings
Note that some of the baselines Table rely on specialized loss functions and/or training methods designed for long-tail settings, whereas we do not leverage such techniques. Combining the TGT framework with a long-tail specific loss function as opposed to using the standard cross-entropy loss function can potentially improve its performance. We leave this direction for future explorations.
4.2 TGT IN LOW-DATA REGIME
65
To further showcase effectiveness of knowledge trans- fer via TGT, we simulate a low-data regime by vary- ing the amount of available training data for Ima- geNet (Russakovsky et al. 2015) and studying its impact student's performance. We use the same model architectures as in Section 4.1 but finetune the teacher labeler the entire ImageNet. We then compare the performance of the student trained via TGT, with the students trained via normal training (with one-hot labels) and standard distillation.
Accuracy
55
50
One hot Distilation TGT
0.1
0.2
0.3 0.4
0.5
Fraction of Dataset
Figure 2: Comparison among normal training (one-hot), standard distillation (distillation), and TGT simulated low-data regimes. We imitate a low-data regime via subsampling the ImageNet training set and evaluate the trained student models the entire eval set. We em- ploy 450k training steps for normal training and standard distillation, and 112k training steps for TGT. TGT outperforms other meth- ods in less training steps, thus, effectively simulating an increased sample size.
Fig. 2 shows that both TGT and standard distillation utilize additional training data more effectively than normal training, with TGT being the most efficient of the two. Interestingly, employing TGT is equivalent to an increase in sample size by 4x, compared to the normal training. This verifies that TGT generates informative training instances for the student.
4.3 TEXT CLASSIFICATION
Setup. We evaluate the proposed TGT framework on four benchmark text classification datasets: Amazon-5 (Zhang et al. 2015), IMDB (Maas al. 2011). MNLI (Williams al.) 2018), and Yelp-5 (Zhang et al.) 2015). Following Xie et al. (2020a), we also consider an extremely sub-sampled version of 5mazon-5 and Yelp-5 consisting of only 2.5k labeled examples. Again, we utilize off-the-
8

---

Published as a conference paper at ICLR 2023
shelf teacher models, in particular BART-base (Lewis et al. 2020) and RoBERTa-large large (Liu et al. 2019a) as the teacher generator and teacher labeler, respectively. Following Rashid et al. (2021), I. we employ DistilBERT (Sanh et al. 2019b) model student architecture. Both teacher networks pretrained on a very large generic text corpus of size 160GB. The teacher labeler is finetuned on each task-specific dataset while the teacher generator is not specialized to any specific task.
Results. We compare TGT with other data augmentation and distillation baselines in Table 2 Note that TGT considerably improves the performance and beats the state-of-the-arr methods MATE- KD (Rashid et al., 202T) and UDA (Xie et al. 2020a). Interestingly, by using TGT on a randomly initialized student, we can match the performance of finetuning (with one-hot labels) a pretrained model on Amazon-5 and Yelp-5. We highlight that baselines such as MATE-KD always work with a pretrained student model. Thus, the improvements realized by TGT with a randomly initialized student demonstrates enormous saving in overall data and training time requirement as it eliminates the need for pretraining on a large corpus. This further establishes that TGT can enable data-efffiieet knowledge transfer from the teacher to the student.
4.4 TEXT RETRIEVAL.
Method
recall @@20
recall 100
Setup. Finally, we evaluate TGT on Nat- ural Questions (NQ) (Kwiatkowski et al.]
Teacher (labeler) model
0.7957
0.8855
One-hot
0.6453
0.8198
2019) a text retrieval benchmark. The task is to find matching passage for a question, out of a large candidate pas- sage corpus (21M). We use RoBERTa-Baan Base dual-encoder model Oğuz et al. (2021) as teacher labeler and BART-base (Lewis)
Distillation (standard)
0.7486
0,8608
Uniform negatives
0.7536
0.8496
TGT (ours)
0.7659
0.8763
Table Performance of TGT and various baselines on the NQ retrieval task. (Kwiatkowski et al., 2019). The teacher labeler follows the setup of (Oguz et al. 202I) I) that pretrains RoBERTa-base and also PAQ (Lewis et al. to NQ (Kwiatkowski al.
et al. (0200) as teacher generator. We uti-
lize DistilBERT dual encoder model as our student architecture. We follow the stan- dard retrieval distillation setup where the teacher labeler provides labels for all the within-batch question-to-passae pairs for the student match.
large corpus 2021) and then finetuned
2019 BART-base (Lewis)
et al. 2020) is employed to serve as a task-agnostic generator. All student models follow the architecture of DistilBERT(Sanh et al., 2019b)). TGT significantly outperforms standard training (One-hot) and teacher- label only distillation (Distillation). TGT closes the teacher-stude student gap by 37% at @20, 63% at @ 100) compared to the standard distillation. See Appendix F.4 for more details on the experimental setup.
Besides one-hot training and standard dis- tillation, we consider another baseline, namely uniform negatives. In uniform neg- atives, for each question-tor -totioosn--- pair in NQ, we uniformly sample 2 additional passages from the passage corpus during training. TGT instead dynamically generates 2 confusing passages for each question-passage pair with BART generator, infusing the isotropic perturbation as per Eq. (4).
Results. 3 shows that TGT significantly improves performance, closing the teacher-student gap by 37% at recall and 63% at recall 100 compared to the standard distillation. Unlike TGT, uniform negatives only partially helped (slight improvement on recall @ 20 but degradation one recall 100 compared to the standard distillation). A plausible explanation is that, due to the extremely large passage corpus (2 uniformly sampled passages not very relevant to the matching to the matching pair.
pair in NQ. TGT instead generates informative passages that are close
5 CONCLUSION AND FUTURE DIRECTIONS
We have introduced a simple and theoretically justified distillation scheme (TGT) that adaptively generates samples with the aim of closing the divergence between student and teacher predictions. Our results show it to outperform, in aggregate, existing distillation approaches. Unlike alternative methods, it is also applicable to both continuous and discrete domains, the results on image and text data show. TGT is orthogonal to other approaches that enable efficient inference such as quantization and pruning, and combining them is an interesting avenue for future work. Another potential research direction is to employ TGT for molti data which would require accommodating multiple generative models with their own latent spaces, raising both practical and theoretical challenges.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
TGT framework relies on the availability of good-quality teacher for the underlying domain to provide efficient distillation. The impact knowledge distillation on transferring the teacher model's biases to the resulting student model is far from well understood. Moreover, the teacher generator that TGT utilizes are often large pretrained models trained on lot of unfiltered data. As result these large models can have potential biases without the awareness of the user. Also, how various biases present in the generator impact the student model's faimess/bbiss is not addressed in our work. A deeper study of this issue is required for proposed method, as well as for the knowledge distillation as ML technique in general.
REFERENCES
Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, and Chenlei Guo. Knowledge distillation from internal representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7350-7357, 2020.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (GANs). In Proceedings of the 34th International Conference on Machine Learning, pp. 224 232, 2017.
Sanjeev Arora, Andrej Risteski, and Yi Zhang. Do GANs learn the distribution? some theory and empirics. In International Conference on Learning Representations, 2018.
Kuluhan Binici, Nam Trung Pham, Tulika Mitra, and Karianto Leman. Preventing catastrophic
forgetting and distribution mismatch in knowledge distillation via synthetic data. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 663-671, 2022.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Kerbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Cristian Bucilua, Rich Caruana, and Alexandru Niculescu-Mizil Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '06, pp. 535-541, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933395. doi: 10. 150402. 50402.1150464.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aww margin loss. In Proceedings of the 33rd International Conference Neural Information Processing Systems, pp. 1567-1578, 2019.
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5514-3522. 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, pp. 1597-1607, Jul 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. In Advances in Neural Information Processing Systems, volume 33, pp. 22243-22255. 20200..
10

---

Published as a conference paper at ICLR 2023
J. Cho and B. Hariharan. On the efficacy of knowledge distillation. In 2019 JEEE/CVF International Conference on Computer Vision (ICCV), pp. 4793 4793-4801, Los Alamitos, CA, USA, nov 2019. IEEE Computer Society.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HIT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Pupers), pp- 4171-4186, 2019.
Luc Devroye, László Györfi, and Gábor Lugosi. A probabilistic theory of pattern recognition, volume 31, Springer Science & Business Media, 2013.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. Advances in Neural Information Processing Systems, volume 32, 2019.
Zihe Dong, Xin Sun, Junyu Dong, and Haoran Zhao Adversarial metric knowledge distillation. In 2020 the 6th International Conference on Communication and Information Processing, pp- 159-164, New York, NY, USA, 2020.
Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Veselin Stoyanov, and Alexis Conneau. Self-training improves pre-training for natural language under- standing. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5408-5418, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/20211 maacl m2.n.426..
Yang Fan, Fei Tian, Tao Qin, Xiang- Yang Li, and Tie- Yan Liu. Learning to teach. In International Conference on Learning Representations, 2018.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
Dylan J Foster, Spencer Greenherg, Satyen Kale, Haipeng Luo, Mehryar Mohri, and Karthik Sridha- ran. Hypothesis set stability and generalization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché Buc, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Lec-Ad Gottlieb, Arych Kontorovich, and Robert Krauthgamer. Adaptive metric dimensionality reduction. Theoretical Computer Science, 620:105-118 2016.
László Györfi, Michael Kohler, Adam Krzyžak, and Harro Walk. A distribution-free theory of nonparametric regression, volume 1. Springer, 2002.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference Machine Learning, pp. 2555-2565 PMLR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Xuanli He, Islam Nassar, Jamie Kiros, Gholamreza Haffari, and Mohammad Norouzi. Generate, annotate, and learn: Generative models advance self-training and knowledge distillation. arXiv preprint arXiv:2106.06166 2021.
Benjamin Heinzerling and Kentaro Inui. Language models as knowledge bases: On entity representa- tions, storage capacity, and paraphrased queries. In Proceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main Volume, pp. 1772-1791, On- line, April 2021. Association for Computational Linguistics. doi: 10. http:////222..aacmmmi..111..1.
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge distillation with adversarial samples supporting decision boundary. AAAI' 19/AAI 19AAAAAAAAAAAAAAAAAAA 19. AAAI Press, 2019. ISBN 978-1-57735-889-11
11

---

Published as a conference paper at ICLR 2023
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.025.: 2015.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi- supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In Marina Meila and Tong Zhang (eds.), Pro- ceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4651-4664. PMLR, 18-24 Jul 2021.
Xianyan Jia, Shutao Song. Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, et al. Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes. arXiv preprint arXiv: 11205, 2018.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04900 2020.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466 16633466,220 2019.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings the 57th Annual Meeting of the Association for Com- putational Linguistics. pp. 6086-6096, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 1t.18863//11/11-1611
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising eequence-to-scquee pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880 2020.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. Paq: 65 million probably-asked questions and what you can do with them. Transactions of the Association for Computational Linguistics, 9:1098-111 -115, 2021.
Hao-Ting Li. Shih-Chich Lin, Cheng- Ych Chen, and Chen-Kuo Chiang. Layer-level knowledge distillation for deep neural network learning. Applied Sciences, 9(10):196, 2019.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zetllemoee, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining 2019a.
approach. arXiv preprint
Ziwei Liu, Zhonggi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019b.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019c.
12

---

Published as a conference paper at ICLR 2023
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Anmual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 201 1. Association for Computational Linguistics.
A. Maurer and M. Pontil. Empirical bernstein bounds and sample-variance penalization. In Proceed- ings of the 22nd Conference on Learning Theory (COLT), June 2009.
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective distillation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7632-7642. PMLR, 18-24 Jul 2021a.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations, 2021b.
Paul Micaelli and Amos J Storkey. Zero-shot knowledge transfer via adversarial belief matching. Advances in Neural Information Processing Systems, 32, 2019.
Takeru Miyato, Shin-Ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1189):199--199 2019. doi: 10. 109.T00MM....21..22555 2.2588811
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018
Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. Zero-shot knowledge distillation in deep networks. In International Conference on Machine Learning, pp. 4743-4751. PMLR, 2019.
Alex Nowak- Alessandro Rudi, and Francis Bach. Max-margin is dead, long live max-margin! arXiv preprint arXiv:2105. 2021.
Barlas Oğuz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Wen-tau Yih, Sonal Gupta, et al. Domain-matched pre-training tasks for dense retrieval. arXiv preprint arXiv:2107.136622 2021.
Genevieve Patterson and James Hays. Sun attribute database: Discovering, annotating, and recogniz- ing scene attributes. In 2012 IEEE Conference Computer Vision and Pattern Recognition, pp. 2751-2758. IEEE, 2012.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoe V. Le. Meta pseudo labels. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 11557-11568. June 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
Aniruddh Raghu, Maithra Raghu, Simon Kornblith, David Duvenaud, and Geoffrey Hinton. Teaching with commentaries. In International Conference Learning Representations, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831, PMLR, 18-24 Jul 2021.
Ahmad Rashid, Vasileios Lioutas, and Mehdi Rezagholizadeh. MATE-KD: Masked adversarial TExt, a companion to knowledge distillation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1062-1077 Online, August 2021. Association for Computational Linguistics. doi: 10. 00.18853/v1/200 acl-
13

---

Published as a conference paper at ICLR 2023
Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019a.
Suman Ravuri and Oriol Vinyals. Seeing is not necessarily believing: Limitations of biggans for data augmentation. Learning from Limited Labeled Data: ICLR 2019 Workshop, 2019b.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buee E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1085-1097, Florence, Italy, July 2019. Association for Computational Linguistics.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115
(3):211-254) 2015. doi: 10. 1263-015-0816-y.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transforma- tions and perturbations for deep semi-supervised learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
Dvir Samuel, Yuval Atzmon, and Gal Chechik. From generalized zero-shot learning to long-tail with class descriptors. In Proceedings of the IEEE/CVE Winter Conference Applications of Computer Vision (WACV), 2021.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint 1eXivv:9900.1 2019a.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019b.
Edgar Schönfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. Generalized zero-shot learning via aligned variational autoencoders. red, 2:D2, 2019.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 596-608. Curran Associates, Inc., 2020.
Samuel Don Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew Gordon Wilson. Does knowledge distillation really work? In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In 2017 IEEE International Conference Computer Vision (ICCV), pp. 843-852, 2017. doi: 10. 00.100//CCV.2011..99
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4332, Hong Kong, China, November 2019. Association for Computational Linguistics.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: compact task-agnostic BERT for resource- devices. In Proceedings of the 58th Annual Online, July 2020.
Meeting of the Association for Computational Linguistics, pp. Association for Computational Linguistics.
14

---

Published as a conference paper at ICLR 2023
Samson Tan, Shafiq Joty, Min- Yen Kan, and Richard Socher. It's morphin' time! Combating linguistic discrimination with inflectional perturbations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2920-2922 Online, July 2020. Association for Computational Linguistics.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models:
consistency targets improve semi-supervised deep learning results. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
C. Villani. Optimal transport: old and new. Springer Verlag, 2008.
Jianfeng Wang, Thomas Lukasiewicz, Xiaolin Hu, Jianfei Cai, and Zhenghua Xu. Rsg: A simple but effective module for learning imbalanced datasets. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3784-3793, June 2021.
Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Kdgan: Knowledge distillation with generative adversarial networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Ryan Webster, Julien Rabin, Loic Simon, and Frédéric Jurie. Detecting overfitting of deep generative networks via latent recovery. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11255-11774, 2019. doi: 10. 10.11099CVPR.229900115..
Zhenzhen Weng, Mehmet Giray Ogut, Shai Limonchik, and Serena Yeung. Unsupervised discovery of the long-tail instance segmentation using hierarchical supervuperi Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2603-2612, 2021.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pp. 1112-1122. Association for Computational Linguisties, 2018.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6256-6268. Curran Associates, Inc., 2020a.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 33, 2020b.
Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020c.
Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020d.
Jaemin Yoo, Minyong Cho, Taebum Kim, and U Kang. Knowledge extraction with observable data. Advances in Neural Information Processing Systems, 32, 2019.
Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. Word-level textual adversarial attacking combinatorial optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8066-6088 Online, July 2020. Association for Computational Linguistics. doi: 10. 1/216655/112222.accmmmmmm
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28:649-657, (00-657,2111 2015.
Zhaowei Zhu, Tianyi Luo, and Yang Liu. The rich get richer: Disparate impact of semi cumi-supereieed learning. arXiv preprint arXiv:2 10.06282, 2021.
15

---

Published as a conference paper at ICLR 2023
A FURTHER COMPARISON WITH MATE-KD
MATE-KD (Rashid et al. 2021 alternative trains generator model and student model, with the hope of generating most adversarial examples for the students during the training. This cause the generator to drift away from true data distribution. contrast, we keep the pre-trained teacher-gen teacher-generator model fixed throughout the training process of the student. Our objective behind employing the generator model is to leverage the domain knowledge it has already acquired during its pre-training. While we do want to generate 'hard instances' for the student, we also want those instances be relevant for the underlying task. Thus, keeping the generator fixed introduces a regularization where the training instances the student encounters do not introduce domain mismatch.
Keeping in mind the objective of producing new informative training instances that are in-domain, we introduce perturbation in the latent space realized by the encoder of the reacher-generato model (see Figure This is different from directly perturbing original training instance in the input space itself, as done by MATE-KD. KD.. As evident from our theoretical analysis and empirical evaluation, for fixed teacher-gente generrrrr model, employing perturbation in the latent space leads to more informative data augmentation and enables good performance on both image and text domain.
B BACKGROUND AND NOTATION
For a, R. we use (b) denote that there exits a constant y > such that a Zybb. Given collection of n i.i.d. random variables Ur = (wI..... ........ Un) Du and function 7 :ul R, we define the empirical mean of (r(u)).... (u,)) as
C generated from distribution
Eu, [T()]= = Σ T(u:).
kE(M)
For the underlying multiclass classification problem defined by the distribution ②) == 2==D xxy, we assume that the label set y with K classes takes the form [K] == {l...., K}. We use F denote the collection of potential classification models that the learning methods is allowed to select from, namely function class or hypothesis set:
FC
(8)
which is subset of all functions that map elements of the instance space X to the elements of IR
Given a classification loss function & RRK - and a model f : X - R- and a sample glabeled {( ys (e())) generated from D, we define the empirical risk for f E F as follows. 8n
Re. (8. (glabeled) = Egiatettdd [(((()))
n Sein] ((f(x.).
Further, we define the population risk for f associated with data distribution D as follows.
(D) = (E(])))
(10)
Note that, when the loss function é is clear from the context, we drop é from the notation and simply use Ry (glabeled) and Ry (D) to denote the the empirical and populations risks for f. respectively.
Given function class F, the loss function & induces the following function class.
ge.F (z. v) (f()))
F}.
(11)
Definition B.1 (Rademacher complexity of Gr,F). Now, given a sample glabeled ((r,, v.) Mu)eeen ~ 2> and vector (a.,.... 0m) [+1 with i.i.d. Bernoulli random variables, empirical Rademacher complexity Ris (9..F.))) and Rademacher complexity RR, (Gr,F) are defined as
1
Radeeld (9r,F) =
-F
Err
sup gEst,F
Σ
71
and
H. (Gt,))
EgnDn Rigihhatan (Gr,z)]
(12)
16

---

Published as a conference paper at ICLR 2023
Let S, {2; higjn] be set of unlabeled samples generated from Dx. Then, given teacher model
h: X + TRK and distillation loss la TR TR k R. define the empirical (distillation) risk
for f E F to be
R (8.) = Es, (ea(I(X),
Σ ((((ai)) EEim
(13)
T.
Accordingly, the population (distillaaiion) risk for f defined as
(D) == Ex-Dx (X(
(14)
Again, when la is clear from the context, we simply use R,, and R2 to denote the empirical and population distillation risk for respectively.
Note that, for (student) function class JF and a teacher model h, la produces an induced function class Gea. (F), defined as
(a-
(15)
Definition B.2 (Rademacher complexity of F. Given a sample S,, = {2: he|n] D3 and vec- tor (o....., om) {+1,-1} with i.idd Bernoulli randoms variable, empirical Rademacher complexity Rs, (g and Rademacher complexity Ri, (g are defined as
T Σ
Rs, (G),
- Fo sup DEG
ang(aa))
(16)
n
and
Rr (9)) = ESSDDR
[Rg, (9
(17)
respectively.
C DEFERRED PROOFS FROM SECTION
3
C.I PROOF OF THEOREM
In this subsection, we present a general version of Theorem 3.1 Before that, we state the following relevant assumption on the distillation loss la.
Assumption C.1. Let É RK x y + R be bounded loss function. For a teacher function h: X K the distillation loss la takes the form
Ea(f(z),(z))
Σ h(x), ((f(x).)).
VE[K]
Remark C.2. Note that the cross-entropy loss (f(())), k(z)) = --E, h(x)y log (f(x),) here, one of the most common choices for the distillation loss, indeed satisfies Assumption C.I
The following results is general version of Theorem 3.1 in the main body.
Theorem C.3. Let generator with the encoder Enc and decoder Dec ensures the approximation guarantee in Eq. (6) for x. Let Dec and teacher labeler be Lipschtiz,uunctions F be function class of Lipschitz functions, and the distillation loss ta be Lipschtiz. Then, with probability at least 1 8. the following holds for any f F.
R (Dx) Rh (8.) O(n-1))))))))))) + Le + +o(V (
(log(160)
(18)
For the sake of brevity, simply include the softmax-operation in the definition of h and f. i.c., h(z) and (ar valid prohahility distributions over y [K].
17

---

Published as a conference paper at ICLR 2023
where L denotes the effective Lipschitz constant of the induced function class J in Eq. 15 Additionally, if the distillation loss la satisfies Assumption C.7 with a classification loss f, then Eq. (18) further implies the following.
(D) I ( (6,)) + O + Le 10(
log(1/0) + OOVK EDX [1 - h(x))l
(19)
Proof.
Note that
R (Dx) = Edx MxX(
- Es., (alx) + sup
Es., (X(()(Xx,4())) - EDX (oxxla(((()). M(X))
SEF
Es, [a([(())))
sup gEg
Es,, [a(() - ED [o()
(1) Es, (e)(((((())) Rgg (C
(20)
where (i) follows from the definition of Gh in Eq. 15 and (i) follow from the standard sym-
metrization argument (Devroye al. 2013 Mohri al. 2018). Next, we our focus to the empirical Rademacher complexity Rs. (.).))) Recall that S, {01, x2,..., Fn contains n i.i.d.
samples generated from the distribution Dx. We define another set of n points
S, = f+ = Dec Enc(x11,.... -n = Dec o Enc(s,))
follows from our assumption the quality of the generator (cf. Eq. (6 that
||Dec Enc(x;) xill Se, Vi E [nx].
(21)
Note that
Rg. (gh..))
E sup Ear(o=)
REC a
where or denote a vector with n i.d Bernoulli random variables.
Rs.
(G
1 -
Es
sup
Σ oi (g(ā:)) u(āx)) + g(xx))
gEgh
E
sup
Σ
(ig(T;) +
gEgs F
i
Eg
sup
Σ [o((9(x)) - g((t)))
n
gEg f4
1
Eg
sup
Egg((E))
sup
Σ (g(x;) - g(5))
n
gEgp air
gEsp
n
Ex
sup
Eorg(T.)
+ -
Σ L. 1|:0- -5:1 3:ll
n
gEg fg.5
"1
1
Eo
sup
Eag((T.)
+
n
gEst k
< n
-
1
sup nEGR
Σ (Deg(De((zs)
+ Te,
(22)
18

---

Published as a conference paper at ICLR 2023
where Enc(x)), for i [n]. By definition of G (Dec(c)) (a(f(z), k(z)) for some F. Now, we can define new function class from IRd R:
hh, Dec F = =z + La(f Dec(z), h Dec(z)) : F}.
(23)
Therefore, it follows from Eq. and Eq. 23 that
Rs. (Ghg.)) S Re, (eh.Dee) + Le,
(24)
where e,, (Enc(z1),. .... Enc(x,)) R. It follows from the standard concentration results for empirical Rademacher complexity around Rademacher complexity that with probability at least -
Rg, (er S R. (9
[V( log (
(25)
Since f E F, and Dec are Lipsehitz functions, Ge collection of Lipschitz functions from Re
R. Thus, it follows from the standard results (Gottlieb et al. 2016 Theorem 4.3) that
Ri, ( C.g.Dee
Fa.F
S ((n=aa) +).
(26)
Now, Eq. (18) follow from Eq.
20
Eq. 24
Eq.
25 and Eq. 26 Finally, Eq.
19 follows by
combining Lemma D.4 with Eq.
18
C.2 PROOF OF THEOREM 3.2
Here, present the following result, which along with Theorem the main body.
C.5 implies Theorem 3.2 stated in
Theorem C.A. Let & (ei)ee(n) be i..dd samples generated from distribution DD x. Further, let fn E F denote the student model learned via distillation on Sn with and la as the teacher model have
and distillation loss, respectively. Then, with probability at least
Rh in.jn (Dx) S Rh Ra.f
(8.) W(Dx, Dx) + R, (Gh @ (V log (
:
)
(27)
where Ri (G = EgnDi Rgg (Ch denote that Rademacher complexity of the induced
function class F. defined in Eq. (1)). I is constructed with the TGT frumework based on generator with the encoder Enc and decoder Dec, then Eq. (27 further specialized to
ta-f (Dxx S R R. (8.) + W(Dx, Dx) + (g + OV 1og (1) 4
(28)
where Gl,Dr defines the following induced function class from Ted (iee, the latent space of the generator) to TR
FaiF {2 + lalf Dec(z),
(ec())) :
(29)
Proof.
Note that
Rhh Raiia (Dx) = E,y (X)([(
s (Es,)(a(((xX))xx()))
sup JEF
Eg [ea((((), - Eb (((((()
Eg, [o(X)] - Ex [(())
sup
gEg
(x,, Rg. (G
(30)
where the last two inequality follows from the definition of Gh x (cf. Eq. 15 and the standard symmetrization argument Devrryee et al. 2013 Mohri et al. 2018 respectivery.
19

---

Published as a conference paper at ICLR 2023
Now, the standard concentration results for empirical Rademacher complexity implies that, with probability at least 1 - ó, we have the following.
Rg (G S Eguton Rig (G + +0(11000
log (1) n
-
(31)
= (9 0 log (z)
(32)
follows from Lemma D.3 that
Rh La (Dx) S
(Dx) + M(Dx,Dx Rh La
(33)
Now the first part of Theorem C.4 as stated in Eq. 27 follows by combining Eq. (30 Eq. Eq. 33
31 and
now focus on establishing Eq. (88). Note that, for a sample S,. = (......, Fn. generated by TGT framework, there exists ..1,.... Fn
C Rd such that
z Dec(z)), Vi [n].
(34)
Thus,
Rg (Gl, = Eo
Σ oig(Ts)
sup
gEst
gESt
(
Eg
sup 1
Σ (re((De(22))
n1
<
EE
sup
Rg, Ear
(35)
where (1) employs Eq.
34 Thus, combining Eq.
30 and Eq. 35 gives us that
R
(Dx)
- Eg, (X),
h(X) + Rg, (Gh.
(36)
Now, similar to the proof of Eq. (27 we can invoke Lemma D.3 and the concentration result for empirical Rademacher complexity to obtain the desired result Eq. (28) from Eq. (36).
Remark C.5. Note that, if the distillation loss E satisfies Assumption C.1 with loss function then, one can combine Theorem C.4 and Lemma D.4 to readily obtain bounds on R Re (D) with an additional term
OVK Exx [| A(X)ll2)) This term captures the quality of the teacher labeler h.
C.3
WEIGHTED ERM: AN ALTERNATIVE TRAINING PROCEDURE FOR TGT
Note that given the samples S, (ār) generated from D.x and a teacher labeler h. we minimize the following empirical risk for student training:
R R
(8m)
Σ M())) =E
(37)
However, as we notice in Theorem C.4 this leads to an additional W(Dx. Dx) penalty term in the generalization bound. One standard approach to address this issue to consider the following weighted empirical risk.
A.IS R
(S (8,)
1 Σ (a(f(e.)
PDy (3:) Pix (T:)
(38)
n
20

---

Published as a conference paper at ICLR 2023
where PDx and Piy denote the probability density function (pdf) for Dx and Dx we define a new induced function class related to the weighted empirical risk:
Accordingly.
PDx (T)
g {x (a(f(E.),
FF
(39)
PDx (7)
Importantly, we have
18. (Dx) E
En.
R R = R (Dx)
(40)
Thus, following the analysis utilized in Theorem of the form.
C.4
obtain high probability generalization
A.IS (9 R (Dx) S Rh, n.
+
log (t) :
(41)
which avoids the W(Dx, x, Dx) term.
In what follows, we explore an alternative approach to highlight the importance of the sam- TGT. By leveraging the variance- based generalization
pling approach adapted by (gradient-bact)
bound (Maurer & Pontil 2009) that were previously utilized by Menon et al. (2021a) in the context
distillation, obtain the following result for the weighted empirical risk in Eq. (38)
Proposition C.6. Let h, la and 8n be as defined the statement of Theorem C.4
Further, assume
IS that d.f (f) == (a)) h(T))
po (2) is bounded for all ā supp(Dx) Then, for any J F,
PDx (2)
the following holds with probability at least 1 - 5.
, < Rtat 12/..((8,, (8.) (1),
1.11
(42)
where (I) denotes
(V
Var.g (c,,f (2)) log( M(n)
log Min)
Here, M(n) sups. cx. N/1 n, h. (8,,,, ). loc), with N(c, N (8n): (1s0) denoting the covering number (Devroye el al. 2013) of the set
h 1.
(S.) == ((g(x)) ..... g(xx))
g F).
Proof. By utilizing the uniform convergence version of Bennet's inequality and uniform bound for V Vargg (eS (f)), where Varg, (els(a) (7)) denotes the empirical variance of ps based S,,, the following holds with probability at least 5 (Maurer & Pontil 2009).
(Dx) S p/.11 ((,)) + ((
Var (4 (T)) (Z) log( M(n)
log
M(n)
VFEF. (43)
Ea.f
h. Since,
(1.1s(Dx)) = Bax [nhb5((8)) = (Dx), the statement of Theorem C.6 follows from En,f
Eq. 43
Note that by combining Eq. 42 with Theorem D.4 translate the bound on R (Dx) to a bound on (D) with additional penalty term that depends on the quality of the teacher labeler h.
Remark C.7. Eq. (42) suggests general approach to select the distribution Dx that generated the training samples 8, In order ensure small generalization gap, it is desirable that the variance term Var, (eS5 (f)) is as small as possible. Note that, the distribution that minimizes this variance takes the form
log (x) x log la (J(x), + logppy (x), X..
(44)
3 Note that the formulation assumes that Dx & Dx i.c., Dx is absolutely continuous w.r.t. Dx. Also, one replace the pdf's with probability functions Dx and Dxx discrete distributions.
21

---

Published as a conference paper at ICLR 2023
This looks like the lagrangian form of Eq. (3). Interestingly, TGT framework with gradient-based sampling (cf. equation 5) focuses on generating samples that maximizes the right hand side RHS of Eq. (44) by first taking sample generated according to Dx and then perturbing it in the latent space to maximize the loss la (f()),)())) Thus, the resulting distribution Dx has pdf that aims to approximate the variance minimizing pdf in Eq. (44)
Here is worth pointing out that, since exact form of pb (-) and PDx (-) is generally not available during the training, it's straightforward to optimize the weighted risk introduced in Eq. (38)
As introduced in Section 3] TGT framework optimizes the empirical risk in Eq. (37) as opposed to
minimizing Eq. (38). In this case, one can obatain variance based bound analogous Eq. 42 that
takes the form:
R (Dx) S R (8m) + (II) W(Dx, Dx),
(45)
where, (II) denotes
o( Vard (z), log( M(n)
log( M(n)
72
72
with eh (G) = (f))), h(t)) and M(n) depending the covering number for the induced function class G (cf. Eq. (5)).. Notably, this bound again incurs penalty of W(Dx Dx).
Remark C.8. Note that Eq. (45) suggests general approach to select the distribution Dx that generates the training samples Sn. In order to ensure small generalization gap, we need to focus on two terms depending on Dx: (1) the variance term Var (Ca.f (ø)); and (2) the divergence term W(Dx, x, Dx). We note that finding a distribution that that jointly minimizes both terms a non-trivial task. That said, in our sampling approach in Eq. ( we control for W(Dx, Dx) by operating the latent space of good quality teacher generative model and minimize variance by finding points with high loss values through gradient ascent, thereby striking a balance between the two objectives.
D TOOLBOX
This section presents necessary definitions and lemmas that utilize to establish theoretical results presented in Section 3 (and restated in Appendix C
Definition D.1 (Wasserstein-1 metric). Let (X. p) be metric space. Given two probability distribu- tions D3 and D3 over X, W distance between D and D3 is defined as follows.
inf [ XXX, dT(
inf W(D 2z) *EIIDD 23)
Ex [d(X ren(D xD)) fxxx
(46)
where IT D2x) denotes the set of all joint distributions over X x X that have Dj and D as their marginals.
Lemma D.2 (Kantorovich- - duality (Villani 2008 ). Ler Lip, (p) denote the set of all functions in the metric p. i.e., for any Lip, (p). 1f(x) - f(x') S p(z, x'), Va,x'.
1 1-Lipschitz f-Lipschiit.uunctiions
(47)
Then,
(DD sup SELiP, (p)
(EX-DY - [(XX))).
(48)
Lemma D.3. Let la RK RR.K - R be loss function employed during the distillation. For given teacher h X + RK and a function class F, we assume the the induced function class
Gha = (x + (a(f()), h(x)) F}
(49)
fuructionn contained in the class of L-Lisschitz functions with respect to a metric p. Then, for any two distributions D) and D2z we have
Rh (D') R S W(D D3). YFEF,
(50)
where W(D) L.DD..'' (cf. Definition D.7
denotes the Wasserstein-1 metric between the two distribution D) and D
22

---

Published as a conference paper at ICLR 2023
Proof. Note that
( R (Dx) Bx [a(f(x),,(() ExinD3 (aa(f("'), (x'))
s sup
Ex [XWDY (x(x)) - ExIDD [g(x')))
se
(
ExNDY g(X)
g(X)
L. sup
E -ExINDy
L
gEg
(i)
S L sup
(Ex=Dy [a(x)) Ex (x(x)))
(ELip,())
(iv)
W(D), Dxx)
(51)
where (1) follow by dividing and multiply by L; (ii) follows as, for any P is is 1-Lipschitz; and (fii) follows from Lemma D.2
Lemma D.4. Let the distillation loss la satisfy Assumption C.I with bounded loss function C:R. RRKK x y + R. Then, given a teacher h + ER and a student model f : RK we have
(2x) - Re,5 (D) & OVVK EDx
[[ - -x(X)ll2),
(52)
where Dryx (Dylx(1)..... Dr....,,II..... (K)) is treated vector in R
Proof. Note that
Rh, (Dx) (D)
EDx (a(I(x), - Re,s(D)
ED (a(((()),)(X) - EED((F()), YI1
ED
Σ h(X), (((()))))) - -Bxx
Σ
Dyix(v) (y)x(v))_(((X)..)))
ые[K]
μE|K]
EDx
Σ [E[
(h(X), Dr(x(U)) - ((f(x),
Ex 1 ((X)l22 [x)
(53)
where (i) follow from the Mauchy-Schward inequality. Now the statement of Lemma D.4 follows from the assumption on the loss L is bounded.
E ADDITIONAL EXPERIMENTS
E. LONG-TAIL IMAGE CLASSIFICATION
Please Table 4 for Places365-L7 result. The relevant discussion is provided in Section 4.1 We also provided an expanded version of Table (from the main text) in Table 5 with additional baselines.
F DETAILS TO REPRODUCE OUR EMPIRICAL RESULTS
Hereby we provide details reproduce our experimental results.
F.I LONG-TAIL IMAGE CLASSIFICATION (SEC. 4.1
Dataset. The full balanced version of 3 datasets (ImageNet Place365 SUN397 Ø
are available
in hen.flowwdatassesttttttpoo http://ww...aataoee/////.oooooor Next to obtain the the
https://www.tet htte:s//wccceerroooo///aaasaasoa
https://wwwt httess//wwttaarrrrrooooo/////sta
23

---

Published as a conference paper at ICLR 2023
Approach
Architecture
Balanced Accuracy
# parameters
FLOPS
LWS KKng al.
2020
ResNet-112
37.6
60 M
11B
I.DA-DRS-RS RSG Wane et al. 2021
ResNet-152
39.3
60 M
11
OLTR (Liu al.
20196
ResNet-152
35,9
60 M
11 B
Plices365.1
DRAGON Loss
Lass (Samuel et 2021
ResNet
38.1
26 M
4.1
Teacher (labeler) model
EfficienNet-b33
42.1
12
1.8
One hot
MubileNeiV3-0.75
26.8
4,01 M
156 M
Distillation
MobileNerV3-0.75
33.0
4.01 M
156 M
TGT (random)
MobileNerV3-00.75
34.7
4.01 M
156 M
TGT (gradient-hased)
MobileNerV3-0075
35.0
4.01 M
156 M
Table 4: Performance of TGT Places-LT (Liu et al., 2019c). The table shows the top-1 accuracy on the corresponding balanced eval sets for TGT and different long-tail baselines from the literature (taken from (Samuel et al. 2021)). We also state the number of model parameters and inference cost (in terms of FLOPs) for all the methods. Note that TGT leads to performance improvements standard distillation. Note that, for Places-LT, TGT does outperform stated baselines for the literature that rely on specialized loss functions and/or training procedures designed from the long-ta -ail setting. One reason for this could be that the BigBiGAN does not generate very informative samples for Places-LT due to distribution mismatch. That said, as discussed in Section 4.1 I one can combine the TGT framework with long-tail specific loss functions as opposed employing the standard cross-entropy loss function as a way to further improve its performance.
Approach
Architecture
Balanced Accuracy
parameters
FLOPPs
LDAM-DRV * Can al. 2019)
ResNet-50
47.8
26 M
4.1
LWS (Kang R020
ResNeXt-50
49.9
25 M
4.2
Logit udjustment Menon al.
2021b
ResNet-50
50.4
26 M
4.1
LDAM-DRS-RSC Wang al. 2021
ResNeXt-50
51.8
25 M
4.2
InageNNet
DRAGON Bal'Loss Samuel al
2021
ResNet-10
46.5
M
819 M
DRAGON Bal'I Samuel et al.
202
ResNet-50
53.5
26 M
4.1 B
Teacher flabeler) model
EfficienNNet-b3
79.2
M
1.8
One hot
MobrieNeeV330075
35.5
4,01 M
156 M
Distillation
MohhieNeeVV.0075
47.2
4.01 M
156 M
TGT (random)
MobileNeeV3-0.75
53.2
4,01 M
156 M
TGT (gradient-busedd
MobileNetV3-0075
53.3
4.01 M
156 M
LDAM-DRS-RSC Waan al. 2021
ResNeXt-50
29.8
25 M
4.2
CAD-VAE Schönfeld al 2019
ResNet-101
32.8
42 M
7.6
SUN397-LT
LWS (Kang et al.
2020)
ResNeXt-50
33.9
25 M
4.2 B
DR DRAAGNN+ Bal" Loss Samuel al. 2021
ResNet-101
36.1
42 M
7.6
Teacher flabeler) model
EflicieniNet-b3
65.3
12
1.8 B
One hot
MobileNeeV3.0.75
39.3
4,01 M
156 M
Distillation
MobileNetV3-0.75
42.2
4.01 M
156 M
TGT (random)
MobileNeeV3-00.5
44.3
4.01 M
156 M
TGT (gradient-based)
MobileNerV3-00.5
44.7
4.01 M
156 M
Table 5: Performance of TGT and various baselines from the literature on ImageNet-LTT and SUN-LT.
Note that this table expands Table [ (in the main text) as it includes additional baselines. Rows with *
denote results taken from Menon et al. (021b) and the rest were taken from Samuel al. (202I). We report top-1 accuracy on balanced eval sets. We also state the number of model parameters and inference cost (in terms of FLOPs) for all the methods. Note that TGT leads performance improvements over standard distillation on both datasets, particularly for ImageNet-LT where the teacher generator models the task distribution well. TGT also often outperforms stated baselines that rely on much larger and expensive models.
long-tail version of the datasets, we downloaded image ids from repository of "Large-Scale Long- Tailed Recognition in an Open World (Liu et al. 2019b)" according which we subsampled the full balanced dataset.
Teacher fine-tuning. For teacher labeler, we follow "Sharpness Aware Minimization' Foret et al. 2020) codebase (available at http::::://// http:://../ htt///githbbb. httt:s::://ithuut on the long-tail datasets. We start with pretrained EfficientNet-B3 model checkpoint available from
com/ poooole--oo re research. sam) to fine-tune
17 877xulys
24

---

Published as a conference paper at ICLR 2023
official repository and used default parameters from the codebase. We fine-tuned all 3 datasets (ImageNet-LT, SUN397-LT, Place365-LT) for epochs.
We directly used teacher generator as BigBiGAN ResNet-50 checkpoint from the official repository https:/ httt:://gitbbb. com/ (We did not fine-tune it.)
bigbigan
Student training. We start from randomly initialized MobileNetV3-00.7 model. We employed SGD optimizer with cosine schedule (peak learning rate of 0.4 and decay down to 0). We also did linear warm-up (from 0 to peak learning rate of 0.4) for first epochs. The input image size are unfortunately different between EfficientNet-B3 model, BigBiGAN SeeiGGAN--eeNee5,,, and MobileNetV3-0.75 models. From original images in dataset, we use Tensorflow's 's bicubic resizing to obtain appropriate size image for each mode. We did grid search over the perturbation parameters and (c.f. Eq. and Eq. (5). All
and grid are listed in table below:
Hyper-paran
ImageNet-LT
Place365-LT
Sun397-LT
Num epochs
90
30
30
Optimizer
SGD
Schedule
Cosine
Warm-up epochs
5
Peak learning rate
0.4
Batch size
256
Teacher labeler image size
300 300 x
Teacher generator image size
256 x 256 x 3
Student image size
224 224 x
Perturbation noise (o)
[0, 0.001, 0.01, 0.1]
Gradient exploration
-Seep size (17)
(0, 0.001, 0.01, 0.1)
Num steps
2
Table 6: Hyper-parameters for long-tail tail image classification
F.2 TGT IN LOW-DATA REGIME (SEC.
Dataset. We used ImageNet dataset from tensflow-datase repository https://wwa Lensorflow wtt.l//ww...///aaattsss///////// We used in-built sub-sampling functionality available in ten- to simulate the low-data
(ttteess//ww..eeerooooooooooooe. regime.
Teacher model. For teacher labeler, we directly used trained EfficientNet-B3. model checkpoint available from "Sharpness Aware Minimization" repository 10
For teacher generator, we directly used trained BigBiGAN checkpoint from the official repository https: //tpthghttt///t/// com/ deepmi nd/ (We did not fine-tune either of the mod-
els.)
Student training, We start from randomly initialized MobilcNetV3-0075 model. We employed SGD optimizer with cosine schedule (peak learning rate of 0.4 and decay down to 0). We also did linear warm-up (from ( to peak learning rate of 0.4) for first epochs. The input image size are unfortunately different between EfficientNet-B: model, BeBBiGAN-RRsNeete Tensorflow's 's bicubic resizing to obtain appropriate size
and MobileNetV3--0.7 models.
From original images in dataset, image for each mode. Following standard practice in literature He et al. (2016); Jia al. (2018), we train one-hot and standard distillation student models for 90 epochs (= 450k steps). We use 4x less steps for TGT than the simple distillation baseline, which amounts to 450k/4 = 112k steps.
nts/
noi ottprp/eee////eaaaaoeeeeee//aaar
/ts// whwckpo.ehhcccccee nts/
noi hog.rreeeereffffcaeeeeeeeeeeeeen
25

---

Published as a conference paper at ICLR 2023
F.3
TEXT CLASSIFICATION (SEC.
4.3
Dataset, We conduct text classification experiments on following datasets:
A downloaded from http::///oo../////c22
httpss//w..eaaatooooooo///at221. cata log/imdbrrevieew
IMDB from (ensorflow-datasee
MNLI from http:0/00..aa..aaoe/e//////////.
cata Tog/mo lttlmlltimmniiiiimmm [ii_nniinni
Yelp-5 downloaded from http::::t:og..////pgpg/oo
Optimizer. For all training, we employed ADAM optimizer with linear decay schedule (peak learning rate of 3e-5 and decay to 0). We also did linear warm-up start. We used batch size of 128.
Teacher fine-tuning. For teacher labeler, we started from RoBERTa-Base (Liu et al. 2019ap pretrained checkpoint from official FAIRSEQ repository https: / / //ggttttuuuu/uu com/ Bacebookreseark fai rseq We fine-tuned using default parameters, other than number of steps which same as those listed in Table
For teacher generator, we directly use pre-trained BART-Base (Lewis et al. 2020) checkpoint 12 from official FAIRSEQ repository https://githubl www http://ccbbbboreeerrrc//eaaaaaa. (We did not fine-tune it.)
We start from DistilIBETT pretrained checkpoint downloaded from HuggingFace repository 13 We perturb by adding Gaussian noise of o² variance in between encoder-decoder as well as masking out p fraction of input. Then we generate new examples by running a greedy decoding of BART teacher generator for sequence length of 512. For dual input classification task, like in MNLI, we generate the two inputs independently. We did a grid search over the perturbation parameters and masking fraction p. All hyper-parameters and grid are listed in table below:
Student training.
Hyper-param
Amazon-5
IMDB
MNLI
Yelp-5
2.5k
3M
2.5k
650k
Num steps
5000
75000
20000
75000
5000
75000
Warm-up steps
1000
2000
500
2000
1000
2000
Optimizer
Adam
Schedule
Linear
Peak learning rate
3e-5
Batch size
128
Max Sequence length
512
Perturbation noise (a)
(0. 0.01, 0.1]
Masking fraction (p)
[0, 0.1, 0.2]
Table 7: Hyper paraeerprraeeters for student training of text classification
F.4 TEXT RETRIEVAL (SEC. 4.4
Dataset. From official "Dense Passage Retrieval" repository at https:://itthu.com com/ facebooirreseachh/eer we download passage corpus 14
Further, from the same reposi-
tory, we download a pre-processed version of natural questions open dataset (Lee et al. 2019) which has been aligned to passage corpus Finally, we download pre-processed version of
json httpp://h..aaauuuul....///////sr bi biencooerr-ng-dev.jaa..
26

---

Published as a conference paper at ICLR 2023
Probably Asked Questions (PAQ) dataset (Lewis al. 2021) dataset from official repository of "Domain-matched Pre-training Tasks for Dense Retrieval" available at athttps://qithie
com/ Facebookresearcch http://tweeeeareeppppscesasaaaaa which has been aligned to the same passage corpus
Optimizer. For all text retrieval model training, we employed ADAM optimizer with linear decay schedule (peak learning rate of le-5 and decay to le-7). We also did linear warm-up (from 0 to peak learning rate of le-5) for 1K steps. We used batch size of 128.
Teacher fine-tuning, For teacher labeler dual encoder (a question encoder and passage encoder), we utilized RoBERTa-Base (Liu et al. 2019a) pretrained checkpoint 17 from official FAIRSEQ repository htwpt:ttttttoo. //g1 thub. com facebook resea rch/ fai rseq We then conducted first round of fine-training for 300k iterations with passage-aligned PAQ dataset. We used same configu- ration as Oğuz al. (20IT) except Oguz et al. trained with PAQ longer. After the pretraining, the teacher is fine-tuned on NQ-open (Kwiatkowski et al. 2019) downloaded with 40K steps. Similar Karpukhin et al. (2020); Oguz et al. (2021), the teacher is trained with within-bate negatives and the softmax-based cross-entropy loss.
For teacher generator, we directly use pre-trained BART-Base (Lewis al. 2020) checkpoint from official FAIRSEQ repository https://githib: com facebookkekkoooo resaa rch cerrrrc//aaeeeee rseg (We did not fine-tune it.)
This same teacher labeler and generator is used for all student training except for the direct training (one-hot).
Student training. We start from DistilIBERT pretrained checkpoint downloaded from Hugging Facee repository 19 All students are trained with 40K steps. The teacher labeler will label all-pair within the batch and will label additional 2 passages per each question pasioon pair for the uniform negative sampling baseline and TGT. We employed a off-the-shelf BART-base model as our generator (Lewis et al. 2020) and isotropic perturbation was added by random Gaussian noise of scale 0.1 combined with = 0.2 for masking the original passage.
G QUALITATIVE EXAMPLES OF GENERATED EXAMPLES
G.1 IMAGE CLASSIFICATION
We show some representative examples of generated images using TGT-random as well TGT- gradient based from the experiment on ImageNet classification in Table g
G.2 TEXT CLASSIFICATION
We show some representative examples of generated text using TGT from the experiment on MNLI classification in Table
json1.
zip
27

---

Published as a conference paper at ICLR 2023
Input
TGT-Random Example
TGT-Gradient based Example
Data label: lion
Teacher label: brown bear
Teacher label: chow
Data label: cheeseburger
Teacher label: potpie
Teacher label: cheeseburger
Data label: digital clock
Teacher label: tape player
Teacher label: grocery store
Data label: wall clock
Teacher label: shield
Teacher label: gong
Table 8: Image examples
28

---

Published as a conference paper at ICLR 2023
Input
TGT Example
Data lahel: Cont Contradicos ct.
Teacher label: Neutral
bought with the royalties she earned from her first book, The Tales of Peter Rabbit. [SE?] The house was bought with the money she inherited from her grandfather.
The house
The book published in the United States in 1987 with the royalties she received from her first book, The Tales of Feter Rabbit. [SEP] The house was bought with the money she inherited from her grandfather.
Data label: Entail
Teacher label: Entail
longer bargain in Spain, though very good quality products may still be priced lower than at home. [SEP] Leather goods are still very cheap in Spain.
Leather goods
Leather and leather goods a bargain Spain, though very good quality products may still be priced lower than at home and abroad. [SEP] Leather goods are still very cheap at Spain.
longer
Data label: Entail
Teacher label: Neutral
Then I got up softly could, and felt in the dark along the left-hand wall. [SEP] The wall was wet.
Then got up softly could, and walked the way I felt in the dark along the left [SEP] The wall was wet.
Data label: Entails
Teacher label: Entail
But then this very particular island is hardly in danger of being invaded except, of course, by tourism. [SEP] This island is 1east tikely to be invaded by tourism.
But then this very particular island is not in danger of being invaded except, of course, by tourism. [SEP] The island is likely to be invaded by tourism
Data label: Contradicts
Teacher label: Neutral
All you need do is just wander off the beaten path, beyond the bustling tourist zone. [SEP] There is no point going off the beaten path, there is nothing there.
All you need the beaten path, and youll in the bustling tourist zone of the city, [SEF] There is no point going off the beaten path, there is nothing there.
is just wander off
Data label: Entails
Teacher label: Neutral
The silt of the River Maeander has also stranded the once-mighty city of Miletus. [SEP] The River Macander has beon dopositing silt near Miletus for nearly two mi millennia.
The silt of the River Mae has also stranded the ance-mighty city of Miletus. [SEP]
River Maeander has been depositing silt near Miletus for more than two decades.
Data label: Entails
Teacher label: Entails
It was harely the most enlightened of times, not with the conflict in Indochina rapidly becoming Americaš costliest and mest divisive war. [SEP] The war in Indochina has cost America 100 billion dollars so far.
It was hardly the most enlightened of times, not with the war in Indochina becoming Americas costliest and most divisive war. [SEP] The war in Indochina has cost America 100 billion dollars so far.
Table 9: Text examples
29


Reviews:
[{"review_id": "xFhgzKuFJ56", "reviewer": "Reviewer_Qnjd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a general teacher-guided training (TGT) framework for efficient model distillation. TGT explicitly leverages the low dimensional representations extracted by the pretrained teacher generative models, which is then theoretically shown to be able to improve the generalization of the student model. TGT does not need to go through a large volume of data, thus and can better work in limited data regime or long-tail settings. ", "summary_of_the_review": "None", "strength_and_weaknesses": "Strength:\n\n1. I think this paper will be valuable to the community. For the large pretrained models, even distilling to a lighter proxy can be unaffordable. Cleverly utilizing the teacher model and distilling the student efficiently is a very important problem. I like the theoretical analysis showing the benefit of utilizing TGT on generalization error bounds. These are good contributions.\n\n2. The paper appropriately cites related works and discusses their relationship to the teacher-guided training method. \n\n3. The paper is well-motivated and clearly written. Experiments are properly conducted to validate the effectiveness. \n\nWeakness:\n\n1. A minor disadvantage of TGT is that it relies upon an encoder-decoder style generative model as a teacher, which limits its practicability as most of the models trained in the industry are deterministic and do not have an encoder-decoder structure.  \n\n2. I am not sure if all three terms in Equation (2) are utilized in the experiments. If so, will the first supervised learning term dominate and nullify the better generalization applying TGT?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bVr00aMs3D", "reviewer": "Reviewer_hz78", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies an important problem, i.e., given a pre-trained large teacher model, how to transfer its knowledge to a compact low-compute smaller model. The proposed method is developed on top of the standard knowledge distillation approach. The authors assume that there is an available encoder and decoder, such that any training sample can be encoded into the latent space. Then one can find the maximally disagree point in the data manifold by exploring the latent space of the encoder, and train the student network by minimizing this disagreement. Extensive empirical results on both image and text data are provided.", "summary_of_the_review": "Personally, I think that this paper is around the borderline. Currently, I'm slightly leaning reject. My rating may change after seeing the comments from other reviewers.", "strength_and_weaknesses": "Strength:\n- The problem is important. The proposed TGT framework is well-motivated.\n- The experiments are extensive.\n- A theoretical analysis is provided.\n\nWeaknesses:\n- It may be difficult to say that the proposed method is 'novel'. TGT is based on a standard knowledge distillation framework. The idea of 'minimizing the disagreement' is not new (e.g., [*1]). In general, I'm not surprised that this idea can improve the performance.\n- From Table 1, one can see that TGT (gradient-based) only marginally outperforms the random baseline, i.e., TGT (random). In addition, the random baseline is not provided in other experiments. Therefore, I'm not fully convinced by the effectiveness of the proposed 'Gradient-based exploration' mechanism, which is an important contribution of this paper.\n- In Table 1, the teacher model is pre-trained on a much larger dataset (i.e., JFT-300M) than the baselines. The comparison may be unfair. In addition, JFT-300M is not publicly available, which may hurt the reproducibility of this paper.\n\n## Post-rebuttal\nThe responses from the authors have addressed most of my concerns (e.g., novelty and reproducibility). I have updated my score. However, I still think that adding more results of TGT (random) is important.\n\n\n[*1] Miyato, T., Maeda, S. I., Koyama, M., & Ishii, S. (2018). Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8), 1979-1993. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JVSmRHIWcd", "reviewer": "Reviewer_u9os", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel technique for producing compact versions of large models using a new approach to distillation named Teacher Guided Training. The authors develop a sound mathematical framework based on maximizing the match between the teacher and student model distributions.  The authors establish some useful performance bounds. The main insight of this work is in operating in the latent space which allows handling both discrete and continuous domains seamlessly. The technique also significantly improves upon the state of the art on a variety of test domains both in the reduction of parameters and the accuracy. ", "summary_of_the_review": "The paper nicely motivates the main techniques proposed. The literature survey is thorough and insightful. The proposed technique is backed up by a sound mathematical framework that is systematically established and well explained. The technique is shown to work well on a variety of domains and yields noteworthy improvement over the state of the art. It also advances the state of the art in the architecture and mathematics of its approach to the problem. \n", "strength_and_weaknesses": "Strengths:\n\n1. Innovative teacher-student distillation approach that is well motivated and explained.\n2. Sound and insightful mathematical framework that enables application to both discrete and continuous domains.\n3. Consistent and significant improvement over the state of the art in both model size reduction and accuracy. \n\nWeaknesses\n1. No significant weaknesses.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "CV7c6EVLHW", "reviewer": "Reviewer_PrCv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets solving knowledge transfer problems. Specifically, the authors distill large models to compact models for efficient machine learning. Teacher-guided training (TGT) framework is proposed for training high-quality compact models which learn knowledge from pretrained generative models. Experiments show the effectiveness of TGT.", "summary_of_the_review": "Strengths:\n+ This paper is well-written. It is enjoyable to read.\n+ The proposed framework is simple and effective. The overall idea is neat.\n+ Experiments are comprehensive and show the effectiveness of TGT.\n\nWeaknesses:\n+ The contributions are a little limited. Knowledge distillation is an existing approach.\n+ More insightful analyses should be provided. How the knowledge is transferred in TGT?\n+ Is the conclusion in 4.2 still hold true in NLP dataset?", "strength_and_weaknesses": "Strengths:\n+ This paper is well-written. It is enjoyable to read.\n+ The proposed framework is simple and effective. The overall idea is neat.\n+ Experiments are comprehensive and show the effectiveness of TGT.\n\nWeaknesses:\n+ The contributions are a little limited. Knowledge distillation is an existing approach.\n+ More insightful analyses should be provided. How the knowledge is transferred in TGT?\n+ Is the conclusion in 4.2 still hold true in NLP dataset?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster