Toward Student-oriented Teacher Network Training for Knowledge Distillation
Authors: Chengyu Dong
Liyuan Liu
Jingbo Shang
Venue: ICLR 2024 poster (2024)
Keywords: Knowledge distillation
Teacher-student training
Empirical risk minimization
Primary area: learning theory

Abstract: How to conduct teacher training for knowledge distillation is still an open problem. It has been widely observed that a best-performing teacher does not necessarily yield the best-performing student, suggesting a fundamental discrepancy between the current teacher training practice and the ideal teacher training strategy. To fill this gap, we explore the feasibility of training a teacher that is oriented toward student performance with empirical risk minimization (ERM). Our analyses are inspired by the recent findings that the effectiveness of knowledge distillation hinges on the teacher’s capability to approximate the true label distribution of training inputs. We theoretically establish that ERM minimizer can approximate the true label distribution of training data as long as the feature extractor of the learner network is Lipschitz continuous and is robust to feature transformations. In light of our theory, we propose a teacher training method SoTeacher which incorporates Lipschitz regularization and consistency regularization into ERM. Experiments on benchmark datasets using various knowledge distillation algorithms and teacher-student pairs confirm that SoTeacher can improve student accuracy consistently.

Published as a conference paper at ICLR 2024
TOWARD STUDENT-ORIENTED TEACHER NETWORK TRAINING FOR KNOWLEDGE DISTILLATION
Chengyu Dong' Liyuan Liu? Jingbo Shang'
University of California, San Diego "iicrosoft Research (cdeng, jshang luesd. Lucliuenderosott.ccm
ABSTRACT
How to conduct teacher training for knowledge distillation is still an open problem. It has been widely observed that best-performing teacher does not necessarily yield the perffornn student, suggesting a fundamental discrepancy between the current teacher training practice and the ideal teacher training strategy. To fill this gap, we explore the feasibility of training a teacher that is oriented toward student performance with empirical risk minimization (ERM). Our analyses are inspired by the recent findings that the effectiveness of knowledge distillation hinges the teacher's capability to approximate the true label distribution of training inputs. We theoretically establish that ERM minimizer can approximate the true label distribution of training data long as the feature extractor of the learner network is Lipschitz continuous and is robust to feature transformations. In light of our theory, we propose a teacher training method SoTeacher which incorporates Lipschitz regularization and consistency regularization into ERM. Experiments on benchmark datasets using various knowledge distillation algorithms and teacher- student pairs confirm that SoTeacher can improve student accuracy consistently.
INTRODUCTION
Knowledge distillation aims train a small yet effective student neural network following the guidance of a large teacher neural network (Hinton et al. 2013). It dates back to the pioneering idea
of model compression (Buciluă et al. 2006) and has a wide spectrum of real-world applications, such
as recommender systems Tang & Wang 2018) Zhang et al.) (2020), question answering systems (Yang
ct al. 2020. Wang et al.. 2020) and machine translation (Liu et al. 2020).
Despite the prosperous interests in knowledge distillation, one of its crucial components, teacher training, largely neglected. The existing practice of teacher training is often directly targeted at maximizing the performance of the teacher, which does not necessarily transfer to the performance of the student. Empirical evidence shows that a teacher trained toward convergence will yield an inferior student (Cho & Hariharan, 2019) and regularization methods benefitting the teacher may contradictorily degrade student performance (Müller et al. 2019). As also shown in Figure T the teacher trained toward convergence will consistently reduce the performance of the student after certain point. This suggests a fundamental discrepancy between the common practice in teacher training and the ideal learning objective of the teacher that orients toward student performance.
In this work, we explore both the theoretical feasibility and practical methodology of training the teacher toward student performance. Our analyses are built upon the recent understanding of knowledge distillation from a statistical perspective. In specific, MMeon et al. (202 1) show that the soft prediction provided by the teacher is essentially an approximation
the true label distribution, and true label distribution as supervision for the student improves the generalization bound compared to one-hot labels. Dao et al. (202T) show that the accuracy of the student is directly bounded by the distance between teacher's 's prediction and the true label distribution through the Rademacher analysis.
Based on the above understanding, a teacher benefitting the student should be able to learn the true label distribution of the distillation data Since practically the distillation data is often reused from
For simplicity, we refer to the training data of the student model in knowledge distillation as the distillation data (Stanton al. 2021)

---

Published as a conference paper at ICLR 2024
75
Final Test Acc (%)
Test Acc (%)
Training Loss
74
65
Standard
Standard (Train Loss)
60
SeTacheer (Tesi: Acc)
72
55
SaTcacher (Train Loss)
71.
45
70-
SoTeacher (Test Acc
180
220
Epoch
Teacher Checkpoint Epoch
(a) Teacher performance with different num- bers of teacher training steps.
(b) Student performance when distilled from different teacher checkpoints.
Figure 1: We train teacher models on CIFAR- 100, saving checkpoint every 10 epochs, and then use this checkpoint to train a student model through knowledge distillation. With our method, the teacher trained with a focus on improving student performance, leading to better student performance even if the teacher's own performance is not high.
the teacher's training data, the teacher will have to learn the true label distribution of its own training data. This might appear to be infeasible using standard empirical risk minimization, as the teacher network often has enough capacity fit all one-hot training labels, in which case, distilling from teacher predictions should largely degrade to directly training with one-hot labels. Existing works tend evade this dilemma by distilling from teacher predictions only on data that is not used in teacher training (Menon et al., 2021: Dao et al. 2021).
Instead, we directly prove the feasibility of training the teacher learn the true label distribution of its training data. We show that the standard empirical risk minimizer can approach the true label distribution of training data under mixed-feature data distribution, as long feature extractor of the learner network is Lipschitz continuous and is robust to feature transformations.
In light of our theory, we show that explicitly imposing the Lipschitz and consistency constraint in teacher training facilitate the learning of the true label distribution and thus improve the student performance. We conduct extensive experiments on two benchmark datasets using various knowleds wedee distillation algorithms and different muacher-student architecture pairs. The results confirm that our method can improve student performance consistently and significantly.
We believe work is among the first attempts to explore the theory and practice of training student-oriented teacher in knowledge distillation. To summarize, our main contributions are as follows.
We show that is theoretically feasible to train the teacher to learn the true label distribution of the distillation data even with data reuse, explaining the effectiveness of the current knowledge distillation practice.
We show that by adding Lipschitz and consistency regularization during teacher training, it can better learn the true label distribution and improve knowledge consistently.
PRELIMINARIES
We study knowledge distillation in the context of multi-class elassification. Specifically, we are given a set of training samples D = {(x(1) (()) )E(NN, where [N] == 21,2. , .N.. is drawn from probability distribution px,y that defined jointly input space X' and label space y =K]. For convenience, we denote 1(y) R&K the one-hot encoding of label y.
Learning Objective of Teacher Practice. In common practice, the teacher network f is trained minimize the empirical risk given loss function namely
min (E {((()).))
where E D is the empirical expectation.
Ideal Learning Objective of Teacher. Recent advances understanding knowledge distillation
suggest teacher should approximate the true label distribution of the distillation data, which is often
2

---

Published as a conference paper at ICLR 2024
reused from the teacher's training data. From this perspective, ideally the teacher should learn the true label distribution of its training data, namely
min B(DD)DD II[(x) (x))
I
Here, p²(x) = py|x (-1ar) denotes the true label distribution of an imput æ, namely the (unknown) category distribution that its label is sampled from, which is not necessarily one-hot. And || an arbitrary p-norm.
can be
Our Research Questions. One can find that there is a fundamental discrepancy between the learning objective of the teacher in the common practice and the ideal learning objective of the teacher. In particular, minimization of Eq. I would lead to f(z) = 1(y) for any input x, which significantly deviates from p" (x:) and thus challenges the effectiveness of knowledge distillation. Therefore, in this work, we explore the following two questions.
(i) Can a teacher network learn the true label distribution of the training data with the standard teacher training practice?
(ii How to train a teacher to better learn the true label distribution and improve student performance?
We will present our findings of these two questions in Sec. and Sec. respectively.
3 THEORETICAL FEASIBILITY TO LEARN TRUE LABEL DISTRIBUTION OF TRAINING DATA
We now explore the theoretical feasibility of training teacher network that learns the true label distribution of the training data under the empirical risk minimization framework. Unless specifically stated otherwise, by "data" we refer to the data used to train the teacher, and by "network" refer to the teacher network. Note that throughout the discussion here, our major focus is the existence of a proper minimizer, instead of the details of the optimization process.
3.1 NOTATIONS AND PROBLEM SETUP
Notation. Here, we introduce our notations in addition to the ones mentioned in Section 2 We will use calligraphic typefaces to denote sets, egg. the dataset D. We use |D] to denote the size the set. We use o to denote function composition. We use Õ(1) to denote polynomial terms 17.
Data Distribution.
We consider data distribution which we refer to the mixed-feuture distribution. Our distribution can be viewed as simplified version the "multi-view distribution" introduced in AolAlll--Zii & Li 22020). Our distribution can also be viewed variation of Latent Dirichlet Allocation (LDA) (Blei et al. 2001), a generative data distribution widely used to model text data.
Definition 3.1 (Mixed-feature distribution). We first generate the imput x following LDA. We define a feature vocabulary Z that denotes the names of all possible features in the data imputs. For example, in image classification we can have 2 = {'eye', 'tail", 'wheel', ...). Now for each data example i,
1. Sample M feature names from 2, namely Zm pz, where pz discrete distribution defined over Z.
2. For each feature name we sample its inpul representation x. E ER,, namely Ig ~ Px((12)) where (-12) is continuous distribution with finite variance, namely Var [ar[X|z] 22] V2. The finite variance here means that the representation of the same feature sampled in different inputs should be similar.
For each feature name z, we trunsform its inpul representation by a function 7 Rbb - RR,, where is sampled from set of possible transformation, functions T, namely v 7(22). For example, in image classification 7f can be rotation, mirroring. or resizing.
4. We concatenate the transformed representations of these features to obtain the input, namely [Zz, ''z.''.. , TI where BbxM This views each data input concatenation of M patches and each patch is b-dimensional vector, which follows the assumption in Aallllee-Zer & Li (2020)
3

---

Published as a conference paper at ICLR 2024
Next, we generate the label y. We assume each feature name defines label distribution py (-12). The label distribution of an input 2 the geometric average of the label distributions of the feature names, namely y (-(x))) = (IIm Py (112,m))
Note that in our data distribution, the assumption that each patch is vector is solely for the simplicity of illustration and can be trivially generalized, as our theory is not dependent on it. The specific shape of each patch can be arbitrary, for example, can be 3-rank tensor (height, width, channel) in image classification.
One difference between our data distribution and "multi-views distribution" is that different label sets y for the same input data under our data distribution. This is more realistic as models datasets that have baarse-gaained/nne
define
label sets respectively but the same input data.
Network Architecture.
We consider a multi-layer neural network that produces probabilistic outputs, namely f: R [0, 1jK The network consists of feature extractor and classification head, namely f = fo fE, defined as follows respectively.
SE: RbxM + R Mxd denotes the feature extractor, whose architecture can be arbitrary as long it processes each patch independently. With little abuse of notation, will also write hm == =f((=m)
fc :R: RK denotes the probabilistic classification head, which consists of 1x1 convolu- tional layer and a modified Softmax layer, namely (c) Softmax Softmax(wclh (wch), where wc R R1xkxd The 1xl convolutional layer is similar to the assumption in (Allen-Zhu & Li 2020), while the modified Softmax is slightly different from the standard Softmax in terms of the denominator,
exp(I/M E... hm) namely Softmax (h) (T.m En xxp(fmm.)))4/Ar
where h w..h
Learning. Given a dataset D, we train the network under the empirical risk minimization framework as indicated by Eq. ( We consider cross- cross-entropy loss {(f(x),) v) -1() log fx).
3.2 A HYPOTHETICAL CASE: INVARIANT FEATURE EXTRACTOR
For starters, we investigate a hypothetical case where the feature extractor is invariant, which means that it can always produce the same feature map given the same feature patch, regardless of which input this feature patch is located at or which transformation is applied to this feature patch. Given such an assumption, showing the learning of true label distribution of the training data can greatly simplified. We will thus briefly walk through the steps towards this goal to shed some intuitions.
Definition 3.2 (Invariant feature extractor). We call fE an invariant feature extractor, for any two inputs and j. for any transformations 7 and y' (n((((())) (1) Zm Zmn, namely the feature names of the patches m and mn' match.
fEl'' (x())) as long as
We first show that given an invariant feature extractor, the minimizer of the empirical risk has the property that its probabilistic prediction of each feature converges to the sample mean of the labels whose corresponding inputs contain this feature.
Lemma 3.3 (Convergence of the probabilistic predictions of features). Let Os = N E (iIzEZ00) 1)) where z denote the set of feature names the i-th input, and thus {ilz 2(0) denotes the set of imputs that contain feature Let f be minimizer of the empirical risk (Eq. (7)) and assume f' is an invariant feature extractor. Let p (I22) == Softmx(wee (x:)) be the probabilistic prediction of feature z. We have
pj- (x=(x=)= D2-
The intuition here is that since the feature extractor is invariant and the number of possible features limited, the feature maps fed to the classification head would be a concatenation of M vectors selected from fixed set of |z| candidate vectors, where each vector corresponds one feature in the vocabulary. Therefore, the empirical risk can be regrouped - N Σ, (g) log f (x((()
+ Σ.Ez d: logp/ (x.). Then by Gibbs' inequality, the empirical risk can be minimized only when the probabilistic prediction of each feature ps- (I:2) is equal to Øz-
4

---

Published as a conference paper at ICLR 2024
We now proceed to show that the above sample mean of multiple labels ys will converge to the average distribution of these labels, even though they are non- non-identically distributed. Since each label is a categorical random variable, this directly follows the property of multinomial distributions (Lin
et al. 2022).
Lemma 3.4 (Convergence of the sample mean of labels). Let p(1=)= = x Σ [ile pr (+((0))
we have with probability at least 1 - ó,
ly: p(-12)1 S S (OVKN--M22-881
It is also feasible to achieve the above lemma by applying Lindeberg's central limit theorem, with weak assumption that the variance of each label is finite.
Next, we show that the average label distribution p(-1z) approximates the true label distribution of the corresponding feature: Lemma 3.5 (Approximation of the true label distribution of each feature).
(1x(-12) py (+1=)] s Ô (MIZI-1).
The intuition is that since the average distributions p(-12) here all contributed by the true label distribution of the inputs that contain feature z, it will be dominated by the label distribution of feature 2, given some minor assumption on the sampling process of features when generating each input.
one can see that the probabilistic prediction of each feature given by the empirical risk minimizer will approximate the true label distribution of that feature. Subsequently, we can show that the probabilistic prediction of each input approximates the true label distribution of that input, which leads to the following main theorem.
Finally, combining Lemmas 3.3
3.4 3.5
Theorem 3.6 (Approximation error under hypothetical case). Given the setup introduced in Section 3.1 let f be a minimizer of the empirical risk (Eq. ( and assume f' is an invariant feature extractor. Then for any imput E DD with probability at least 5,
(=(x)=p'<))) S (VKN-MZ--16) (0
+ 6 (M
(6)
3.3 REALISTIC CASE
We now show that in a realistic case where the feature extractor is not exactly invariant, is still possible approximate the true label distribution, as long the feature extractor is robust to the variation of features across inputs and also robust to feature transformations.
Definition 3.7 (Lipschitz (Lipschitz-continuuuu continuous feature extractor). We call fE a Lx (6) l|. long (4) as 2m = 2m', namely the feature names of the patches m in and patch m'' in j match.
feature extractor, if for any two inputs i and j. [| f
< Lx|lum =
Definition 3.8 (Transformatton-robuu feature extractor). We call f At-trrnfformation--rbuss feature extractor, if for any patch m in any input i, and for any transformation y, Ilfe ( Lp-
continuuus and transformation-robrr feature extractor, the probabilistic predictions of features will still converge the sample mean of the labels where the corresponding inputs contain this feature, up to some constant error. This requires the assumption that the input representation of the same feature is similar across different inputs, which is intuitive and covered by Definition B.1 All other lemmas introduced in the hypothetical case still hold trivially as they are not dependent on the network. Therefore, we can have the following result.
Similar to Lemma 3.3 given
Theorem 3.9 (Approximation error under a realistic case). Given the setup introduced in Section 3.1 let f be minimizer of the empirical risk (Eq. (7) and assume ft. is and L trw.rrrfoomaaionn-eeem robust feature extractor. Let max; Vz. Then for any input x D, with probability at least - 8.
|| (x) - --"(() s (VKN-IME--55) + -MI :
( Ö(Lr). (7)
5

---

Published as a conference paper at ICLR 2024
4 SoTeacher
Based on our theoretical findings, we investigate practical techniques for training the teacher model to more accurately approximate the true label distribution of the training data.
Lipschitz regularization. Theorem 3.9 suggests that it is necessary to enforce the feature extractor to be Lipschitz continuous. This may also be achieved by current teacher training practice, as neural networks are often implicitly Lipschitz bounded (Bartlett et al. 2017). Nevertheless, we observe that explicit Lipschitz regularization (LR) can still help in multiple experiments (See Sect. (5) Therefore, we propose to incorporate a global I. Lipschitz constraint into teacher's 's training. For the implementation details, follow the existing practice of Lipschitz regularization (Yoshida & Miyato 2017 Miyato
itt al. 2018) and defer them to Appendix C
Consistency regularization.
Theorem 3.9 also shows that learn the true label distribution, is
necessary to ensure the feature extractor is robust to feature transformations. This is aligned with the standard practice which employs data augmentation as regularization. However, when data is scarce, it is better to explicitly enforce the model to be robust to transformations. This is known as
consistency regularization (CR) (Laine & Aila 2017; Xie al. 2020; Berthelot et al. 2019 Sohn
et al. 2020 that is widely used in semi-supervised learning.
Considering the training efficiency of consistency regularization, we utilize temporal ensem- bling (Laine & Aila 2017), which penalizes the difference between the current prediction and the aggregation of previous predictions for each training input. In this way, the consistency under data augmentation is implicitly regularized since the augmentation is randomly sampled in each epoch. And it is also efficient as no extra model evaluation is required for an input.
To scale the consistency loss, loss weight is often adjusted based Gaussian ramp-up curve in previous works (Laine & Aila 2017). However, the specific parameterization of such a Gaussian curve varies greatly across different implementations, where more than one additional hyperparameters have to be set up and tuned heuristically. Here to avoid tedious hyperparameter tuning we simply linearly interpolate the weight from 0 to its maximum value, namely (tR(t) = A the total number of epochs.
max where T is
Summary. To recap, our teacher training method introduces two additional regularization terms. The loss function can thus be defined as f= fStand. + fR + &CR where fStand is the standard I
empirical risk defined in E and ÄLR is the weight for Lipschitz regularization. Our method is simple to implement and incurs only minimal computation overhead (see Section 5.2)
5 EXPERIMENTS
In this section, we evaluate the effectiveness of our teacher training method in knowledge distillation. We focus on compressing a large network smaller one where the student is trained on the same data set as the teacher.
5.1 EXPERIMENT SETUP
We denote our method as student-oriented teacher training (SoTeacher), since it aims to learn the true label distribution to improve student performance, rather than to maximize teacher perfor- mance. We compare our method with the standard practice for teacher training in knowledge distil- lation (Standard) (i.e., Eq. (T). We conduct experiments on benchmark datasets including CIFAR- 100 (Krizevskyy) 2009), Tiny-ImageNV
(Tin 2017). and ImageNet (Deng et al. 2009). We experi-
ment with various backbone networks including ResNet (He et al. 2016). Wide ResNet (Zagoruyko &
Komodakis (016) and ShuffleNet (Zhang et al. 2018b Tan et al. 2019). We test the applicability of SoTeacher from different aspects of model compression in know knowledge distillation including reduction of the width or depth, and distillation between heterogeneous neural architectures.
For knowledge distillation algorithms, we experiment with the original knowledge distillation method (KD) (Hinton et al.] 2015), and a wide variety of other sophisticated knowledge distil- lation algorithms (see Sect. 5.2). We report the classification accuracies on the test set of both teacher and the student distilled from All results except ImageNet are presented with mean and standard deviation based on independent runs. For Imageen. we also report the top-5 accuracy. For hyperparameters, we set max 1 for both datasets, ALR 10-5 for CIFAR- 100 and ÅLR 10-6

---

Published as a conference paper at ICLR 2024
for Tiny- Imaeegegee///mgeeeett More detailed hyperparameter settings for neural network training and knowledge distillation can be found in Appendix D
Table 1:
Test accuracy of the teacher and student with knowledge distillation conducted on CIFAR- 100. SoTeacher achieves better student accuracy than Standard for various architectures, depsite lower teacher accuracy.
WRN40-2/WRR40-11
WRN40-2/WRN66-2
ResNaa322c4/ShuffeeeWe22
Student
Teacher
Student
Teacher
Student
Teacher
Standard
73.73 +: 0.13
76.38 + 0.13
74.87 0.45
76.38 + 0.13
74.86 + 0.18
79.22 + 0.03 78.49 + 0.09
SoTeacher
74.35 +: 0.23
74.95 + 0.28
75.39 + 0.23
74.95 + 0.28
77.24 + 0.09
no-CR
74.34 + 0.11
0.111
74.30 + 0.12
75.20
0.24
74.30 + 0.12
76.52 + 0.52
77.73 + 0.17
no-LR
73.81 + 0.15
76.71 + 0.16
75.21 + 0.13
76.71 + 0.16
76.23 + 0.18
80.01 + 0.18
Table 2: Test accuracy of the teacher and student with knowledge distillation conducted on Tiny- ImageNet and ImageNet. The student network is ResNet18, while the teacher network is ResNet34 for Tiny-ImageNet and ResNet152 for ImageNet. Due computation constraints, we are not able to perform ablation experiments on ImageNet.
Tiny-ImageNet
ImageNet
Student (Top-1)
Student (Top-5)
Teacher (Top-1)
Teacher (Top-5)
Student (Top-1)
Teacher (Top-1)
Standard SoTeacher
66. 0.17
85.74
64.94
0.32
84.33 0.40
71.30 71.45
77.87
66.83 + 0,20
86.19 + 0.22
64.88 0.48
84.91 = 0.41
77.11
no CR
66.39
0.27
86.05 + 0.17
64.36 0.43
84,10 0.27
no-LR
66.48 + 0.43
86.20 = 0.40
64.26 + 1.54
84.48 = 0.84
5.2 RESULTS
End-to-end Knowledge Distillation Performance. Tables ! and show the evaluation results on CIFAR-100 and improve the student's test accuracy consistently across different datasets and teacher/student architecture pairs. Note that the success of our teacher training method is not due to the high accuracy of the teacher. In Tables [nd 2 one may already notice that our regularization method will hurt the accuracy of the teacher, despite that it can improve the accuracy of the student distilled from it.
respectively. Our teacher training method SoTeacher
On ImageNet particularly, the gain in student's s accuracy by SoTeacher is minor, potentially because the large size of the dataset can already facilitate the learning of true label distribution as suggested by our theory. Nevertheless, one may observe that the teacher's accuracy is significantly lower when using SoTeacher, which implicitly suggests our teacher regularization method is tailored towards the student's ' accuracy.
Table 3: Estimation of the uncertainty quality of the teacher network trained by Standard and SoTeacher. The uncertainty quality is estimated by the ECE and NLL both before and after tempera- ture scaling (TS).
Datasct
Teacher
Method
ECE
NLL
ECE (w/ TS)
NLL (w/ TS)
CIFAR-100
WRN40-2
Standard
0.113 + 0.003
1.047 + 0.0U7
0.028 + 0.004
0.905 + 0.008
SoTeacher
0.057 : 0.002
0.911 + 0.013
0.016 + 0.003
0.876 : 0.012
CIFAR-100
ResNe132x4
Standard
0.083 + 0.003
0.871 + 0.010
0.036 = 0.001
0.815 = 0.008
SoTeacher
0.037 + 0.001
0.777 + 0.006
0.021 + 0.001
0.764 + 0.003
Standard
0.107 + 0.007
1.601 = 0.037
0.043 = 0.002
1.574 = 0.015 1.505 1 0.033
Tiny-ImageNet
ResNet34
SoTeacher
0.070 1 0.007
1.496 1 0.031
0.028 1 0.002
Ablation Study, We toggle off the Lipschitz regularization (LR) or consistency regularization (CR) in SoTeacher (denoted as no-LR and no-CR, respectively) to explore their individual effects. As shown in Tables I and 2 LR and CR can both improve the performance individually. But on average, SoTeacerr achieves the best performance when combining both LR and CR, also demonstrated
7

---

Published as a conference paper at ICLR 2024
in our theoretical analyses. Note that in Table 2 using Lipschitz regularization is not particularly effective because the regularization weight might not be properly tuned (see Figure 2
Quality of True Distribution Approximation. To further interpret the success of our teacher training method, we show that our regularization can indeed improve the approximation of the true label distribution thus benefiting the student generalization. Directly measuring the quality of the true distribution approximation is infeasible as the true distribution is unknown for realistic datasets. Follow previous works (Menon et al. 2021), we instead estimate the approximation quality by reporting the Expected Calibration Error (ECE) (Guo et al., 2017) and NLL loss of the teacher on holdout set with one-hot labels. Since scaling the teacher predictions in knowledge distillation can improve the uncertainty quality (Menon et al. 202I), we also report ECE and NLL after temperature scaling, where the optimal temperature is located on an additional holdout set (Guo et al.] 2017). As shown in Table our teacher training method can consistently improve the approximation quality for different datasets and teacher architectures.
Effect of Hyperparameters. to study the effect of two regularization terms introduced by our teacher training method. For Lipschitz regularization, we modulate the regularization weight ÅLR. For consistency regularization, we try different maximum regularization weights >CR and different weight schedulers including linear, cosine, cyclic, and piecewise curves. Detailed descriptions of these schedulers can be found in Appendix D As shown in Figure 2 both Lipschitz and consistency regularizations can benefit the teacher training in terms of the student generalization consistently for different hyperparameter settings. This demonstrates that our regularizations are not sensitive to hyperparameter selection. Note that the hyperparameter chosen report the results in Tables 2 might not be optimal since we didn't It perform extensive hyperparameter search in fear of overfitting small datasets. It is thus possible to further boost the performance by careful hyperparameter tuning.
We conduct additional experiments on Tiny-ImageNet as example
In particular, Figure 2ia) shows that, as Lipschitz regularization becomes stronger, the teacher accuracy constantly decreases while the student accuracy inereases and converges. This demonstrates that excessively strong Lipschitz regularization hurts the performance of neural network training, but it can help student generalization in the knowledge distillation context.
[a]
[b]
F 67.0
Test Acc. (%) (Top-1) 1)
Test Acc. (96) Top-1
06.5
[Top=
Student
66.0
Student
Student
66-
Teacher
Test Acc.
66-
64
Tea thac
#4-
Teacher
0.0
2
Undor Standard
Cycl
Piecewise
AuR
ACR
mac
Ace Scheduler
Figure 2: Effect of varying the hyperparameters in our teacher training method, including the weight for Lipschitz regularization ÄLR. the weight for consistency regularization ACR, and its scheduler. The settings of our method used to report the results (e.g. Table 2 are denoted as "A" The standard teacher training practice denoted as for comparison.
Other Knowledge Distillation Algorithms. Besides the original knowledge distillation algorithm, experiment with various feature distillation algorithms including FitNets (Romero al. 2015).
AT (Zagoruyko ko & Komodakis 2017). 7).. SP (Tung & Mori 2019), CC (Peng et al. 2019). VID (Ahn
RKD (Park al. 2019) PKT (Passalis & Tetas 2018), AB (Heo al. 2019). 9 FT Kim ct al. 2018). NST (Huang& & Wang 2017). CRD (Tian et al. 2020) and SSKD (Xu et al. 2020). For the
al.
2019),
implementation these algorithms, we refer to existing repositories for knowledge distillation Tian
et al. 2020 (Shah et al. [2020] Matsubara] 2021) and mnttor-proviee codes. Although these distillation algorithms match all types of features instead predictions between teacher and student, they will achieve the best distillation performance when combined with the prediction distillation (i.c. original KD). Therefore, our teacher training method should still benefit the effectiveness of these distillation algorithms. We also experiment with curriculum distillation algorithm RCO (Jin et al.) 2019) which distills from multiple checkpoints in teacher's ''s training trajectory. Our teacher training method should also benefit RCO as the later teacher checkpoints become student-oriented. As shown in Table 6
8

---

Published as a conference paper at ICLR 2024
our SoTeacher can boost the distillation performance of almost all these distillation algorithms, demonstrating its wide applicability.
Student fidelity. Recent works have underlined the importance of student fidelity in knowledge distillation, namely the ability of the student to match teacher predictions (Stanton et al. (02T). Student fidelity can be viewed as measure of knowledge distillation effectiveness that is orthogonal to student generalization, as the student is often unable to match the teacher predictions although its accuracy on unseen data improves (Furlanello al. 2018 Mobahi al. 2020). Here we measure the student fidelity by the average agreement between the student and teacher's 's top-1 predicted labels on the test set. As shown in Table 4 our teacher training method can consistently and significantly improve the student fidelity for different datasets and reacher-studen pairs, which aligns with the improvement of student generalization shown by Table I and 2 This demonstrates that the teacher can better transfer its "knowledge" to the student with our training method.
Table 4: Average agreement (%) between the student and teacher's top-1 predictions on the test set.
CIFAR-100
Tiny-ImageNe ResNet34/ResNet188
WRN-40-2/WRN--4
WRN-40-2/WRN-1122
ResNet22x44Shtffeeeee2
Standard
76.16 1 0.144 77.92 = 0.27
76.92 1 0.29
76.63 1 0.25
71.33 1 0.07
SoTeacher
79.41 0.11
80.36 = 0.13
73.36 + 0.25
RELATED WORK
Understand knowledge distillation. There exist multiple perspectives in understanding the effectiveness of knowledge distillation. Besides the statistical perspective which views the soft approximation of the true label distribution (Menon et al. 2021 I
prediction of the teacher
Dao et al. 202T), another line of work understands knowledge distillation from a regularization perspective, which views the teacher's soft predictions as instance-wise label smoothing Yuan et al. 2020 Zhang & Sabuncu 2020; Tang et al. 2020). More recently, -AllenZZhu & Li 2020) understands knowledge distillation from a feature learning perspective by focusing on the data that possesses a "multi-view" structure, that multiple features co-exist in the input and can be used to make the correct classification. Knowledge distillation is effective because the teacher can learn different features and transfer them to the student. Our theory is built on similar assumption on the data structure, albeit we require the teacher to learn the true label distribution of the feature. Our theory thus can also be viewed as bridge between the statistical perspective and feature learning perspective.
Alleviate "teacher overfitting". Since in knowledge distillation, the distillation data is often reused from the teacher's training data, a teacher trained toward convergence is very likely to overfit its soft predictions on the distillation data. Intuitively, it is possible to tackle this problem by early stopping the teacher training (Cho & Hariharan 2019). However, a meticulous hyperparameter search may be required since the epoch number to find the best checkpoint is often sensitive to the specific training setting such as the learning rate schedule. It is also possible save multiple early teacher checkpoints for the student to be distilled from sequentially (Jin et al. 2019). Additionally, one can utilize a "cross-fitting" procedure to prevent the teacher from memorizing the training data. Namely, the training data is first partitioned into several folds, where the teacher predictions on each fold are generated by the teacher trained only on out-of-fold data (Dao et al., 2021). One can also train the teacher network jointly with student's 's network blocks, which imposes a regularization toward the student performance (Park et al. 202I). Different from these attempts, we train the teacher to directly learn the true label distribution of its training data, leading to simple and practical student-oriented teacher training framework with minimum computation overhead.
CONCLUSION AND FUTURE WORK
In this work, we rigorously studied the feasibility to learn the true label distribution of the training data under a standard empirical risk minimization framework. We also explore possible improvements of current teacher training that facilitate such learning. In the future, plan to adapt our theory to other knowledge distillation scenarios such as transfer learning and mixture of experts, and explore more effective student-oriented teacher network training methods.

---

margin bounds for Published as a conference paper at ICLR 2024
REFERENCES
Tiny imagenet, 2017. URL htw.:/tppps//ww...gge.e....////p
Sungsoo Ahn, Shell Xu Hu, Andreas C. Damianou, Neil D. Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. CVPR, pp. 9155-9163, 2019.
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. ArXiv, abs/2012.098116 2020.
Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky.
neural networks. In NIPS, 2017.
David Berthelot, Nicholas Carlini, Ekin Dogus Cubuk, Alexey Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. ArXiv, abs/1911 19911111975. 2019.
David M. Blei, A. Ng, and Michael I. Jordan. Latent dirichlet allocation. J. Mach. Learn. Res., 3: 993-1022, 2001.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil Model compression. In Proceedings of the 12th ACM SIGKDD international conference Knowledge discovery and data mining, pp- 535-541, 2006.
Xinjiao Chen. new generalization of chebyshev inequality for random vectors. ArXiv, abs/0707.0805, 2007.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. ICCV, pp. 4793-4801, 2019.
Tri Dao, Govinda M. Kamath, Vasilis Syrgkanis, and Lester W. Mackey. Knowledge distillation as semiparametrie inference. ArXiv, abs/2104.09732, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Chengyu Dong, Liyuan Liu, and Jingbo Shang. Double descent adversarial training: An implicit label noise perspective. ArXiv, abs/21 10.03135, 2021.
Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandku- mar. Born again neural networks. ICML, 2018.
Yonatan Geifman, Guy Uziel, and Ran EI- Yaniv. Bias-reduced uncertainty estimation for deep neural classifiers. In ICLR, 2019.
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J. Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 110:393-416. 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. ArXiv, abs/1706.04599, 2017.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pp. 770-778, 2016.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Laksh- minarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. ArXiv, 1atp//991..281... 2020.
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. AAAI, 2019.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. ArXiv, abs/1503.00231, 2015.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. ArXiv, abs/1 7001700771111 2017.
10

---

Published as a conference paper at ICLR 2024
Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, and Xiaolin Hu. Knowledge distillation via route constrained optimization. ICCV, pp. 1345-1354, 2019.
Jangho Kim, Seonguk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. ArXiv, abs/1802.0440 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. ArXiv, Abss/1610.0222,, 2017.
Zhengzhi Lin, Yueyao Wang, and Yili Hong. The poisson multinomial distribution and its applications in voting theory, ecological inference, and machine learning. 2022.
Junhao Liu, Linjun Shou, Jian Pei, Ming Gong, Min Yang, and Daxin Jiang. Cross-lingual machine reading comprehension with language branch knowledge distillation. In COLING, 2020.
Yoshitomo Matsubara. torchdistill: A modular, configuration-driven framework for knowledge distillation. In International Workshop on Reproducible Research in Pattern Recognition, pp. 24 44. Springer, 2021.
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In ICML, 2021.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. ArXiv, abs/l 802.05957. 2018.
Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-3361, 2020.
Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. Confidence-aware learning for deep neural networks. ICML, 2020.
Rafael Müllrr, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In NeurlPS, 2019.
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, 2015:2901-2990 2015.
Dae Young Park, Moonsu Cha, Changwook Jeong, Daesin Kim, and Bohyung Han. Learning student-friendly teacher networks for knowledge distillation. ArXiv, abs/2102.07650, 2021.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. CVPR, pp. 3962-3971, 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In ECCV, 2018.
Baoyun Peng, Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Yu Liu, Dongsheng Li, and Zhaoning Zhang. Correlation congruence for knowledge distillation. ICCV, pp. 5006-5015, 2019.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. CoRR, abs/1412.6550, 2015.
Shah, Avishree Khare, Neelay Shah, and Khizir Siddiqui. Kd-lib: A pytorch library for knowledge distillation, pruning and quantization, 2020.
Kihyuk Sohn, David Berthelot, Chun-L.ang Li, Zizhao Zhang, Nicholas Carlini, Ekin Dogus Cubuk, Alexey Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. ArXiv, abs/2001.07666 2020.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 34: 6906-6919, 2021.
11

---

Published as a conference paper at ICLR 2024
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. CVPR, pp.
2016.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoe V. Le. Mnasnet: Platform-aware neural architecture search for mobile. CVPR, pp. 2815-2823, 2019.
Jiaxi Tang and Ke Wang. Ranking distillation: Learning compact ranking models with high perfor- mance for recommender system. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H. Chi, and Sagar Jain. Understanding and improving knowledge distillation. ArXiv, abs/2002.03532, 2020.
Sunil Thulasidasan, Gopinath Chennupati, Jeff A. Bilmes, Tanmoy Bhattacharya, and Sarah Ellen Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In NeurIPS, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. ArXiv, abs/1910. 10699, 2020.
Frederick Tung and Greg Mori. Similarity preetetvigg knowledge distillation. ICCV, pp. 1365-1374, 2019.
Qifan Wang, Li Yang, Bhargav Kanagal, Sumit K. Sanghai, D. Sivakumar, Bin Shu, Zac Yu, and Jonathan L. Elsas. Learning extract attribute value from product via question answering: A multi- task approach. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining. 2020.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoe V. Le. Unsupervised data augmentation for consistency training. arXiv: Learning, 2020.
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- supervision. In ECCV, 2020.
Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. Model compression with two-stage multi-teacher knowledge distillation for web question answering system. Proceedings of the 13th International Conference on Web Search and Data Mining, 2020.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. ArXiv, ftths/175..10000 10941, 2017.
Li Yuan, Francis E. H. Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 3902-3910, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. ArXiv, abs/1605.00 0460,,14460 2016.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. ArXiv, abs/1612.039288, 2017.
Hongyi Zhang, Moustapha Cissé, Yann Dauphin, and David Lopez-Paz. mixup: risk minimization. ArXiv, abs/1710.09412, 2018a.
Beyond empirical
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. CVPR, pp.
2018b.
Yuping Zhang, Xiaoran Xu, Hanning Zhou, and Yan Zhang. Distilling structured knowledge into embeddings for explainable and accurate recommendation. Proceedings of the 1 13th International Conference Web Search and Data Mining, 2020.
Zhilu Zhang and Mert Rory Sabuncu. Self-distillation as instance-specifie label smoothing. ArXiv, abs/2006.05065, 2020.
12

---

Published as a conference paper at ICLR 2024
A PROOF
A.1 MODIFIED SOFTMAX
We provide more details for the modified Softmax we defined. Recall that
Definition A.1 (Modified Softmax).
exp( Em hmm)
Softmax(h))
(8)
(IIm Σx exp((mm,t))1//M
Our modified Softmax layer maps E MMK to the probabilistic output f(x) E RKK It can be viewed combination of an average pooling layer and Softmax layer, where the denominator of the softmax layer is geometric average of the sum-exp of the probabilistic output of each batch A...
A.2 LEMMA 3.3
Proof. Given the labeled dataset {(x(1) y()), the empirical risk of the network function f under cross cross-entropy loss can be written as
Σ (0) log (xx11)
Σ ())( loog (n
N
Σ (0) log Softmax (h())
N
exp( 1 Em R
Σ (0)
log
M
N
(Ilm En exp(1 1)1
1/M
1
Σ (0) llog
I1 exppin)
N
Im Ee (
IIm exp(ii!))
MN Σ (() log
m. IIm Σn (1
exp (h!
MN Σ 10) Σ log
Σn exp mk
m
MN Σ 10y Σ
log Softmax (oftmaxx(1())) (1()))
m
MN Σ 1)) Σ
log (
where we have defined piu Softmax(h (1) Softmax(w..f Here ( indicates the m-th patch of the input x.
Now recall our assumption that fø an invariant feature extractor, which means that for any and j. any m and m'. and any and 7. fe(((=)) = f=(1'(1 (z()) as long as 2m = Zm Since (8) (oftmax(1.10) Softmax(wec ())) ))) this means that pm pm
(6) (()) long as zm Zma-
13

---

Published as a conference paper at ICLR 2024
Therefore, the empirical risk can be regrouped as
11)0))
Σ
logpm
MN
m
12
logp=
MN
ZEZ
Σ 2E
Σ {1)=E2()
1(x(1)
log
M
N
where we have let P: = Pm where
Now we use the KKT condition to approach the empirical risk minimization problem since the only variable is Ps- The first-order stationarity condition gives
Σ ((y
)
+1 =1 0,
(10)
P=
MN
where @ indicates the Hadamard product. Now Hadamard product both sides by ps we have
MN Σ 1()(0) + Ap= 0,
and thus
Ps MNA Σ 1()
Now consider the condition 1 Ps have
Σ 1() = 1.
MNA
Since E; 16y(0))
1ill E z()
Hilz
Hilz E 2
x
MN
This leads
Σ 1))
P= N₂
(1)=52(0)
where N₂ = [i|= E Z00
A.3
LEMMA 3.4
Proof. The sum of independently but non-identically distributed multinomial random variables is known as the Possion multinomial distribution (PMD). We utilize known result of PMD to prove this lemma.
Proposition A.2 (Lin et al. 2022 ). Let I = (1i1.... i,m 1,... , be n independent random indicators where Lú IJ and EØ= Lij for each i. Let p. (pa,, .Pim) be the probability vector that I; is sampled from, where E Pij Let X be the sum of these n random
indicators, namely X = (X,,... X,,) Σm= I, Then we have
E X = (p1.." P.n)
(11)
where p.j = Σi Pij.
And
Σ Et Pij(1 pt)) ifj k. Di
(12)
== PisPik: fj
where E is the covariance matrix of X.
14

---

Published as a conference paper at ICLR 2024
Following this result, since 1)) is random indicator and y() ~ py (-12211), we have
E
P(-12)
Σ
(r(222())
(13)
N
(i)=EZ()
And let the covariance matrix of yz to be Eg. Let N₂ = (ilz E 2(0 be the number of inputs that contain feature 2, we have
K
tr E
Σ
(k/z
px (k)22))))
N2
(1)=E2() km
K
Σ
(1
pi (11)2111)
(14)
N2
(i)=E
k=1
(K 1)N
N2
where we utilized the fact that ΣB py (k(z)(1) R by the Cauchy-Schwarz inequality.
Then by a multivariate Chebyshev inequality (see, C.g.,Chor (2007). ) we have with probability
tr tr(2ig) 1 P(112)1
(K 1)N,
(15)
N2
Now finally, derive estimation of Ng. Recall that each input contains M feature names sampled from the feature vocabulary [Z] subject to the distribution pz. Therefore,
N₂ N
P -
E
= -1-
Mpz(2).
(16)
For simplicity, we assume Pz a uniform distribution, which means Pz(2) = 1/| for any z. Note that this is not necessary assumption since N₂ can also be trivially bounded by N. Therefore have
102 3(12))1 -K-1) 1)M 2|
(17)
Recall that M the number features (patches) each input, and [2] is the total number of features in the vocabulary. This result thus indicates larger feature vocabulary can improve the approximation while a more complicated input can hurt the approximation.
A.4 LEMMA 3.5
Proof. To prove this lemma we need an additional weak assumption on the true label distribution of features that are "similar".
Assumption A.3. Let 22,2²) define the pointwise mutual information (PMI) between features z and z'. Let va be concave and monotonically decreasing function, We assume
DEL (py (-1z) ||py (-12)) = (I(2,
(18)
Eq. (18) indicates that when sampling the features in an input, if two features are more likely to be sampled together, their true label distribution should be more similar.
Given Eq. (88 and recall that
1/M
1/M
(-)2 = II py (12mm)
II 8E2(8)
py (-12)
15

---

Published as a conference paper at ICLR 2024
it can be shown that the difference between the true label distribution of the imput that contains feature and the true label distribution of feature can be bounded. In specific, for i {il 2 have
Py (k(]z(1)) (klz)
DKL (pr(-1z) prr (-(((1)))
E (x)(k) log
1
Σ py (k(2)
II
Py (ke)²)
M
Py (k|2)
(EEZ())
1
Σ (k]2)
py (k|22))
log
M
pr (klz)
Z'EZ()
(19)
py (ki|z')
Σ
Py (k|2) log
M
py (k|z)
z'E2(0)
1
Σ
DEL (pr (1))
M
z'EZ(O
1
Σ (I(Zzz²)),
M
z'EZ())
S 2)(,.
where 2(1,2)= = 3/4 E. I(2, 2') is the average PMI between feature and other features 2' in an input that contains z. Here we utilize Jensen's 's inequality concave functions.
Now we can show that the average true label distribution of inputs that contain feature 2 approximates the true label distribution of feature z. Recall the definition of average true label distribution of inputs that contain feature z is p(-1z) N Σ (ile Pv (-)z:())). And we have
(P((12) py (-12)111
1
Σ
pr
pr(( (-2))
N
(11zz00))
1
Σ
(pr(-12(0))
px (112)
N
(1)
Σ
pr(()(00))
pr(
N
(1)x
1/2
(20)
Σ
DI22 (pr((=))lpr(((100)))
N
(i)zEZ()
21/2
Σ
(T(((
N
(1)
2//2Ng
1/2 (i(=))
N
2/2
1/2 (I(z))
|
where 1(2) = N. min I(i, 2) is the average PMI over all inputs that contain feature 2.
16

---

Published as a conference paper at ICLR 2024
A.5
THEOREM 3.6
Proof. Recall the probabilistic output of the minimizer of the empirical risk is
1/M
1/M
(x(x())
II
Pm
II
P=
(21)
(E200)
where pa defined in Section A.2 is the network's probabilistic output of each feature z.
Note that for all k. f (x)(1) (k) = 1 and py (k((xx(0)) = 1. Therefore,
( pr (-10)))))) eexp
[log[S"<> (x)()) log(pr (-1a:(0))
1
exp
(logP= logpy (-12)
M
(EZ(())
1
exp
logg: P: boggy(+1221)
M
2E2()
1
exp
Σ
Blog ll= - (+
M
ZEZ
py (-|2)]112 8
(22)
max xE2())
9
max
|0 py (-12)ll2
#E2())
(10, p(-12)ll + 11p(-12) - py pp 8
max =E2())
8
max 2E2())
(1 p(-12)112 11p(-12) - (++))))
8
(K 1)M
1
21/2 M
5 +
v
1/2 (mmin)
21
2
where B min(ming (k)) ming py (k(z=(1)) and Imi = min I(2).
A.6 THEOREM 3.9
Proof. We first present a Lemma similar to Lemma 3.3 which shows that the probabilistic predictions of features will still converge to the sample mean of the labels where the corresponding inputs contain this feature, up to some constant error.
Lemma A.4 (Convergence of the probabilistic predictions of features with Lipschitz-continuon and transformation-robus feature extractor). Let Ds := 1/ CHlez(0) (2)( where z denote the set feature names in the i-th imput, and thus [i]= E 2 denotes the set of inputs that contain feature Let f be minimizer of the empirical risk (Eq. 4T)) and assume ft is sxx-Lischhit--onninee and L--ttrasformatioon-reuss feature extractor. Let pf (Ix) = Softmax(wc fo. (xx)). We have with probability 8,
lpy- - Ezll S La,
(23)
where I, == 2Tr + Tx))(%)
17

---

Published as a conference paper at ICLR 2024
Proof. Since fi is Lx-LLipschitz and La-transformaiio-ronn feature extractor, for any j, for any m, m' and any 7, 7,, as long as 2m = 2m' = 2, we will have,
[[f((mm)) f=((")))|
= =Ilfe(1(a())) fe(n' f
(24)
<1VE(?(<!!)) - + - Sx(( [[]((a! (
$2Lr + Lx|l=! -x
Now since the feature representation in the input space always concentrated, by Chebyshev's inequality we will have with probability
| (0 - - =0:01=00 (
Therefore, we have with probability 1- 6,
[ - S S S Lar
where I. == 2Lr + (x))
The empirical loss minimization now becomes
min MN E 21y Σ
log 1m
(25)
T
s.L.
Ip
( Iiz, 2m 2mn
Let S. {ilz E z for simplicity, we can have
MN Σ (0)
log p
MN 11
Σ logp!
SEZ())
1
Σ Σ
12y(0)
logg!"
MN
(26)
#EZ (ilzEZ(1)
ΣΣ ((y()) logp!"
= MN
2ES
Note that this basically means that we assign the label of an example ( each feature contained in this example's 's input.
Therefore. 25 will be equivalent to the following problem.
Σ Σ (1) log pg
min (
MN
ZEZ iES,
s.t. II
| < L., 1 p = Vz, Vi,j jj.
Note that the constraint 1 S L, is only imposed in each subset Sz.
Using the KKT condition, the above problem can be further formulated as follows.
1
ΣΣ 1( logp!"
min (
MN
ZEZ FES.
Σ
Σ
Maish (ID!") - -p Lz)
j.keS.itk
(27)
Σ
Asa(1 (1.1)) p/. 1),
IES.
s.t. Mzijk (1pp()) (8) || L.) Mayk 2 0, 22, Vj.k S..J
|p! 1/ || < L,,
1 1.p())
(1) =1, 22, Vj,klL Sz,j # k.
18

---

Published as a conference paper at ICLR 2024
Using the first-order stationarity condition have, V Vi Sz:
V
P
1
+2
Σ (a - _ + X,1 = 0,
(28)
MN
where for simplicity we neglected the subscript z since the condition the same for all
Sum
28
over we have
E 1
2
3E (o p!²)) Σ A1100.
(29)
MN
P
Note that E;
Exys Mik (pe)
Ps
(1)
since each pair of i, appears twice in the sum, where jank
the same but the sign of p= pz
(1)
(k) different. Therefore
1
1
+ -
(30)
MN
Notice that to minimize the loss, if ( اا), it is necessary that (0) = (0)
Now we dot product both sides of 30 with (k), we have
Hily KH
+ Σ A = O. ViE (1 = k))
MN
ps [k]
which means that
Hily k3] MN E: Ai
[k] [
yf k.
Now recall the constraint that when j. This at least indicates that
!p! p!-1 L..
!p 1lo S L..
Therefore,
(ily = WHHl
S L,,, if + (0)
MN MNE, X
This implies that
MN E, A E10000
Now consider the condition 1 p
1, we will have
Σ N MN
Thus
|100) 1z
< Lzz
where we recall i. = ト 2. 1gy(9)
Now to prove Theorem 3.9
we only need combine Lemma A.4 and Lemmas 3.4 3.5 reasoning is exactly same as the proof of Theorem 3.6
where the
19

---

Published as a conference paper at ICLR 2024
B LIMITATIONS
In this paper we focus on the theoretical feasibility of learning the true label distribution of training examples with empirical risk minimization. Therefore we only analyze the existence of such desired minimizer, but neglect the optimization process to achieve it. By explore the optimization towards true label distribution, potentially more dynamics can be found to inspire new regularization techniques.
Also, our proposed method for training sudent-oriented teacher may not be able to advance the Mfateoof-the-ar significantly, as the regularization techniques based inspired by our analyses (e.g.. Lipschitz regularization and consistency regularization) may more or less be leveraged by existing training practice of deep neural networks, either implicitly or explicitly.
C IMPLEMENTATION DETAILS
Lipschitz regularization. Following previous practice using Lipschitz regularization for gener- alization on unseen data (Yoshida & Miyato 2017) or stabilizing generative model (Miyato et al., 2018), we regularize the Lipschitz constant of a network by constraining the Lipschitz constant of each trainable component. The regularization term is thus defined as &LR = E, Lip(/), where denotes a trainable component in the network f. The Lipschitz constant of network compo- nent Lip(f) induced by a norm || is the smallest value L such that for any input features h, h', (l) - [h''ll S L|/ - I/ Here we adopt the Lipschitz constant induced by 1-norm, since its calculation is accurate, simple and efficient. For calculating the Lipschitz constants of common trainable components in deep neural networks, we refer to (Gouk et al. 2021) for a comprehensive study.
We design our consistency regularization term as LCR = E. f(z;) f(z;) where we follow previous work (Laine & Aila 2017) and employ MSE to penalize the difference. Here f(x) is the aggregated prediction of an input æ, which we calculate as the simple average of previous predictions f(x); = Σ.= f()),, where we omit the data augmentation operator for simplicity. At epoch we simply skip the consistency regularization. Note
Consistency regularization.
that such prediction average can be implemented in an online manner thus there is no need to store every previous prediction of an input.
D DETAILS OF EXPERIMENT SETTING
D.1 HYPERPARAMETER SETTING FOR TEACHER NETWORK TRAINING
For all the experiments on CIFAR-100, we employ SGD as the optimizer and train for 240 epochs with batch size of 64. The learning rate is initialized at 0.05 and decayed by factor of 10 the epochs 150, 180 and 210, with an exception for ShuffleNet where the learning rate is initialized 0.01 following existing practice (Tian al. 2020 Park et al. (02I). The weight decay and momentum are fixed as 0.0005 and 0.9 respectively. The training images are augmented with random cropping and random horizontal flipping with a probability of 0.5.
For Tiny- ImageNet experiments, we employ SGD as the optimizer and conduct the teacher training for 90 epochs with batch size of 128. The learning rate starts 0.1 and decayed by a factor of 10 at epochs 30 and 60. The weight decay and momentum are fixed as 0.0005 and 0.9 respectively. The training images are augmented with random rotation with maximum degree of 20, random cropping and random horizontal flipping with probability of 0.5. For student training the only difference is that we train for additional 10 epochs, with one more learning rate decay at epoch 90, aligned with previous settings (Tian et al. 2020).
For consistency regularization in our teacher training method, we experiment with various weight schedules besides the linear schedule mentioned in the main paper. We list the formulas for these schedules in the following. Here denotes the epoch number, T' denotes the total number of epochs, and X max denotes the maximum weight.
20

---

Published as a conference paper at ICLR 2024
Table 5: for different feature distillation algorithms
8
8
Standard
SoTeacher
Standard
SoTeacher
Standard
SoTcacher
PitNets
100
50
AT
1000
500
SP
3000
1500
CC
0.02
0.01
VID
1.0
0.5
RKD
1.0
0.5
PKT
30000
15000
AB
1.0
0,5
FT
200
100
NST
50
25
CRD
0.8
0.5
Cosine schedule:
(
t
π
(t) cos
ACR
T
2
Cyclic schedule:
2 ()
Acn(t) = 11
A
max
T
Piecewise schedule:
0
0 0<tS
ACR(t)= =
max
/2
T/3 2T/33,
CR
max CR
<
D.2 HYPERPARAMETER SETTING FOR KNOWLEDGE DISTILLATION ALGORITHMS
For knowledge distillation algorithms we refer to the setting in RepDistiller Specifically, for original KD, the loss function used for student training is defined as
FaCross-Ftropy + (1-a))kn)
We grid search the best hyper-
hyper-parameters parameters that achieve the optimal performance, namely the loss
scaling ratio is set as 0.5 and the temperature is set as 4 for both CIFAR-100 and Tiny-ImageNet. For all feature distillation methods combined with KD the loss function can be summarized as (Tian et al. 2020
f CCCross-Eotoppy alKD 3
where we grid search the optimal 7 and to be 1.0 and 1.0 respectively. When using our teacher training method, all these hyperparameters are kept same except that for all feature distillation algorithms the scaling weights corresponding to the feature distillation losses 3 are cut by half, as we wish to rely more on original KD that is well supported by our theoretical understanding. Table 5 list 3 used in our experiments for all feature distillation algorithms. For SSKD (Xu et al. 2020)
the hyperparameters are set as A₁ 1.0, Ag = 1.0, Àa = 2.7, A₁ = 10.0 for standard training and d1 = 1.0, X 1.0, Ag- 1.0, Ay 10.0 for methods. For the curriculum distillation algorithm RCO we experiment based on one-stage EEI (equal epoch interval). We select 24 anchor points (or equivalently every 10 epochs) from the teacher's saved checkpoints.
E ADDITIONAL EXPERIMENT RESULTS
Training overhead. Compared to standard teacher training. the computation overhead of SoTeacher mainly due the calculation of the Lipschitz constant, which efficient as it only requires simple arithmetic calculations of the trainable weights of neural network (see Section 4) Empirically we observe that training with SoTeacher only slightly longer than the standard training for about 5%. The memory overhead of SoTeacher is incurred by buffering average prediction for each input. However, since such prediction requires no gradient calculation we can simply store it in mapory-mapped file.
http
21

---

Published as a conference paper at ICLR 2024
Table 6: SoTeacher consistently outperforms Standard on CIFAR-100 with various KD algorithms.
WRN40-2/W VRN40-2/WRN40-1
WRN40-2/WRNI6-2
ResNer32x4/Shuffte22
Standard
SoTeacher
Standard
SoTeacker
Standard
SoTeacher
FitNet
74.06 + 0.20
74.88 + 0.15
75.42 + 0.38
75.64 + 0.20
76.56 0.15
77.91 + 0.21
AT
73.78 #
75.12 = 0.17
75.45
+ 0.28
75.88 + 0.09
76.20 = 0.16
77.93 + 0.15
SP
0.22
74.71 + 0.19
74.67
+ 0.37
75.94
0.19
78.06 1 0.34
CC
0.12
74.76 + 0.16
75.08 1 0.07
75.67 1
7.14
77.68 + 0.28
VID
0.30
74.89 - 0.19
00,11:0.000
7.11
0.11
77.37 0.16
RKD
0.08 74.66 0.44
0.08
75.16 + 0.21
75.59
0.12
77.46 0.10
PKT
0.44
75.45
+ 0.09
75.53
77.84 7
AB
74.86 1 0.10
70.09 + 0.66
70.38 +
7..27
78.05 0.21
FT
0.15
74.75 + 0.13
75.19
+ 0.15
75.68 +
76.1
77.56
NST
+0.4
74.74 +.14 0.14
74.95 + 0.23
75.68 1.18+0106
76.07
77.71+ 0.10
CRD
7..1
75.06 - 0.37
75.52 + 0.12
75.95 1
0.12
78.09 0.13
SSKD
0.22
7
76.31
+ 0.07
76.32 1: 0.09
7..4
79.37 # 0.11
RCO
0.32
74.81 = 0.04
75.24
+ 0.34
75.50 + 0.12
76.75 0.13
77.59 + 0.31
Table 7: Performance of the knowledge distillation when training the teacher using existing regular- ization methods for learning quality uncertainty on unseen data.
WRN40-2/WRN40-11
Student
Teacher
Standard SoTeacher
73.73 1: 0.13
76.38 1 0.13
74.35 + 0.23
74.95 0.28
Eg (5 10-4)
73.73 + 0.13
76.38 + 0.13
61 (10-9)
73.60
73.52 +: 0.05
Mixup 0.2)
73.19 0.21
77.30 + 0.20
Cutmix (a 0.2)
73.61 + 0.26
78.42 = 0.07
Augmix (a 1. 3)
73.83 1 0.09
77.80 1 0.30
CRI. = 1)
74.13 + 0.29
76.69 + 0.16
F EXPERIMENTS WITH UNCERTAINTY REGULARIZATION METHODS ON UNSEEN DATA
Uncertainty learning on unseen data. Since the objective of our student-orientes teacher training is to learn label distributions the training data, is related to those methods aiming to learn quality uncertainty on the unseen data. We consider those methods that are feasible for large teacher network training, including (1) classic approaches to overcome overfitting such as & and &g regularization, (2) modern regularizations such as label smoothing (Szegedy al. 2016) and data augmentations such as mixup (Zhang et al.] 2018a) and Augmix (Hendrycks et al. 2020), (3) Post-training methods such temperature scaling (Guo al. 2017). as well as (4) methods that incorporate uncertainty
learning ojective such as confidence-awand learning (CRL) (Moon et al. 2020).
We have conducted experiments on CIFAR-I00 using all these methods and the results can be found in Appendix F Unfortunately, the performance of these regularization methods is unsatisfactory in knowledge distillation only CRL can slightly outperform the standard training. We believe the reasons might be two-folds. First, most existing criteria for uncertainty quality on the unseen data such as calibration error (Naeini et al. 2015) or ranking eтror (Geifman et al. 2019). only require the model to output an uncertainty estimate that is correlated with the probability of prediction errors. Such criteria may not be translated into the approximation error to the true label distribution. Second, even if model learns true label distribution unseen data, it does not necessarily have to learn true label distribution on the training data, as deep neural networks tend to memorize the training data.
Experiment setup. We conduct experiments on CIFAR-100 with teacher-student pair WRN40- 2/WRN40-1. We employ the original KD as the distillation algorithm. The hyperparameter settings the same as those mentioned in the main results (see Appendix (D). For each regularization method we grid search the hyperparameter that yields the best student performance. The results are summarized in Table
22

---

Published as a conference paper at ICLR 2024
Classic regularization. We observe that with stronger & & regularization the student performance will not deteriorate significantly as teacher converges. However, it also greatly reduces the performance of the teacher. Subsequently the performance of the student is not improved as shown in Table
Label smoothing. Label smoothing is shown not only improve the performance but also the uncertainty estimates of deep neural networks (Müller et al.) 2019). However, existing works have already shown that label smoothing can hurt the effectiveness of knowledge distillation (Müller et al.) 2019), thus we neglect the results here. An intuitive explanation is that label smoothing encourages the representations of samples lie in equally separated clusters, thus "erasing" the information encoding possible secondary classes in a sample (Müller et al. 2019).
Previous works have demonstrated that mixup-like data augmentation techniques can greatly improve the uncertainty estimation on unseen data (Thllasidasan et al. 2019 Hendrycks et al. 2020). For example, Mixup augmented the training samples == ax + (1 a)
Data augmentation.
and y :- ory same class as x.
a'y', where (æ', v/) ) is randomly drawn pair not necessarily belonging to the
As shown in Table 7 stronger mixup can improve the performance of the teacher, whereas it can barely improve or even hurt the performance of the student. Based on our theoretical understanding of knowledge distillation, distribution of an input stochastically throughout the training, thus hampering the learning of true label distribution.
conjecture the reason might be that mixup distorts the true label
Temperature scaling. Previous works have suggested using the uncertainty on validation set to tune the temperature for knowledge distillation either in standard learning (Menon al., (02T) or robust learning (Dong et al. (022). However, the optimal temperature may not be well aligned with that selected based uncertainty (Menon et al. (02T). We neglect the experiment results here the distillation temperature in our experiments is already fine-tuned.
Uncertainty learning. CRL designs the loss function = fCE + MCRLL where fcE is the cross-entropy loss and ÉCRL is an additional regularization term bearing the form of
fCRL = max (0, - -g(((=)),, c(x;;) - -p(=,) + lc(x,) - c(2)))),
(31)
where p(x) = maxx ((x) is the maximum probability of model's prediction on training sample x and
c(x) =
Σ (arg max f(x) =
the frequency of correct predictions through the training up to the current epoch. Here g(c,, cj = if > Cj and (Ca, 0ÿ) otherwise. Although originally proposed to improve the uncertainty quality of deep neural networks in terms of ranking, we found that CRL with a proper hyperparameter can improve the distillation performance, as shown in Table 7
We note that the effectiveness of CRL on distillation can be interpreted by our theoretical understand- ing, its regularization term (31) is essentially a special form of consistency regularization. To see this we first notice (31) is margin loss penalizing the difference between p(z) and c(z) in terms of ranking. We then rewrite c(x) as
c(x) 1,
Onehot (f())))
(32)
which is similar to our consistency regularization target, except the prediction first converted into one-hot encoding. Such variation may not be the best for knowledge distillation wish those secondary class probabilities the prediction be aligned as well.
23


Reviews:
[{"review_id": "mQSaTi4lHr", "reviewer": "Reviewer_UYE1", "summary": "This paper studies the problem on how to conduct teacher training to improve student model in knowledge distillation setting. Based on recent findings that the effectiveness of knowledge distillation hinges on the teacher’s capability to approximate the true label distribution of training inputs, this paper theoretically establish that the ERM minimizer can approximate the true label distribution of training data as long as the feature extractor of the learner network is Lipschitz continuous and is robust to feature transformations. In light of the theory, the paper proposes a teacher training method called SoTeacher which incorporates Lipschitz regularization and consistency regularization into ERM. Experiments on benchmark datasets using various knowledge distillation algorithms and teacher student pairs confirm that the proposed teacher training method can improve student accuracy consistently.", "questions": "My questions are listed in the weaknesses section.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: \n- This paper studies the feasibility of training the teacher to learn the true label distribution of its training data.\n- The paper theoretically proves that when the network is Lipschitz continuous and is robust to feature transformations, the ERM minimizer can approximate the true label distribution of training data. Empirical experiments show that adding Lipschitz and Consistency regularizations can improve student accuracy consistently.\n\nQuality:\n- The paper is well-written and easy to follow.\n\nSignificance:\n- Knowledge distillation is an important topic in both academia and industry. This paper studies the problem on how to conduct teacher training to better approximate the true label distribution on training data, which could benefit the community.", "weaknesses": "- Per my understanding, to make the teacher learn the true label distribution of its training data, the key is to resolve the overfitting issue on training data, because a teacher trained towards convergence tends to overfit the hard label. This paper proposes to add Lipschitz and Consistency regularizations to alleviate the overfitting issue. It would be great if the paper can have a more comprehensive comparison with other methods that were proposed to solve the same issue, e.g., early stopping. As shown in Figure 1(b), it seems the Standard teacher early stopped after 150 epochs has a pretty decent performance.\n- The Lipschitz and Consistency regularizations are one of the most important contributions of this paper, but the readers need to read Appendix to fully understand how these two regularizations are applied. Considering the length of the Appendix is not very long, I would suggest making them clear in the main paper.\n- In the experiments, it is unclear how the number of epochs impact the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ehmKd7Knkv", "reviewer": "Reviewer_c1De", "summary": "This paper presents SoTeacher, a method to train teacher models that lead to better student models after knowledge distillation. The main idea is to incorporate Lipschitz and consistency regularization in training the teacher model. The authors claim that this is motivated by the theoretical analysis that the authors conduct, which shows that the ERM minimizer can approximate the true label distribution as long as the feature extractor of the teacher is Lipschitz continuous and robust to feature transformations.", "questions": "1. Does the presented theory explain why a better teacher may not necessarily lead to a better student?\n2. Why does the method not perform as well on ImageNet?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Training better-suited and student-oriented teachers is an important problem in knowledge distillation. Theoretical analysis of distillation is also of interest to the field.\n* The theory sheds light on why distillation may work in practice: a teacher network can learn the true label distribution of the data under mild assumptions.\n* Some empirical results are presented that support the effectiveness of the method.", "weaknesses": "* The dependence on $\\delta$ in Lemma 3.5 seems to be very strong. Can Matrix Bernstein be used instead of Chebyshev to get a $\\log(1/\\delta)$ dependence?\n* The proposed theory does not explain why (higher test accuracy) teachers may not lead to better students. \n* The method seems to be at odds with the theory. If the assumptions of Lipschitz continuity and transformation-robustness of the feature extractor tend to be satisfied in practice, then the theoretical results hold. So, why do we need the additional regularization terms that are introduced by SoTeacher?\n* The approach introduces two additional hyperparameters corresponding to the two additional regularization terms that have to be tuned.\n* Evaluations on ImageNet, the largest and most real-world dataset considered, show only a marginal improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fHxSVQGJrS", "reviewer": "Reviewer_peLw", "summary": "This paper studies knowledge distillation and, in particular, the extent to which the teacher-model affects the student's performance.\n\nThe starting point of the authors' investigation is recent literature which suggests that the effectiveness of knowledge distillation hinges on the teacher's capability to approximate the true label distribution of training inputs. The authors show theoretical evidence which suggest that, as long as the feature extractor of the learner model is (i) Lipschitz continuous; (ii) is robust to feature transformations;  the ERM minimizer can approximate the true label distribution. Then, based on their theoretical considerations, the authors propose the \"SoTeacher\"  training method which incorporates Lipschitz regularization and consistency regularization into the teacher's training process.", "questions": "— Does the proposed method and theory works well/applies in NLP datasets/binary classification contexts?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Interesting theoretical approach for understanding the mechanics of distillation along with several experimental evidence to support the author's claim. The proposed method seems to give consistent improvements whenever applied (although sometimes rather marginal).", "weaknesses": "— The proposed method often times gives marginal improvements, while it introduces two extra hyper-parameters for the training of the teacher model (which is typically very costly to re-train in practical large-scale applications. That said, the authors do show that their method is relatively robust to the choice of these hyperparameters.)\n\n— The role of the teacher as a \"provider of estimates for the unknown Bayes conditional probability distribution\" is a theory for why distillation works that applies well mainly in the context of multi-class classification, and especially in the case where the input is images. (Indeed, there are other explanations for why knowledge distillation works, as it can be seen as a curriculum learning mechanism, a regularization mechanism etc see e.g. [1])\n\nIn that sense, I feel that the author should either make the above more explicit in the text, i.e., explicitly restrict the scope of their claims to multi-classifcation and images, or provide evidence that their technique gives substantial improvements on binary classification tasks in NLP datasets (but even in vision datasets).\n\n— One of the main reasons why knowledge distillation is such a popular technique, is because the teacher can generate pseudo-labels for new, unlabeled examples, increasing the size of the student's dataset. (This is known as semi-supervised distillation, or distillation with unlabeled examples, see e.g. [2, 3, 4]. ) It seems that the current approach, typically decreases the test-accuracy of the teacher model — a fact which typically impairs the student's performance in the semi-supervised setting [3, 4].\n\n\n[1] Understanding and Improving Knowledge Distillation [Tang∗, Shivanna, Zhao, Lin, Singh, H.Chi, Jain] \n\n[2] Big self-supervised models are strong semi-supervised learners [Chen, Kornblith, Swersky, Norouzi, Hinton] \n\n[3 ]Does Knowledge Distillation Really Work? [Stanton, Izmailov,  Kirichenko, Alemi, Wilson]\n\n[4] Weighted Distillation with Unlabeled Examples [Iliopoulos, Kontonis, Baykal, Trinh, Menghani, Vee]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A8YMlIpwFe", "reviewer": "Reviewer_Xgmp", "summary": "The authors investigates how to properly train a teacher model aimed at improving the student generalization performance, rather than the teacher generalization performance. Rooted in a theoretical analysis showing that a teacher model can indeed learn the underlying true label distribution they empirically propose to do so, by modifying the training objective of the teacher to include Lipschitz and consistency regularization.", "questions": "- Figure 1: Why does a student distilled from a teacher checkpoint with 55-60% accuracy yield a test accuracy of about 71%, thereby exceeding the teacher significantly? \n- Figure 1: It is known that early-stopping the teacher is beneficial, and SoTeacher does not improve over an early-stopped teacher (at 150 epochs). It is unclear why this approach is better than early stopping?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Research into how to train teachers explicitly aimed at improving student performance is important, and in large part orthogonal to most common distillation on improving knowledge distillation techniques.\n- The authors provide both theoretical analyses and supporting empirical experiments.\n- There are a sufficient amount of (diverse) experiments investigating the proposed approach.", "weaknesses": "I will separate my concerns into two parts; one on the theoretical analysis and one on the empirical experiments.\n\n**Theoretical**\n- The use of $\\mathbb{E}$ for the *empirical* expectation is very unconventional, and I would strongly suggest to change this notation. This also conflicts with mentions of distributions later on. It is unclear when we are considering random variables and emprical observations.\n- *\"[...] of the distillation data, which is often reused from the teacher’s training data.\"* It appears as reusing the teacher's training data is an underlying assumption of the paper, which is a notable assumption, and should be stated explicitly as an assumption.\n- *\"In particular, minimization of Eq. (1) would lead to $f(x) = 1(y)$ for any input x, [...]\"*. Why does this hold? There are no assumptions on $\\ell(f(x), y)$ in (1), and the statement would not hold in general?\n- Page 4 above (3). Is this the same softmax as other places? If not, why? If so, fix notation.\n- *\"[...] a continuous distribution with finite variance, namely $\\mathrm{Var}[X|z] \\leq \\nu_z$\"*. Note, $\\mathrm{Var}[X|z] \\leq \\nu_z$ is *bounded* variance not only finite variance. What is the true assumption here?\n\n**Empirical**\n- The authors never compares to the conventional and very simple setting of early-stopping the teacher and/or carefully tuning the temperature scaling of the teacher predictions as is commonly done in practice.\n- You both state that *\"This demonstrates that our regularizations are not sensitive to hyperparameter selection.\"* and argue that *\"It is thus\npossible to further boost the performance by careful hyperparameter tuning\"*, which seems contradicting.\n- About Figure 2(a): *\"[...] the teacher accuracy constantly decreases while the student accuracy increases and converges.\"* The teacher accuracy does not constantly decrease, but at best appear to have a downward trend. Furthermore, it is a stretch to say that the student accuracy converges due to no increase in 1 of 4 intervals. *\"This demonstrates that excessively strong Lipschitz regularization hurts the performance of neural network training [...]\"*. This is not supported by the figure where the teacher test accuracy for $\\lambda_{LR} = 0.5 \\cdot 10^{-6}$ and $\\lambda_{LR} = 2 \\cdot 10^{-6}$ is equal. Hence, the stronger regularization does not really affect the teacher performance.\n- Throughout the paper, it is argued that a reduced teacher performance can yield improvements in the student generalization. However, from Figure 2(c)  at the performance of both the teacher and the student increase when using the cosine, cycle or piecewise schedules compared to the standard procedure. In fact, only the case of linear schedule (used elsewhere in the paper) show an increase in student performance combined with a decrease in teacher performance. Thus, it is unclear what the effect the experiments actual show? Is the improvement merely an artifact of other parts of the training scheme?\n- There are no comparisons to Park et al. (2021), \"Learning Student-Friendly Teacher Networks for Knowledge Distillation\", although they provide experimental analysis on identical settings as this paper, and appear to notably outperform the proposed SoTeacher procedure. Granted, Park et al. (2021) propose a more involved procedure, but a comparison is needed.\n- Section on Student fidelity: Care must be taken, when interpreting teacher-student agreement (as argued by Stanton (2021)). Improved agreement does not imply that the student learns better from the teacher. This can merely be an artifact of improved student performance (or decreased teacher performance).\n\n**Minor**\n- The introduction lacks reference to Mirzadeh et al (2020), \"Improved Knowledge Distillation via Teacher Assistant\" on the teacher-student gap.\n- *\"[...] namely $x_z := \\gamma(x_z)$.\"* Overloaded notation here. Also, there is an inconsitent use of $x_{z_m}$ vs. $x_m$. For instance, it is unclear what $x_m$ in $h_m := f_E(x_m)$ refers to?\n- *\"[...] for any two inputs $i$ and $j$ [...]\"* and other places. It is odd to refer to inputs by their indices. Replace by $x^{(i)}$ or write *\"[...] for any two inputs indexed by $i$ and $j$ [...]\"*\n- Mention somewhere that the proofs are in the appendix.\n- Table 1: Reconsider the \"WRN40-2/WRN40-1\" notation in the table, as it is not clear what is the teacher and student.\n- The related works on \"Understand knowledge distillation\" miss references to strong theoretical works from Phuong and Lampert (2019), \"Towards Understanding Knowledge Distillation\", Borup and Andersen (2021), \"Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation\", and Mobahi et al. (2020), \"Self-Distillation Amplifies Regularization in Hilbert Space\".\n- The related works on \"Alleviate “teacher overfitting”.\" lacks reference to Dong (2019), \"Distillation ≈ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)