SoTeacher: Toward Student-oriented Teacher Network Training for Knowledge Distillation
Authors: Chengyu Dong
Liyuan Liu
Jingbo Shang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Knowledge distillation
Teacher-student training
Empirical risk minimization

Abstract: How to train an ideal teacher for knowledge distillation is still an open problem. It has been widely observed that a best-performing teacher does not necessarily yield the best-performing student, suggesting a fundamental discrepancy between the current practice in teacher training and the distillation objective. To fill this gap, we explore the feasibility of training a teacher that is oriented toward student performance with empirical risk minimization. Our analyses are inspired by the recent findings that the effectiveness of knowledge distillation hinges on the teacher’s capability to approximate the true label distribution of training inputs. We theoretically established that (1) the empirical risk minimizer can provably approximate the true label distribution of training data if the loss function is a proper scoring rule and the hypothesis function is locally-Lipschitz continuous around training inputs; and (2) when data augmentation is employed for training, an additional constraint is required that the minimizer has to produce consistent predictions across augmented views of the same training input. In light of our theory, we propose a teacher training method SoTeacher which renovates the empirical risk minimization by incorporating Lipschitz regularization and consistency regularization. Experiments on two benchmark datasets confirm that SoTeacher can improve student performance significantly and consistently across various knowledge distillation algorithms and teacher-student pairs.

Under review as a conference paper at ICLR 2023
SoTeacher: TOWARD STUDENT-ORIENTED TEACHER NETWORK TRAINING FOR KNOWLEDGE DISTILLATION
Anonymous authors Paper under double-bling review
ABSTRACT
How train an ideal teacher for knowledge distillation is still an open problem. It has been widely observed that a best-performing teacher does not necessarily yield the best-performing student, suggesting fuuddamental discrepancy between the current practice in teacher training and the distillation objective. To fill this gap, we explore the feasibility of training a teacher that oriented toward stu- dent performance with empirical risk minimization. Our analyses are inspired by the recent findings that the effectiveness of knowledge distillation hinges on the teacher's capability to approximate the true label distribution of training inputs. We theoretically established that (1) the empirical risk minimizer can provably approximate the true label distribution of training data if the loss function is a proper scoring rule and the hypothesis function is locally-Lipschin continuous around training inputs; and (2) when data augmentation is employed for training, an additional constraint is required that the minimizer has to produce consistent predictions across augmented views of the same training input. In light of our theory, we propose a teacher training method SoTeacher which renovates the em- pirical risk minimization by incorporating Lipschitz regularization and consistency regularization. Experiments on two benchmark datasets confirm that SoTeacher can improve student performance significantly and consistently across various knowledge distillation algorithms and teacher-student pairs.
INTRODUCTION
Knowledge distillation aims to train a small yet effective student neural network following the guidance of a large teacher neural network (Hinton et al. 2015). It dates back to the pioneering idea of model compression (Buciluá et al. (0066) and has a wide spectrum of real-world applications, such
recommender systems (Tang& & Wangl [018 Zhang et al. (2020). , question answering systems (Yang
et al. 2020 Wang et al. 2020) and machine translation (Liu et al. 2020).
of its crucial components, teacher training, is largely neglected. Existing training practice of teacher networks is often directly targeting at maximizing the performance of the teacher, which does not necessarily transfer the performance of the student. Empirical evidence shows that teacher trained toward convergence will yield an inferior student (Cho & Hariharan 2019) and regularization methods benefitting the teacher may contradictorily degrade student performance (Müller et al. 2019). As also shown in Figure [ the teacher trained towards convergence will consistently reduce the performance of the student after certain point. This suggests a fundamental discrepancy between the common practice in neural network training and the learning objective of knowledge distillation.
Despite the prosperous research interests in knowledge distillation,
In this work, we explore both the theoretical feasibility and practical methodology of training the teacher toward student performance. Our analyses are built upon the recent understanding of knowledge distillation from statistical perspective. In specific, Menon et al. (202 show that the soft prediction provided by the teacher is essentially an approximation to the true label distribution, and true label distribution as supervision for the student improves the generalization bound compared to one-hot labels. Dao et al. (202T) show that the accuracy of the student is directly bounded by the distance between teacher's prediction and the true label distribution through the Rademacher analysis.

---

Under review as a conference paper at ICLR 2023
75
Standard (Test Acc]
Final Test Acc
Acc
Standard (Train Loss)
60
SeTeacher (Test Acc)
72-
55
Teachr (Train Lass)
Standard Acc)
70-
SaTeacher (Test Acc)
45
160
220
140
200
240
Epoch
Teacher Checkpoint Epoch
(a) Test. accuracy and training loss of the teacher at different epochs in training.
(h) Student performance when distilled from different teacher checkpoints.
Figure 1: On CIFAR-100, we store teacher checkpoint every 10 epochs and distill student from it. We can observe that: (1) standard teacher trained towards better teacher performance may consistently deteriorate student performance upon knowledge distillation; (2) a teacher trained using SoTeacher can achieve better student performance even with lower teacher performance.
Based on the above understanding, a teacher benefitting the student should be able to learn the true label distribution of the distillation data | Since practically the distillation data is often reused from the teacher's training data, the teacher will have to learn the true label distribution of its own training data. This might appear to be infeasible using standard empirical risk minimization, as the teacher network often has enough capacity fit all one-hot training labels, in which case, distilling from teacher predictions should not outperform direct training based on one-hot labels. Previous theoretical analyses tend to evade this dilemma by distilling from teacher predictions only data that is not used in teacher training (Menon et al. 2021 Dao et al. 2021).
Instead, we directly prove the feasibility of training the teacher learn the label distribution of its training data. We show that the standard empirical risk minimizer can approach the true label distribution of training data as long the loss function is a proper scoring rule and the hypothesis function is locally Lipschitz continuous around training samples. We further show that when data augmentation is employed for training, our argument still holds true under an additional constraint, Le., predictions on the same training input under different augmentations have to be consistent.
In light of our theory, we show that explicitly imposing the Lipschitz and consistency constraint in teacher training can facilitate the learning of the true label distribution and thus improve the student performance. We conduct extensive experiments on two benchmark datasets using various lewgeegge distillation algorithms and different teacher-studed architecture pairs. The results confirm that our method can improve student performance consistently and significantly.
To summarize, main contributions be listed follows.
We show that is theoretically feasible to train the teacher to learn the true label distribution of the distillation data even with data reuse, which explains why the current knowledge distillation practice works.
We show that explicitly imposing the Lipschitz and consistency regularization in teacher training can better learn the true label distribution and improve the effectiveness of knowledge distillation.
We believe work is among the first attempts to explore the theory and practice of training Suudent-oriented teacher in knowledge distillation. We hope our exploration can serve as a stepping stone rethinking the teacher training and unleashing the full potential of knowledge distillation.
PROBLEM FORMULATION
Standard practice of teacher network training. We study knowledge distillation in the context of multi-class classification. Specifically, we are given a set of training samples D { y:) Je)]e where [N] 1,2, .N. D is drawn from probability distribution Px.y that is defined jointly over input space X c d and label space y [K]. In common practice, the teacher network
For simplicity, refer to the student's 's training data in knowledge distillation as the distillation data Stanton 2021
2

---

Under review as a conference paper at ICLR 2023
f: x [0, |0,1)00 is trained to minimize the empirical risk given loss function f, namely f arg min ((,))D ((f(x),
(1)
where (x the empirical expectation.
Learning objective of teacher in knowledge distillation. ing knowledge distillation suggest a teacher should approximate the true label distribution of the distillation data, which is often reused from the teacher's training data. This implies
Recent advances in understand-
f"
arg min ((D,)) [If(z) p" (1))
Here, p'(x) (-(x) denotes the true label distribution of an input namely the (unknown) category distribution that its label is sampled from, which is not necessarily one-hot. And || is distance function induced by an arbitrary p-norm.
Our Research Questions. One can find that there is a fundamental discrepancy between the
common practice of teacher training (Eq. 4 and the learning objective of the teacher in knowledge distillation (Eq. ( In particular, if the minimizer f in Eq. [1 is attained, f (z) 1, for any distillation input x, which significantly deviates from p (x) and challenges the effectiveness of knowledge distillation. Therefore, in this work, we explore the following two questions.
(i) Can a teacher network learn the true label distribution of the training data with the standard teacher training practice?
(ii) How to train a teacher to better learn the true label distribution and improve student performance?
We will present our findings of these two questions in Sec. and Sec. respectively.
THEORETICAL FEASIBILITY TO LEARN TRUE LABEL DISTRIBUTION OF TRAINING DATA
We now explore the theoretical feasibility of training a teacher that learns the true label distribution of the training data under the empirical risk minimization framework. Note that throughout the discussion here, our major focus is the existence of a proper minimizer, instead of the details of the optimization process.
3.1 HYPOTIEETICAL CASE WITH IDENTIFABLE SIBLINGS
We first show that in hypothetical case where the siblings of input known, the empirical minimizer can produce predictions close to the true label distribution on the training data. We define the notion of siblings the set of all inputs sharing the same label distribution in the set of training inputs Dx = (x, v) E D}.
Definition 3.1 (Siblings). We define the siblings of x in Dx under p' as Sp (x) = fa' E Dxlp" (x') p0 (x)).
The siblings of have the property that the sample mean of their labels converges to x's true label
distribution, namely Eres,. [1)] pxx). To see that, note the label of an input x a random variable following the true label distribution. namely y ~ ( and thus E[1y] = =p"(x) (Neal
2007). And per the definition of siblings, p p²('') p" (xx) for inputs x' E Spp (x), which means that the labels of the siblings are identically distributed.
We now propose that if the loss function is properly selected and the network function is properly regularized, the network minimizes the empirical risk on Sp. (x) will produce predictions close the true label distribution. We first introduce the notion of a proper scoring rule (Gneiting & Raftery 2007) and then discuss this proposition.
Definition 3.2 (Proper scoring rule). Let & be class of probability mass functions defined on V. el.,)) a proper scoring rule with respect to E for all E E. E(p. q) 2 E(q. q). with equality if and only if q. Note that many common loss functions are proper scoring rules (Lakshminarayanan et al. 2017), including the negative-lor likelihood loss (NLL) and the Mean Square Error (MSE) (also known as the Brier score (Brier 1950)).
3

---

Under review as a conference paper at ICLR 2023
Proposition 3.3 (Approximation eπror of the empirical risk minimizer on siblings). Let E(.,)) be proper scoring rule. Let be a family of functions that attains the same prediction on the siblings of namely for all F. fxz)= f(x), Vx' E (x)( Let f E F be a function that minimizes the empirical risk on Sp (I). Then for any x. E Sp (2)), with probability at least 6,
2K V (Sp-(x) (x)
2
(
log
8
The proof is rather straightforward. based the property of the proper scoring rule, can only achieve the minimum when f(z) = E.ss, f (z) [yy], which converges to the true label distribution as mentioned above. For example, when is NLL, we have B[-1, log f [x)) = -B[1y] log f (x) 2 - 11 log [8] [1y) with the minimum attained if and only if fax) = B[1y] (the sample mean is
abbreviated E for simplicity). Note that the last inequality here is in fact Gibbs' inequality.
3.2 REALISTIC CASE
Proposition 3.3 requires identifying the siblings of an input in Dx under the true label distribution p" which might not possible in realistic scenario, because p" is usually unknown and Dx may not necessarily contain the siblings of x. We now introduce the notion of neighborhood and show such a requirement can relaxed.
Definition 3.4 (Neighborhood). The neighborhood of an input is defined as the set of inputs that are close to x, namely ((x) = (x' E Dxx ||
x1 S r).
Now if we assume the true label distribution p' is L--locally- Lipschitz continuous, we will have for any x' (z), (x') p" (z)]l < Tr. This means that any input in the neighborhood of x: shares a similar true label distribution as æ, which is a relaxation of its siblings with an error controlled by L.
On the neighborhood set of x, we can still show that the empirical risk minimizer will produce predictions close to the true label distribution. However, due to the relaxation, additional error terms will be introduced to the approximation bound Eq. B And similar to siblings, the constraints on the loss function and network function would still be necessary. First, the loss function has to be a proper scoring rule. Second and slightly differently, the network function has to be locally-Lipschle continuous in the neighborhood s.
Finally, to establish the approximation bound all training inputs, need to partition the training set into multiple neighborhoods where the constraint of the empirical risk minimizer can be satisfied individually. We achieve this by introducing a covering.
Definition 3.5 (r-external covering). We say C is r-external covering of Dx, Df S Unecta' e X || - Il| < r}. We denote N as the covering number, namely the minimum cardinality of any external covering D..
We now ready present our main theorem.
Theorem 3.6 (Approximation error of the empirical risk minimizer to the true label distribution ). continuous be proper scoring rule. Let C be an r-external covering of Dx- Let F be a family of functions that are T-loocall-Llissciitt continuous in a norm ball of radius around x E C. Let f F be function that minimizes the empirical risk Eq. ( Let 2 be a number and Dx be a subset of Dx with cardinality at least (1 1/x 1/ for any x E Dxx any p-norm -| with probability at least 1 - 8. we have
Assume p" LL--locall-LLiscee
norm ball of radius r around x E X. Let E(-.))
2kNrK K 1 (x) D' (x))
log 3
LrEK +LEK 2.'r.
N
Here εK = -1 + K| K-11 where (() E RRKK denotes the vector where an arbitrary entry is 1 and others are 0, and 1 E R denotes the vector where all entries are 1.
One can find that as long as L and L are small enough, and the size of the training set is sufficiently large, the predictions can approach the true label distribution majority of the training inputs with
4

---

Under review as a conference paper at ICLR 2023
high probability. This suggests that teacher trained with standard empirical risk minimization is still possible to learn the true label distribution of its training data, thus resolving the dilemma of why knowledge distillation with data reuse improve student generalization. Eq. (4) also shows that the bound will be tighter if K is small, which implies learning the true label distribution of the training data will be significantly easier on classification tasks with fewer classes.
3.3 THE CASE WITH DATA AUGMENTATION
When data augmentation is employed in training, we note that the neighborhood defined in Sect. 3.2 needs be relaxed to include augmented training inputs. We now define the augmented neighborhood.
Definition 3.7 (Augmented neighborhood), Let A X + X denote set of data uugmentation functions. The augmented neighborhood of an input x defined as the set of imputs that are close to any augmented æ, namely Na(x)= {x's E Dxl || A(x))l S Tr, for any A A}.
Now if we assume the true label distribution is nearly invariant under data augmentations, namely (Axx)) p" (=)] L', will still have for any x' E Ma(x), ||p" (z') p" (x)]l I Lr + L'. Therefore, the approximation bound on the neighborhood can be readily extended to the augmented neighborhood here.
Theorem 3.8 (Approximation error of the empirical loss minimizer with data augmentation). Con- sider the same setting as Theorem 3.6 except each training input is now augmented by a set of augmentation functions
functiiono A. The empirical risk minimizer now becomes
f arg min E L.. ((f(A(z)),)) FEJ
(5)
Then with probability at least 1 - 8, for any Dx.
(p)))) 2k K
2 log 5
+ LrEK + 2 L7Ex + 1p,
(6)
N
where T. = maw.EC.AAA.. (7())))f''A ((())))(((()))) I max,EC.AEA..rENAAE (x')--
p² (A(x))]
Here L and Le cannot be simply bounded by the local Lipschitz continuities of the hypothesis function and true label distribution respectively, because the augmentation function A can significantly change the data vector of the input, which means [|A(x)) - x|| is not necessarily small. Therefore, to ensure the true label distribution is properly learned with data augmentation, one has to explicitly minimize these two error terms.
Note that although the training set is larger after data augmentation, the bound Eq. (6 is not improved by multiplier compared to the case with no augmentation Eq. 4 This is because the labels of augmented inputs are statistically dependent, thus the bound in the concentration inequality will not become tighter (see Lemma A.2).
4 PRACTICAL METHOD TO IMPROVE THE LEARNING OF TRUE LABEL DISTRIBUTION OF THE TRAINING DATA
Following our theory, we explore practical methods to train the teacher to better learn the true label distribution of the training data.
Loss function. Theorem s.ggest that the necessary condition to learn the true label distribution is to select proper scoring rule as the loss function. This is already sufficed by current teacher training practice where the loss function is typically NLL or MSE.
Lipschitz regularization. Theorem 3.6 also suggests that it is necessary to enforce the hypothesis function to be locally Lipschitz continuous around certain samples. This may also achieved by current teacher training practice, neural networks are often implicitly Lipschitz bounded (Bartlett) et al. 2017). Nevertheless, we observe that explicit Lipschitz regularization (LR) can still help in multiple experiments (See 5). Therefore, propose to incorporate global Lipschitz constraint into teacher's training. For the implementation details, we follow the existing practice of Lipschitz regularization (Yoshida & Miyato 2017 Miyato et al. 2018) and defer them Appendix
5

---

Under review as a conference paper at ICLR 2023
Consistency regularization. When data augmentation is employed in training, Theorem 3.8 suggests that it is necessary to ensure both the true label distribution and the teacher prediction of an imput will not change significantly after data augmentations, We note that these two constraints are missing in the current teacher training practice and thus have to be explicitly imposed. The former can be sufficed by selecting proper data augmentations such random cropping, rotation or flipping commonly seen in the training practice. The latter is known as the consistency regularization
(CR) (Laine & Aila 2017 Xie al. 2020 Berthelot et al. 2019 Sohn al. 2020) that is widely
used in semi-supervised learning.
To maximize the training efficiency of consistency regularization, we utilize temporal ensem- bling (Laine & Aila 2017), which penalizes the difference between the current prediction and the aggregation of previous predictions for each training input. In this way, the consistency under data augmentation is implicitly regularized since the augmentation is randomly sampled in each epoch. And it is also efficient as no extra model evaluation is required for an input.
To scale the consistency loss, a loss weight is often adjusted based on Gaussian ramp-up curve in previous works (Laine & Aila 2017). However, the specific parameterization of such a Gaussian curve varies greatly across different implementations, where more than one additional hyperparameters have to be set up and tuned heuristically. Here to avoid tedious hyperparameter tuning we simply linearly interpolate the weight from 0 to its maximum value, namely (tR 4xTR where T is the total number of epochs.
Summary. To recap, our teacher training method introduces two additional regularization terms. The loss function can thus be defined as 8= fstand. + ALRLLR + ACR&CR, where fSund. is the standard empirical risk defined in [T and ÅrR is the weight for Lipschitz regularization. Our method is simple to implement and incurs only minimal computation overhead (see Section 5.2)
5 EXPERIMENTS
In this section, we evaluate the effectiveness of our teacher training method in knowledge distillation. We focus on compressing a large network to smaller one where the student is trained on the same data set as the teacher, which is well-supported by our theoretical understanding.
5.1 EXPERIMENT SETUP
We denote our method as orrennt teacher training (SoTeacher), since it aims to learn the true label distribution to improve student performance, rather than to maximize teacher perfor- mance. We compare our method with the standard practice for teacher training knowledge distil- lation (Standard) (i.e., Eq. (I). We conduct experiments benchmark datasets including CIFAR- 100 (Kizhevsky 2009) and Imn--ImaeeNet [Tin] 2017). a miniature version of ImageNet (Deng al. 2009) consisting of 100, 000 training samples selected from 200 classes. Unfortunately, due to computational constraints, we were not able to conduct experiments on ImageNet. We experiment with various backbone networks including ResNet (He et al., 2016), Wide ResNet (Zagoruyko & Komodakis 2016) and ShuffleNet (Zhang et al. 2018b Tan et al. 2019). We test the applicability of SoTeacher from different aspects of model compression know ledge distillation including reduction of the width or depth, and distillation between heterogeneous neural architectures.
For knowledge distillation algorithms, we experiment with the original knowledge distillation method (KD) (Hinton et al. 2013), and a wide variety of other sophisticated knowledge distil- lation algorithms (see Sect. 5.2). We report the classification accuracies on the test set of both teacher and the student distilled from it. All results are presented with mean and standard deviation based on 3 independent runs. For Imag-em we also report the top-5 accuracy. For hyperparameters, we set Aaax 1 for both datasets. For Lipschitz regularization, we set ÅIR 10-5 for CIFAR-100 and ALR = 10 6 for ImageNet, as the regularization term is the sum of the Lipschitz constants of trainable parameters, and the teacher model for Tiny-ImageNe has approximately 10 times more parameters than that for CIFAR- -000 More detailed hyperparameter settings for neural network training and knowledge distillation algorithms can be found in Appendix
5.2 RESULTS
End-to-end Knowledge Distillation Performance. Tables [ and [ show the evaluation results on CIFAR-100 and ImageNn, respectively. Our teacher training method SoTeacher improve

---

Under review as a conference paper at ICLR 2023
Table 1: Test accuracy of the teacher and student with knowledge distillation conducted on CIFAR- 100. SoTeacher achieves better student accuracy than Standard for various architectures, depsite lower teacher accuracy.
WRNNU-2/WRN4011
WRN40-2/WRNIG-2
ResNer32x44Shuffeeeee22
Student
Teacher
Student
Teacher
Student
Teacher
Standard SoTeacher
73.73 0.13
76.38 0.13 74.95 0.28
74.87 0.45
76.38 + 0.13
74.86 + 0.18
79.22 = 0.03
74.35 +: 0.23
75.39 + 0.23
74.95 0.28
77.24 + 0.09
78.49 +: 0.09
no-CR
74.34 0.11
74.30 + 0.12
75.20 0.24
74.30 + 0.12
76.52 + 0.52
77.73 0.17
no-LR
73.81 + 0.15
76.71 + 0.16
75.21 0.13
76.71 + 0.16
76.23 + 0.18
80.01 + 0.18
Table 2: Test accuracy of the teacher and student with knowledge distillation conducted on Tiny- ImageNet. The teacher network is ResNet34 and the student network is ResNet18. SoTeacher achieves better student accuracy than Standard with lower teacher Top-1 accuracy.
Student (Top-1)
Student (Top-5)
Teacher (Top-1)
Teacher (Top-5)
Standard
66.19 1 0.17
85.74 1 0.21
64.94 0.32
84.33 1 0.40
Sofeacher
66.83 + 0.20
86.19 = 0.22
64.88 + 0.48
84.91 = 0.41
no-CR
66.39
0.27
86.05 + 0.17
64.36
0.43
84.10 0.27
no-LR
66.48 0.43
86.20 0.40
64.26 1.54
84.48 0.84
the student's 's test accuracy consistently across different datasets and teacher/student architecture pairs. Note that the success of our teacher training method is not due to high accuracy of the teacher. In Tables [ [nd one may already notice that regularization method will hurt the accuracy of the teacher, despite that it can improve the accuracy of the student distilled from it.
Table 3: Estimation of the uncertainty quality of the teacher network trained by Standard and SoTeacher. The uncertainty quality is estimated by the ECE and NLL both before and after tempera- ture scaling (TS).
Dataset
Teacher
Method
BCE
NLL
ECE (w/TS)
NLL (w/ TS)
Standard
0.113 + 0.003
1.047 + 0.007
0.028 + 0.004
0.905 + 0.008
CIFAR-100
WRN40-2
SoTeacher
0.057 # 0.002
0.911 = 0.013
0.016 = 0.003
0.876 = 0.012
Standard
0.083 0.003
0.871 1 0.010
0.036 0.001
0.815 1 0.008
CIFAR-100
ResNet32x4
SoTeacher
0.037 + 0.001
0.777 + 0.006
0.021 + 0.001
0.764 + 0.003
Standard
0.107 = 0.007
1.601 = 0.037
0.043 = 0.002
1.574 = 0.015
Tiny-ImageNet
ResNet34
SoTeacher
0.070 + 0.007
1.496 = 0.031
0.028 = 0.002
1.505 = 0.033
To further interpret the success of our teacher training method, we show that our regularization can indeed improve the approximation of the true label distribution thus benefiting the student generalization. Directly measuring the quality of the true distribution approximation is infeasible as the true distribution is unknown for realistic datasets. Follow previous works (Menon et al. 2021), we instead estimate the approximation quality by reporting the Expected Calibration Error (ECE) Guo et al. 2017) and NLL loss of the teacher on a holdout set with one-hot labels. Since scaling the teacher predictions in knowledge distillation can improve the uncertainty quality (Menon et al., 2021). J.
we also report ECE and NLL after temperature scaling, where the optimal temperature is located on an additional holdout set (Guo et al. (017). As shown Table 3 our teacher training method can consistently improve the approximation quality for different datasets and teacher architectures.
Ablation Study. We toggle off the Lipschitz regularization (LR) or consistency regularization (CR) in SoTeacher (denoted as no-LR and no-CR. respectively) to explore their individual effects. As shown in Tables [ and 2 LR and CR can both improve the performance individually. But on average, SoTeacher achieves the best performance when combining both LR and CR, also demonstrated in our theoretical analyses. Note that in Table 2] using Lipschitz regularization is not particularly effective because the regularization weight might not be properly tuned (see Figure 2)
Effect of Hyperparameters. We conduct additional experiments on Imag-et as an example to study the effect of two regularization terms introduced by our teacher training method. For
7

---

Under review as a conference paper at ICLR 2023
Lipschitz regularization, modulate the regularization weight ALR. For consistency regularization, we try different maximum regularization weights >CR and different weight schedulers including linear, cosine, cyclic, and piecewise curves. Detailed descriptions of these schedulers can be found in Appendix [ As shown in Figure 2 both Lipschitz and consistency regularizations can benefit the teacher training in terms of the student generalization consistently for different hyperparameter settings. This demonstrates that our regularizations are not sensitive to hyperparameter selection. Note that the hyperparameter chosen to report the results in Tables and Land2mmigh might not be optimal since we didn't 't perform extensive hyperparameter search in fear of overfitting small datasets. It is thus possible to further boost the performance by careful hyperparameter tuning.
In particular, Figure a shows that, as Lipschitz regularization becomes stronger, the teacher accuracy constantly decreases while the student accuracy increases and converges. This demonstrates that excessively strong Lipschitz regularization hurts the performance of neural network training, but it can help student generalization in the knowledge distillation context.
[a]
(B]
[C
Test Acc. (%6) (Top
66.5
Test Test Acc. (9)) [Top-
Student
Test Acc. (%) (Top-1)
67
68.5
66.0
Student
66-
Student
66.0
66
68
Teacher
66-
64-
Tea cher
Teacher
0.0
0.5
1.0
1.5
Standard
Uniar
Cooine
Cycle
Piccewise
Aue
10-5
AGe
Acn Scheduler
Figure 2: Effect of varying the hyperparameters in teacher training method, including the weight for Lipschitz regularization ALR. the weight for consistency regularization ACR. and its scheduler. The settings of our method used report the results (e.g. Table 2 are denoted as "A" The standard teacher training practice is denoted as for comparison.
Other Knowledge Distillation Algorithms. Besides the original knowledge distillation algorithm, experiment with various feature distillation algorithms including FitNets (Romero et al.. 2013)).
AT (Zagoruyko ko & Komodakis 2017). SP (Tung & Mori 2019), CC (Peng et al. 2019), VID (Ahn
al.
2019), RKD (Park al. 2019), PKT (Passalis & Tetas 2018), AB (Heo al. 2019). 9). FT (Kim
al.
2018).
NST (Huang& & Wang 2017). CRD (Tian et al. (0200) and SSKD (Xu et al. 2020). For the
implementation of these algorithiss, we refer to existing repositories for knowledge distillation (Tin
et al. 2020 Shah et al.] 2020 Matsubara] 2021) and author-provided codes. Although these distillation
algorithms match all types of features instead of predictions between teacher and student, they will achieve the best distillation performance when combined with the prediction distillation (i.e. original KD). Therefore, our teacher training method should still benefit the effectiveness of these distillation algorithms. We also experiment with curriculum distillation algorithm RCO (Jin et al. 2019) which distills from multiple checkpoints in teacher's s training trajectory. Our teacher training method should also benefit RCO as the later teacher checkpoints become more student-oriented As shown in Table 6 our SoTeacher can boost the distillation performance of almost all these distillation algorithms, demonstrating its wide applicability.
Student fidelity. Recent works have underlined the importance of student fidelity in knowledge distillation, namely the ability of the student to match teacher predictions (Stanton et al., 2021). Student fidelity can be viewed as a measure of knowledge distillation effectiveness that is orthogonal student generalization, as the student is often unable to match the teacher predictions although its accuracy on unseen data improves (Furlanello et al. 2018 Mobahi et al. 2020). Here we measure the student fidelity by the average agreement between the student and teacher's top-1 predicted labels on the test set. As shown in Table our teacher training method can consistently and significantly improve the student fidelity for different datasets and teacher-student pairs, which aligns with the improvement of student generalization shown by Table and This demonstrates that the teacher can better transfer its "knowledge" to the student with our training method.
Training overhead. Compared to standard teacher training, the computation overhead of SoTeacher is mainly due to the calculation of the Lipschitz constant, which is efficient as it only requires simple arithmetic calculations of the trainable weights of neural network (see Section 4 Empirically we observe that training with SoTeacher is only slightly longer than the standard traning for about 5%. The memory overhead of SoTeacher is incurred by buffering an average prediction for each
8

---

Under review as a conference paper at ICLR 2023
Table Average agreement (%) between the student and teacher's top-1 predictions on the test set.
CIFAR-100
Tiny- IntgeNet ResNet34/ResNet18
WRN-40-2/WRN 40-1
WRN-40-2/WRN-16:
ResNet22x44Shufileeee22
Standard SoTeacher
76.16 77.92 = 0.27
76.92 + 0.29
76.63 +: 0.25
71.33 +: 0.07
79.41 = 0.11
80.36 0.13
73.36 + 0.25
input. However, since such prediction requires no gradient calculation we can simply store it in a memory-mapped file.
6 RELATED WORK
Alleviate "teacher overfitting". Since in knowledge distillation the distillation data is often reused from the teacher's training data, teacher trained toward convergence is very likely to overfit its soft predictions on the distillation data. Intuitively, it is possible to tackle this problem by early stopping the teacher training (Cho & Haribaran 2019). However, a meticulous hyperparameter search may be required since the epoch number to find the best checkpoint is often sensitive to the specific training setting such as the learning rate schedule. It is also possible to save multiple early teacher checkpoints for the student to be distilled from sequentially (Jin et al., 2019). Additionally, one can utilize "cross-fitee "cross-fittingg procedure to prevent teacher from memorizing the training data. Namely, the training data first partitioned into several folds, where the teacher predictions on each fold are generated by the teacher trained only on out-of-foll data (Dao et al., 2021). However, this may depart from the standard teacher training practice and induce significant computation overhead. One can also train the teacher network jointly with student's 's network blocks, which imposes a regularization toward the student performance (Park et al., (20TT). However, this may also induce significant computation overhead as every unique student architecture will require retraining the teacher. For multi-generation knowledge distillation specifically (Furlanello et al., 2018), one can penalize the difference between the predictive probability at the true class and the probabilities at other semantically relevant classes to encourage the learning of secondary probabilities in teacher's ' prediction. which is shown to improve the performance of models in later generations (Yang et al. 2019). Different from these attempts, we train the teacher to directly learn the true label distribution of its training data, leading to a simple and practical student-oriented teacher training framework with minimum computation overhead.
Uncertainty learning on unseen data. Since the objective of our student-orientes teacher training learn label distributions of the training data, it is related to those methods aiming learn quality uncertainty on the unseen data. We consider those methods that are feasible for large teacher network training. including (1) classic approaches overcome overfitting such as l and &g regularization, (2) modern regularizations such as label smoothing (Szegedy et al. 2016) and data augmentations such as mixup (Zhang et al.) 2018a) and Augmix (Hendrycks et al. (2200), (3) Post-training methods such as temperature scaling (Guo et al. 2017), as well as (4) methods that incorporate uncertainty as learning ojective such as confidence-awazt learning (CRL) (Moon et al. 2020))
We have conducted experiments on CIFAR-100 using all these methods and the results can be found in Appendix F Unfortunately, the performance of these regularization methods is unsatisfactory in knowledge distillation - only CRL can slightly outperform the standard training. We believe the reasons might be two-folds. First, most existing criteria for uncertainty quality on the unseen data such calibration error Nacini et al., 2015) ranking error (Geifman et al. 2019). only require the model to output an uncertainty estimate that is correlated with the probability of prediction errors. Such criteria may not be translated into the approximation error to the true label distribution. Second, even if model learns true label distribution on unseen data, it does not necessarily have to learn true label distribution on the training data, as deep neural networks tend to memorize the training data.
7 CONCLUSION AND FUTURE WORK
In this work, we rigorously studied the feasibility to learn the true label distribution of the training data under a standard empirical risk minimization framework. We also explore possible improvements of current teacher training that facilitate such learning. In the future, plan to adapt our theory to other knowledge distillation scenarios such as transfer learning and mixture of experts, and explore more effective student-oriented teacher network training methods.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Tiny imagenet, 2017. URL
Sungsoo Ahn, Shell Xu Hu, Andreas C. Damianou, Neil D. Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. CVPR, pp. 9155-9163, 2019.
Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normaliin margin bounds for neural networks. NIPS, 2017.
David Berthelot, Nicholas Carlini, Ekin Dogus Cubuk, Alexey Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. ArXiv, abs/191 1.0985, 2019.
Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78:1-3, 1950.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizzl. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. ICCV, pp- 4793-4801, 2019.
Tri Dao, Govinda M. Kamath, Vasilis Syrgkanis, and Lester W. Mackey. Knowledge distillation as semiparametric inference. ArXiv, abs/2104.009733 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Chengyu Dong, Liyuan Liu, and Jingbo Shang. Double descent in adversarial training: An implicit label noise perspective. ArXiv, abs/21 10.03135, 2021.
Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandku- mar. Born again neural networks. ICMI., 2018.
Yonatan Geifman, Guy Uziel, and Ran EI- Yaniv. Bias-reduced uncertainty estimation for deep neural classifiers. In ICLR, 2019.
Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102:359 378, 2007.
Henry Gouk, Fibe Frank, Bemhard Pfahringer, and Michael J. Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 100:393-416, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. ArXiv, abs/1700.04599, 2017.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pp. 770-778, 2016.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Laksh- minarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. ArXiv, abs/1912.02781,, 2020.
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In AAAI, 2019.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv, abs/1 2015.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. ArXiv, abs//107.000 707.01229, 2017.
Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, and Xiaolin Hu. Knowledge distillation via route constrained optimization. ICCV, pp. 1345-1354, 2019.
10

---

Under review as a conference paper at ICLR 2023
Jangho Kim, Seonguk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. ArXiv, abs/1802.04977 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. ArXiv,
abs/1610.02242, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, 2017.
Junhao Liu, Linjun Shou, Jian Pei, Ming Gong, Min Yang, and Daxin Jiang. Cross-lingual machine reading comprehension with language branch knowledge distillation. In COLING, 2020.
Yoshitomo Matsubara. torchdistill: A modular, configuration-driven framework for knowledge distillation. In International Workshop on Reproducible Research in Pattern Recognition, pp. 24 44. Springer, 2021.
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, Seungycon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In ICML, 2021.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. ArXiv, abs/l 802.05902.5557 2018.
Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-3361, 2020.
Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. Confidence-aware learning for deep neural networks. In ICML, 2020.
Rafael Müllrr, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In NeurlPS, 2019.
Mahdi Pakdaman Nacini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. Proceedings of the ... AAAI Conference Artificial Intelligence. AAAI Conference on Artificial Intelligence, 0015:2901-2907 2015.
Radford M. Neal. Pattern recognition and machine learning. Technometrics, 49:366 -616666-66,,0 2007.
Dae Young Park, Moonsu Cha, Changwook Jeong, Daesin Kim, and Bohyung Han. Learning student-friendly teacher networks for knowledge distillation. ArXiv, abs/2102.00650, 2021.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. CVPR, pp- 3962-3971, 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In ECCV, 2018.
Baoyun Peng, Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu. Yu Liu, Dongsheng Li, and Zhaoning Zhang. Correlation congruence for knowledge distillation. ICCV, pp. 5006-5015, 2019.
Jian Qian, Ronan Fruit, Matteo Pirotta, and Alessandro Lazaric. Concentration inequalities for multinoulli random variables. ArXiv, abs/2001. 2020.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. CoRR, abs/14 141/,650,,55 2015.
Het Shah. Avishree Khare, Neelay Shah, and Khizir Siddiqui. Kd-lib: A pytorch library for knowledge distillation, pruning and quantization, 2020.
Kihyuk Sohn, David Berthelot, Chun-Liang Li. Zizhao Zhang. Nicholas Carlini, Ekin Dogus Cubuk, Alexey Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. ArXiv, his/2001.07663 2020.
11

---

Under review as a conference paper at ICLR 2023
Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel R. D. Rodrigues. Robust large margin deep neural networks. IEEE Transactions on Signal Processing, 65:4265 42200,4202000 2017.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 34: 6906-6919, 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. CVPR, pp. 2818-2826, 2016.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V. Le. Mnasnet: Platform-aware neural architecture search for mobile. CVPR, pp. 2815-2823, 2019.
Jiaxi Tang and Ke Wang. Ranking distillation: Learning compact ranking models with high perfor- mance for recommender system. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018.
Sunil Thulasidasan, Gopinath Chennupati, Jeff A. Bilmes, Tanmoy Bhattacharya, and Sarah Ellen Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In NeurlPS, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. ArXiv, abs/1910.10699, 2020.
Frederick Tung and Greg Mori. Similarity-preseree knowledge distillation. ICCV, pp. 1365-1374, 2019.
Qifan Wang, Li Yang, Bhargav Kanagal, Sumit K. Sanghai, D. Sivakumar, Bin Shu, Zac Yu, and Jonathan L. Elsas. Learning to extract attribute value from product via question answering: A multi- task approach. Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, 2020.
Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdú, and Marcelo J. Weinberger. Inequalities for the 11 deviation of the empirical distribution. 2003.
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, and Quoe Le. Unsupervised data augmentation for consistency training. arXiv: Learning, 2020.
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- supervision. In ECCV, 2020.
Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan Loddon Yuille. Training deep neural networks in generations: A more tolerant teacher educates better students. In AAAI, 2019.
Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. Model compression with two-stage multi-teacher knowledge distillation for web question answering system. Proceedings of the 13th International Conference Web Search and Data Mining, 2020.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. ArXiv, abs/1705. 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. ArXiv, abs/ 060166611666 2016.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. ArXiv, abs/1612.03928,, 2017.
Hongyi Zhang, Moustapha Cissé, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ArXiv, abs/1710.09412, 2018a.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. CVPR, pp. 6848-6856 2018b.
Yuping Zhang, Xiaoran Xu, Hanning Zhou, and Yan Zhang, Distilling structured knowledge into embeddings for explainable and accurate recommendation. Proceedings of the 13th International Conference Web Search and Data Mining, 2020.
12

---

Under review as a conference paper at ICLR 2023
A PROOF OF THEOREM 3.6
Step Bound the difference between the true label distribution and the sample mean. First note that the set of all training inputs can be grouped into several subsets such that the inputs in each subset possess similar true label distribution.
More formally, Let C = { be an r-external covering of with minimum cardinality, namely
S
U (z' ||
x|| S sr).
#EC
where we refer Fy the covering center, and N, is the covering number.
Let {Sj) N be any disjoint partition of S such that Sj S { - syyl S r). We show that Sj attains a property that the true label distribution p" (x) of any input z in this subset will be too far from the sample mean of one-hot labels y, = | Enesy ly in this subset. Namely given any 0, we have
|
(((lp(xx) 8,ll 8) S 2 exp
2
(8)
2K
where E max(s 2L*r,
To prove this property we first present two lemmas. Lemma A.1 (Lipschitz constraint of the true label distribution). Let Sj be a subset constructed above and y; = |S71 Enes, ly Then for any E ST we have,
(1) E < 2L'r.
(9)
Proof. First, since x E Sy, we have ||x - æzll S r, which implies IIp" p" (z,)) S L by the
locally Lipschitz continuity of p". Then for any 2, x' E Sj, we will have || - p' (x')l < 2 by the triangle inequality. Let Ns = 1S7l- Therefore,
1
<2 No - L < 2L'r.
p' ('(x)
p' #EST
(10)
NS
Ns
Further, the linearity of the expectation implies
E Ns -1
Σ
E[1 E[1y()) Ns -1 p'))..
(11)
#EST
#ES;
Therefore llp" (x)
E y s
2L.
Lemma A.2 (Concentration inequality of the sample mean). Let S be set of with cardinality N. Lety y = N-1 Eves ly be the sample mean. Then for any p-norm and any E >> 0, we have
N
P P E y E) 2exp
(12)
2K
Proof. Note that ý obeys multinomial distribution, i.e. y N MNltinoonnii We utilize the classic results on the concentration properties of multinomial distribution based on lj norm /Weismman al. 2003 Qian al. 2020), namely
N 2K
P(ly E [>]
e) < 2 exp
(13)
Further, for any E K by the equivalence of p-norm we have |||||
< ||q/11- Therefore,
{q
||||| 2 $} C {q
|/]/|= 2 $}
(14)
which implies,
P (| 2 e) < ([a/l 2 c).
(15)
Combine inequality 13 and [15 then have
12
n
13

---

Under review as a conference paper at ICLR 2023
One that Lemma A.1 bounds the difference between true label distribution of individual inputs and the mean true label distribution, while Lemma A.2 bounds the difference between the sample mean and the mean true label distribution. Therefore the difference between the true label distribution and the sample mean is also bounded, since by the triangle inequality we have
1p'(() yll
(x)
y
ll
E[y]ll-
(16)
This implies
P ((Ip (x) yl 8) < - 28),
(17)
where E max(s - -2Lrr, Apply Lemma A.2 we then have 8
Step II: Bound the difference between the prediction given by the empirical risk minimizer and the sample mean. Here show that given the locally Lipschitz constraint established in each disjoint partition we constructed above, the prediction given by the empirical risk minimizer will be close to the sample mean. As an example, we focus on the negative log- lileeiihohodd loss, namely ((f(x). y) = - -1y log f(r). Other loss functions that are subject to the proper scoring rule can be investigated in similar manner.
First, we regroup the sum in based on the partition constructed above, namely
N Σ R((,S)) R((,S) = N 1=1
(18)
where = = 1g log f(;;) is the empirical risk in each partition. Since we are only concerned with the existence of desired minimizer of the empirical risk, we can view f as able achieve any labeling of the training inputs that suffices the local Lipschitz constraint. Thus the empirical risk minimization is equivalent to the minimization of the empirical risk in each partition. The problem can thus defined as, for each j 1, Nr.
min ST) f
(19)
s.t. [[f(r) f(z,)]I < Lr, VIe
where the constraint is imposed by the Docally-Lipschit continuity of f. By the following lemma, we show that the minimizer of such problem is achieved only if fāz) is close to the sample mean.
Lemma A.3. Let y = |S3] Emes; Ly- The minimum of the problem (19) is achieved only if f(T,) = (1 + KLr) Lr.
Proof. We note that since the loss function choose is strongly convex, to minimize the em- pirical risk, the prediction of any input æ must be as close to the one-hot labeling as possible. Therefore the problem (19) can be formulated into vector minimization where employ Karush- Karush-Kuhn--a -ucker (KKT) theorem to find the necessary conditions the minimizer.
We rephrase the problem 19
min (p.)
N Σ y log p:
(20)
s.t. p: pll sc Ipt=1, -ptEpp11 1, 0, 2 0.
Case I. We first discuss the case when x + < 1 for all k. First, we observe that for any p. the minimum of the above problem is achieved only if p.xh = p + + Because by contradiction, log(p" + E), and p a belongs to the feasible set,
D < dh + E, we will have - log P which means P. does not attain the minimum.
The above problem can then be rephrased as
Σ
log(p" th + +E), stt. Σp¹ = =1 1, 20,
min
(21)
N
14

---

Under review as a conference paper at ICLR 2023
where we have neglected the condition associated with P; k since they do not contribute to the objective, they can be chosen arbitrarily as long as the constraints are sufficed, and clearly the constraints are underdetermined.
Let N₂ Σ, 1y: = k), we have log(D + = Ex N₂ log(p* + +E). Therefore the above problem is equivalent to
min P
Σ 5t log(p* +5), s.t. = 1, 2 0,
(22)
where y = [N//N, .. .NE/NJ equal to sample mean N-1E, 1ny
To solve the strongly convex minimization problem (2 it is easy to employ KKT conditions to show that
p = y(1 Kε) -.
Case II. We now discuss the case when p is the minimizer of (20) and there exists such that 8K + 1. And P + p", where p* y +
Kε) - is the form of the minimizer in the previous (1+
Considering a non-trivisn con-trivial case p <1 E. Otherwise the true label distribution already close to the one-hot labeling, which is the minimizer the empirical risk. Therefore by Exyxe p** e we have the condition
KE
Σ k
(23)
+ K
Now considering the minimization objective R(p) = - -N--NN Σ; 1, -oog.. P;- For all with y, = k', we must have p." 1. otherwise the optimal cannot attained by contradiction. Then the minimization problem can be rephrased as
min Σ 5"h loo(pp + E), s.t. Σxr
pt 2 2
pp 2 0,
(24)
where the first constraint is imposed by pK 2 1 -
Employ KKT conditions similarly we can have p² 3hh /A- where À a constant. By checking the constraint we can derive À 2 E. 5 (KE).
However, the minimization objective
min tth log gk X Righ
requires to be minimized. Therefore
Enyre 8h (Ke), which implies
yk
n² Ke
(25)
Enette yk
Now since 6- arg min, R(p) and + p", we must have R(D) R(p"). This means
Key
Σ ke
log
Σ kgk
logly (1 Ks)],
(26)
Enyre 8th
which is reduced
Ke
Σ 3d
(27)
+ Ke
Righte
But this is contradict assumption.
15

---

Under review as a conference paper at ICLR 2023
We are now be able to bound the difference between the predictions of the training inputs produced by the empirical risk minimizer and the sample mean in each Sj. To see that we have for each x E Sj.
[[((;)) 3,11 S [[f(x)) f(z,))l + ||(())) -55l
S Ir(1 K|ly, 1
(28)
I Lr(1+ <l] K 11])
By Equation (8) we then have for any Sj
| E2
((]f(z)) - p" (x)) 2 é) 1 2 exp
(29)
2K
which means the difference between the predictions and the true label distribution is also bounded. Here E = max(s - Lr(1 + () - K 11l) r,,r,
In 29 we have managed to bound the difference between the predictions yielded by an empirical risk minimizer and the true label distribution based on the cardinality of the subset (sy], namely the number of inputs in j-partition. However [Sy] is critical to the bound here if |S = 1. then (29) becomes trivial bound. Here show |s]| is non-negligible based simple combinatories.
Step III: Show the disjoint partition is non-trivial.
Lemma A.A. Let {Sj} 1 be a disjoint partition of the entire training set S. Denote (x) as the partition that includes x. Let N N(x) = (z:) and N = |S]. Then for any k. 2 1,
N KNY 2 1
x [
N.
(30)
K
Proof. We note that the problem is to show the minimum number of x such that N(x) 2 N/(NNT). This is equivalent find the maximum number of such that (x)) < N/(nNT). Since only have N" subsets, the maximum can be attained only if for N 1 subsets S S = N (/(N""). Otherwise, always feasible let |ST| (*"
if for any one of these subsets |ST| N N/(NN"), then and the maximum increases. Similarly, if the number of such subsets is less than N" - 1, then it is always feasible let another subset subject to S"| (=(NV" and the maximum increases. We can then conclude that at most N(N - 1)/(k²") imputs can have the property N(x) - (N(NN").
The above lemma basically implies when partitioning N inputs into N subsets, a large fraction of the inputs will be assigned to a subset with cardinality at least (xNN)..) Here N is the covering number and is bounded above based on the property of the covering in the Euclidean space. Apply Lemma A.4 to (29) we then arrive at Theorem 3.6
Difference from the Lipschitz bound in generalization theory. We note that in Theorem 3.6 the constraint on the local Lipschitz continuity of the hypothesis function is more strict than that is required in typical generalization theory of deep neural networks (Sokolic al. 2017 Bartlett et al.. 2017).
Remark A.5. generalization theory we often require the hypothesis function to locally Lipschitz continuous around training inputs, such that the difference between the prediction of unseen imput and the prediction of its nearest training input is bounded. However, for learning true label distribution of the truining data need to bound the difference between the predictions of subset of training inputs. Therefore, our constraint requires the Lipschitz continuity to be established at a larger scale in the input space.
Indeed, we observe from later experiments that our stronger Lipschitz constraint will actually hurt the performance of the teacher on the test set, whereas it can still improve the student performance.
B LIMITATIONS
In this paper we focus on the theoretical feasibility of learning the true label distribution of training examples with empirical risk minimization. Therefore only analyze the existence of such
16

---

Under review as a conference paper at ICLR 2023
desired minimizer, but neglect the optimization process to achieve it. By explore the optimization towards true label distribution, potentially more dynamics can be found to inspire new regularization techniques.
Also, our proposed method for training a student-oriente teacher may not be able to advance the state-of-the-ard significantly, as the regularization techniques based inspired by analyses such as Lipschitz regularization and consistency regularization may more or less leveraged by existing training practice of deep neural networks, either implicitly or explicitly.
C IMPLEMENTATION DETAILS
Lipschitz regularization. Following previous practice using Lipschitz regularization for gener- alization on unseen data (Yoshida & Miyato 2017) or stabilizing generative model (Miyato et al.) 2018), we regularize the Lipschitz constant of a network by constraining the Lipschitz constant of each trainable component. The regularization term is thus defined as LLR = E, Lip(f), where denotes a trainable component in the network f. The Lipschitz constant of network compo- nent Lip(f) induced by a norm || is the smallest value L such that for any input features h, h', (h) - fh''ll S L|h W/ Here we adopt the Lipschitz constant induced by 1-norm, since its calculation is accurate, simple and efficient. For calculating the Lipschitz constants of common trainable components deep neural networks, we refer to (Gouk et al. 2021) for a comprehensive study.
We design our consistency regularization term as &CR N E. - fxx)) where we follow previous work (Laine & Aila 2017) and employ MSE to penalize the difference. Here f(x) is the aggregated prediction of an input æ, which we calculate as the simple average of previous predictions f(z); = E (#)e, where we omit the data augmentation operator for simplicity. At epoch we simply skip the consistency regularization. Note
Consistency regularization.
1
that such a prediction average can be implemented in an online manner thus there is no need to store every previous prediction of an input.
D DETAILS OF EXPERIMENT SETTING
D.1 HYPERPARAMETER SETTING FOR TEACHER NETWORK TRAINING
For all the experiments on CIFAR-100, 100, we employ SGD as the optimizer and train for 240 epochs with a batch size of 64. The learning rate is initialized at 0.05 and decayed by factor of 10 the epochs 150, 180 and 210, with an exception for ShuffleNet where the learning rate is initialized at 0.01 following existing practice (Tin et al. 2020 Park et al., (02T). The weight decay and momentum are fixed as 0.0005 and 0.9 respectively. The training images are augmented with random cropping and random horizontal flipping with a probability of 0.5.
For Tiny-ImageNN experiments, we employ SGD as the optimizer and conduct the teacher training for 90 epochs with batch size of 128. The learning rate starts at 0.1 and decayed by a factor of 10 fixed 0.0005 and 0.9 respectively. The
at epochs 30 and 60. The weight decay and momentum training images are augmented with random rotation with a maximum degree of 20, random cropping and random horizontal flipping with probability of 0.5. For student training the only difference is that we train for additional 10 epochs, with one more learning rate decay at epoch 90, aligned with previous settings (Tian et al. 2020).
For consistency regularization in our teacher training method, we experiment with various weight schedules besides the linear schedule mentioned in the main paper. We list the formulas for these schedules in the following. Here denotes the epoch number, 7 denotes the total number of epochs, and max denotes the maximum weight.
Cosine schedule:
(CR(1) = cos
(1-4
t
π
max ACR
T
2
17

---

Under review as a conference paper at ICLR 2023
Table 5: for different feature distillation algorithms
8
8
Standard
SoTeacher
Standard
SoTeacher
Standard
SoTcacher
PitNets
100
50
AT
1000
500
SP
3000
1500
CC
0.02
0.01
VID
1.0
0.5
RKD
1.0
0.5
PKT
30000
15000
AB
1.0
0,5
FT
200
100
NST
50
25
CRD
0.8
0.5
Cyclic schedule:
Acn(t) - --11--1 (1 ))
X
CR
Piecewise schedule:
0,
0 O<tST/3,
(cn(t)=
CR /2, CR
</3<35
2T/3 <T.T.
D.2 HYPERPARAMETER SETTING FOR KNOWLEDGE DISTILLATION ALGORITIMSS
For knowledge distillation algorithms refer to the setting in RepDistiller Specifically, for original KD, the loss function used for student training is defined as
Cress Entropy + a))
We grid search the best apper-parameters that achieve the optimal performance, namely the loss scaling ratio a is set as 0.5 and the temperature set as 4 for both CIFAR- 100 and Tiny-ImageNer For all feature distillation methods combined with KD the loss function can be summarized (Tian) et al. 2020)
EatoossEanopy alKD 36
where we grid search the optimal y and to be 1.0 and 1.0 respectively. When using our teacher training method, all these byperparameters are kept same except that for all feature distillation algorithms the scaling weights corresponding the feature distillation losses 3 are cut by half, wish to rely more on the original KD that is well supported by our theoretical understanding. Table list used in our experiments for all feature distillation algorithms. For SSKD (Xu et al. 2020) the hyperparameters are set as A 1.0, 22 = 1.0, À3 2.7, A = 10.0 for standard training and Å1 = 1.0, X 1.0, λ3 1.0, b 10.0 for our methods. For the curriculum distillation algorithm RCO we experiment based on one-stage EEI (equal epoch interval). We select 24 anchor points (or equivalently every epochs) from the teacher's 's saved checkpoints.
E ADDITIONAL EXPERIMENT RESULTS
F EXPERIMENTS WITH UNCERTAINTY REGULARIZATION METHODS ON UNSEEN DATA
conduct experiments on CIFAR-100 with teacher-student pair WRN40- 2/WRN40-1. We employ the original KD as the distillation algorithm. The hyperparameter settings are the same as those mentioned in the main results (see Appendix DD. For each regularization method grid search the hyperparameter that yields the best student performance. The results summarized in Table
Experiment setup.
We observe that with stronger & & regularization the student performance will not deteriorate significantly as teacher converges. However, it also greatly reduces the performance of the teacher. Subsequently the performance of the student is not improved as shown in Table
Classic regularization.
https::/ http://ttthhuccccooob///bbbb
18

---

Under review as a conference paper at ICLR 2023
Table 6: SoTeacher consistently outperforms Standard on CIFAR-100 with various KD algorithms.
VRN40-2/WRN40-1
WRN40-2/WRNI622
ResNer32x4/Shuffte22
Standard
SoTeacher
Standard
SoTeacker
Standard
SoTeacher
FitNet
74.06 + 0.20
74.88 + 0.15
75.42 + 0.38
75.64 + 0.20
76.56 0.15
77.91 + 0.21
AT
73.78 +
75.12 = 0.17
75.45
+ 0.28
75.88 + 0.09
76.20 = 0.16
77.93 + 0.15
SP
0.25
74.71 + 0.19
74.67
+ 0.37
75.94
0.19
78.06 1 0.34
CC
0.12
74.76 + 0.16
0.16
75.08 1 0.07
75.67 1
7.14
77.68 + 0.28
VID
0.30
74.89 - 0.19
00,11:0.000
7.17
0.11
77.37 0.16
RKD
74.66 0.44
0.08
0.08
75.16 + 0.21
75.59
0.12
77.46 0.10
PKT
74.89 0.44
75.45 + 0.09
75.53
77.84 75.72
AB
74.86 1 0.10
70.09 + 0.66
70.38 +
76.27
78.05 0.21
FT
0.15
74.75 + 0.13
75.19
+ 0.15
75.68 +
76.1
77.56
NST
+0.4
74.74 + 0.14
74.95 + 0.23
75.68 + 0.16
77.71+ 0.10
CRD
7..4
75.06
0.37
75.52
+ 0.12
75.95 -.05
0.12
78.09 0.13
SSKD
0.22
7
76.31
+ 0.07
76.32
:: 0.09
78.4
79.37 0.11
RCO 0.32
74.81 = 0.04
75.24 0.34
75.50 + 0.12
76.75 0.13
77.59 + 0.31
Table Performance of the knowledge distillation when training the teacher using existing regular- ization methods for learning quality uncertainty unseen data.
WRN40-2/WRN4011
Student
Teacher
Standard SoTeacher
73.73 0.13
76.38 0.13
74.35 + 0.23
74.95 0.28
&2 (5 10
73.73 + 0.13
76.38 = 0.13
6 (10
73.60 1 0.15
73.52 1 0.05
Mixup (a = 0.2)
73.19 0.21
77.30 + 0.20
Cutmix (a: 0.2)
73.61 0.26
78.42 = 0.07
Augmix (a = 1, 3)
73.83 1: 0.09
77.80
1 0.30
CRL (A=
74.13 0.29
76.69 - 0.16
Label smoothing. Label smoothing is shown to not only improve the performance but also the uncertainty estimates of deep neural networks (Müller et al.) 2019). However, existing works have already shown that label smoothing can hurt the effectiveness of knowledge distillation (Müller et al.) 2019), thus neglect the results here. An intuitive explanation is that label smoothing encourages the representations of samples to lie in equally separated clusters, thus "erasing" the information encoding possible secondary classes in sample (Müller al., 2019).
Previous works have demonstrated that mixup-like data augmentation techniques can greatly improve the uncertainty estimation unseen data (Thulasidasan al. 2019 Hendrycks et al. 2020). For example, Mixup augmented the training samples as : o. + (1 a )ar and y = ay same class as
Data augmentation.
a)y', where (æ', v/) is randomly drawn pair not necessarily belonging the
As shown Table 7 stronger mixup can improve the performance of the teacher, whereas it can barely improve or even hurt the performance of the student. Based on our theoretical understanding (Theorem 3.8). we conjecture the reason might be that mixup distorts the true label distribution of an input stochastically throughout the training, thus hampering the learning of true label distribution.
Temperature scaling. Previous works have suggested using the uncertainty on validation set to tune the temperature for knowledge distillation either in standard learning (Menon et al., (202T) or robust learning (Dong et al. (202T). However, the optimal temperature may not be well aligned with that selected based uncertainty (Menon et al. 202T). We neglect the experiment results here the distillation temperature in our experiments is already fine-tuned.
Uncertainty learning, CRL designs the loss function & = bCE + MeRRL, where leE is the cross-entropy loss and &CRL is an additional regularization term bearing the form of
&CRL. max (0, p((()))) -((x)) + lc(ma) c(x;))),
(31)
19

---

Under review as a conference paper at ICLR 2023
where p(z) = maxę f(z) is the maximum probability of model's prediction training sample and
dx)
(arg max f(x) - y)
P=1
is the frequency of correct predictions through the training up to the current epoch. Here g(c, cj) = 1 c > oj and g(s, oÿ) -1 otherwise. Although originally proposed to improve the uncertainty quality of deep neural networks in terms of ranking, we found that CRL with proper hyperparameter can improve the distillation performance, as shown in Table 7
We note that the effectiveness of CRL distillation can be interpreted by our theoretical understand- ing, as its regularization term (31) is essentially a special form of consistency regularization. To see this first notice (31) is margin loss penalizing the difference between p(æ) and c(x² in terms of ranking. We then rewrite c(x) as
c(x) = 1y
Onehot (c)))
(32)
which is similar to our consistency regularization target, except the prediction is first converted into one-hot encoding. Such variation may be the best for knowledge distillation we wish those secondary class probabilities in the prediction be aligned as well.
20


Reviews:
[{"review_id": "nN-6AW16h_S", "reviewer": "Reviewer_3FzY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission focuses on training teacher models with the goal of making them successful in knowledge distillation. The first part of the paper presents necessary conditions under which a teacher model minimizing the empirical loss computed with one-hot labels captures the Bayes classifier probabilities on the training set, thus providing better supervision in knowledge distillation. These theoretical observations indicate that it is desirable to have the teacher network be locally Lipschitz around the training examples and have consistent outputs under data augmentations of a single example. Based on these findings the authors propose a method of training a teacher model that includes Lipschitz regularization and consistency regularization. They show that this way of training produces teacher models that distill better students in various knowledge distillation procedures. The proposed method is also compared with some standard ways of teacher regularization schemes designed to prevent overfitting.", "summary_of_the_review": "In summary, the quality and clarity of the theoretical parts of the paper need to be improved significantly. This is the main determinant of my score.", "strength_and_weaknesses": "#### **Strengths**\n\nThis submission tackles an important and well-motivated problem. They do extensive experiments to show the benefit of the proposed method. While I have some concerns about the validation of the results (see below), I find the experimental section mostly convincing. Most details for reproducing the work are presented. The related work is adequately cited. To my best knowledge the proposed approach is novel in the context of knowledge distillation.\n\n#### **Weakness 1: the theoretical part of the paper is not rigorous enough**\nIn particular, some parts seem to be wrong; some non-trivial proofs are missing; some parts of existing proofs are hard to follow; and there are extra assumptions being made in the proofs that are not listed in the statements.\n\n1. I understand that Sec. 3.1 serves as a warmup, but the definition 3.1 of siblings is not realistic. For real-world datasets, even with full knowledge of $p^*(x)$ the set of siblings for most examples will probably consist of only that example.\n2. The proof of proposition 3.3 should not be omitted. I am not sure that the arguments presented for the negative log-likelihood loss hold for other proper scoring rules. Why should the optimal value of $f^*$ be achieved at the empirical average of one-hot labels for other loss functions? For example, if we take $\\ell(p, q) = \\max_{i \\in [K]} |p_i - q_i|$ and consider a siblings set containing 3 examples with one-hot labels $\\\\{[0,1], [1,0], [1,0]\\\\}$, then it is easy to verify that the optimal value of $f^*$ will be $[1, 0]$, not $[2/3, 1/3]$.\n3. On the main theorem (Thm. 3.6)  \n    a) Is it realistic to expect that $N^r$ is not of the scale of $N$ for real-world datasets? It seems that most examples in such datasets would be far from each other in the input space (measured by a p-norm).    \n    b) The theorem statement is for general loss functions that are proper scoring rules, but the proof is done for the negative log-likelihood. The authors mention that other loss functions can be investigated in a similar manner, but I am not sure about it. The Lemma A.3 relies on the form of the loss function heavily. Furthermore, as in the case of Proposition 3.3 the empirical loss minimizer in a neighborhood doesn’t need to be close to the empirical average of one-hot labels.    \n    c) Eq. (18) is not correct – an average of averages is not equal to the global average. It will be true if the partition subsets are of equal size.   \n    d) “Since we are only concerned with the existence of a desired minimizer of the empirical risk, we can view $f$ as able to achieve any labeling of the training inputs that suffices the local Lipschitz constraint”. Here the authors assume that the function class $\\mathcal{F}$ is so large that empirical losses on different neighborhoods can be optimized separately. For this to happen one probably needs extra assumptions on $\\mathcal{F}$. For example, this is not true for the set of linear functions.    \n    e) The proof of Lemma A.3 is hard to follow. I was not able to verify it. For example in the case I, should not *all* dimensions of $p^k$ be specified in order to verify whether $||p - p^k|| \\le \\epsilon$ holds or not. In general, do authors claim that the minimum of problem (19) is achieved at the same point no matter what p-norm is used in the constraints? I don’t see how this happens.\n\n4. The proof of Thm 3.8 is omitted. It seems that the proof will be different as $f^*$ is minimizing the empirical risk on augmented examples. It is also not clear whether $N^r$ refers to the covering number with augmented neighborhoods or not.\n\n#### **Weakness 2: validity of the results**\nTraining a teacher with early stopping and tuning the knowledge distillation temperature are two standard approaches that help alleviate overfitting the one-hot training labels. This paper does not compare the proposed method with early stopped teachers, justifying that the latter needs tedious hyperparameter tuning. I suggest comparing with teachers trained for 50%, 60%, …, or 100% of epochs. \n\nThe temperature parameter in knowledge distillation usually has a strong effect on the student's performance. It would be great to see if proper tuning of the temperature parameter can bring the same benefit as the proposed method.\n\n#### **Minor comments**\n* “And $\\lVert \\cdot \\rVert$ is a distance function induced by an arbitrary p-norm”. This makes the paper unclear. I suggest picking a concrete p-norm early in the paper.\n* Lipschitz regularization is similar to bounding gradient with respect to inputs x. This kind of regularization was explored in the context of training neural networks that are robust to small input changes (see for example [1]).\n* Def 3.2: in equation (1) $\\ell$ is defined on $(f(x), y)$ pairs, while in most of the text $\\ell$ is defined on a pair of probability distributions over the classes.\n* Notations like $x \\in \\mathcal{S}_{p^*}(x)$ should be avoided.\n* Definition 3.5: change to “We say *a set of inputs* $\\mathcal{C}$ is a … “.\n* $N^r$ can be confused with taking the r-th power of N. You can use $N_r$ instead. \n* In Def. 3.7  $\\mathcal{A}$ is a set rather than a function.\n* In Thm. 3.8, the maximums over a possibly not compact sets should be replaced with supremums.\n* $[N] = \\\\{1,2,\\ldots,N\\\\}$, rather than $[N] \\equiv 1,2,...,N$.\n* In the proof of Thm 4.6, $\\mathcal{S}$ is not defined. I assume it refers to the set of training inputs.\n\n\n#### **References**\n\n[1] Hoffman J, Roberts DA, Yaida S. Robust learning with Jacobian regularization. arXiv preprint arXiv:1908.02729. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S7Bp6RjC1je", "reviewer": "Reviewer_FGnr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is motivated by recent findings in the field of knowledge distillation that the teacher should learn the true label distribution on the inputs rather than the one-hot labels with the best performance. The authors propose to take advantage of the Lipschitz regularization and consistency regularization to train teachers that are better for distillation. The method is evidenced both theoretically and empirically in the image classification setting. ", "summary_of_the_review": "The paper is in good shape with both theoretical and empirical evidence for their proposed method. Moreover, the results of experiments on CIFAR-100 and TinyImageNet are sound to support the main contributions of the paper. However, it is a bit weak from the experiment side due to the lack of experiments on the large-scale ImageNet. ", "strength_and_weaknesses": "**Strengths**\n+ The paper is well-written and easy to follow.\n+ The motivation is clearly explained by theoretical analysis and examples.\n+ The scope of the paper is novel and interesting that improving the training of teachers in the context of knowledge distillation can further boost the performance of multiple KD approaches. \n+ The method are well demonstrated from both theoretical and empirical perspectives.\n+ Comprehensive experiments and ablation studies with promising results well support the main claims and contributions of the paper.\n\n\n**Weaknesses**\n\nHowever, there are still some concerns to be addressed:\n\n- That would be interesting to explore the influence of the proposed regularizers on reducing the overfitting of different teachers. How do they affect different teachers for the same student with KD methods?  \n\n- Table 6 looks to belong to the ablation study, however, it is shown in the appendix as \"As shown in Table 6\" in the main text. It is a bit confusing for readers.  \n\n- The lack of experiments on the large-scale dataset ImageNet and current transformer architectures weaken the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EFPO3jPmwK", "reviewer": "Reviewer_gTte", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is concerned with the following question: \"what sort of teacher should we fit so that a resulting student trained via distillation from this teacher has good performance\". Following recent work on the utility of distillation, which in particular observes that distillation benefits a student by providing a soft signal that may closely approximate the regreression function (i.e. the conditional law of the label $y$ given the input $x,$ henceforth denoted as $p^*(x)$), the paper adopts two directions: firstly, to argue that this regression function can indeed be learnt, and secondly, to investigate how one may improve the learning of the underlying law. \n\nTo this end, the authors first theoretically explore the recovery of the regression function, specifically under the assumption that $p^*$ is locally Lipschitz at a given scale, and argue that the ERM over such a class recovers the the true $p^*$ up to limited error at every datapoint in the training set. Secondly, the authors propose to use existing Lipschitz and Consistency regularised training methods to recover a teacher in a procedure they term \"SoTeacher\". It is empirically argued that while this results in lower teacher accuracy, the accuracy of the corresponding student model increases.", "summary_of_the_review": "On the whole, I find this to be a rather poor submission - while the problem being considered is interesting, the method proposed is overall incremental, and the justification presented for it is entirely insufficient. I would strongly recommend rejection.", "strength_and_weaknesses": "I think that the question of what teacher one should use for distillation is an interesting one, and would likely be of interest to the ICLR community (especially if the answer is different from 'the most accurate teacher'). However, I don't think that the authors meaningfully address this question.\n\nI'll split this critique into two parts\n\nSection 3: In my opinion, the theoretical study in this paper has limited novelty, and limited relevance.\n\nFor the former, I note that recovery of the regression function is well established in nonparametric statistics under smoothness assumptions as studied by this paper - for instance see Tsybakov's  work from the mid 2000s (e.g. Optimal Aggregation of Classifiers, Ann Stat 2004, and references therein). This literature establishes concrete rates for the recovery of Lipschitz regression functions (and subsequent work further studies adaptivity to this parameter, e.g. some of the work of Kpotufe and coauthors). The interested reader will also find that the (minimax optimal) rates with sample size of this nonparametric problem (without further noise assumptions) are _much_ poorer than the $1/\\sqrt{n}$ that appears in the paper. What gives? \n\nThis brings me to the relevance part: in my opinion, the results shown do not actually demonstrate that $p^*$ is learnable. Indeed, the issue of precisely how $r$ affects the data is not dealt with at all. This is crucial, since $r$ appears in the bias term of the error bound (as the $Lr$ and $L^* r$ terms) - this means that if $r$ is very large, then the result is vacuous. However, if $r$ is very small, then the results is similarly vacuous due to the high scaling of $N_r$ with $r$! Indeed, observe that in high dimensions, the volume of an $r$ ball for $r \\ll 1$ is extremely small. So, for instance, if the data is drawn uniformly over a unit ball in $d$-dimensions, the chance of finding even one point within a distance $r$ of a given datapoint is $O(r^{-d})$, exponentially small in the dimension. This means that for reasonably small $r$, $N^r \\approx N$ so long as $N \\ll r^{-d},$ and so the result is just a vacuous bound. \n\nThe above consideration must also be coupled with the fact that Neural Nets, which this theory is meant to apply to, do not have a meaningfully small Lipschitz constant (whatever the metric being used) - the results of Bartlett et al that are cited within the paper only show that the Lipschitz constant is well correlated with the generalisation error (not that the Lipschitz constant of the network is small). \n\nGiven this, I find Section 3 to both be unoriginal and irrelevant. In my opinion this is already a very serious flaw with the paper.\n\nSection 4: Orthogonally to the above, we may ask if the empirical study of the paper is meaningful. In my opinion, this also does not pass muster. There's two aspects to this\n\n- I think the paper does a poor job of demonstrating that the improvements in the student performance come from improvement in the fitting of $p^*$: Given the focus of the rest of the paper, the implicit statement is that the regularisation of the network allows it to learn the regression function better, and hence improves the student. However, there are obvious alternate hypotheses for this: for instance, it could be the case that the regularisation results in a smoothened teacher, and this reduction in teacher complexity makes it easier for a student to fit (or improve upon) the teacher. There are at least two tests for a hypothesis like this - first to support their claim, the authors need to run simulation studies which actually demonstrate that the teacher recovers the regression function better when regularised. Second, they should train the student with a post-hoc smoothened teacher (so train a teacher normally, and then do some local smoothing to generate soft labels) - if the improvement in student performance is similar, then the improvement cannot entirely be attributed to fitting $p^*$ better. \n\n- The algorithmic contributions are minimal: Note that the concrete proposal of the paper is not a novel algorithm, but instead a novel application of existing methods (LR and CR) to the problem of choosing a teacher for distillation. While this is certainly interesting, I question if this is enough to merit publication in ICLR on its own strength - there are certainly a plethora of tricks associated with training DNNs for a variety of problems and domains that are not published in such a conference even though they yield improvements in the resulting models. Typically such a conference requires that the phenomenon underlying the method is well explained and clearly demonstrated, which this paper in my opinion fails to do.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8w4Qxr2lDFJ", "reviewer": "Reviewer_fjbL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the knowledge distillation in the context of multi-class classification, and this paper aims to train a good teacher for it by exploring the feasibility of training a teacher that is oriented toward student performance with an empirical risk minimisation framework. Inspired from recent findings that the good teacher is one whose capability to approximate the true label distribution. The paper investigates the feasibility by beginning with siblings theory, saying in a hypothetical case where the siblings of an input are known, the empirical minimiser can produce predictions close to the true label distribution on the training data. With this siblings,, the paper proves that if the loss function is properly selected (e.g., with proper scoring rules) and the network is properly regularised, the close approximation of the true label distribution can be achieved. To be practical, the work shows the siblings can be relaxed in terms of neighbourhoods with the assumption of L locally-Lipschitz continuous of true labels and  the constraints of the network has to be locally-Lipschitz continuous. When the augmentation is employed, with assumption that the true label distribution is nearly invariant under data augmentations, this neighbourhood is relaxed for augmented inputs with augmented neighbourhood. Based on theoretical analysis, the paper proposes the training teacher method incorporating Lipschitz regularisation and consistency regularisation to improve the empirical risk minimisation. ", "summary_of_the_review": "Overall, the paper is well-written and a nice theory with empirical support. The results are encouraging as being tested on some benchmark datasets and with a number of network baselines. So the proposed method mostly outperforms the standard distillation. Ablation study is also provided.\n", "strength_and_weaknesses": "Strengths:\n\n1. New study on feasibility of true label distribution.\n\n2. Theoretical analysis is supported with empirical results when proposed regularisation and consistency losses proposed improve the student performance which is consistent with theory. \n\n3. Paper is well-written. \n\nWeaknesses:\n\n1. It would be good to detail the regularization and consistency losses $l_{LR}$ and $l_{CR}$. \n\n2. Is Fig. 1 with total loss (including  $l_{LR}$ and $l_{CR}$) or only the original loss? I am also interested in how the Lipschitz regularisation loss and consistency loss change over-time when training SoTeacher? Beside the theory, it is also interesting to understand empirically how these proposed losses influence on the learning of the teacher (e.g., reduce overfitting)?\n\n3. Looking at Fig. 1, it is interesting that if the standard teacher is well early stop (e.g., epoch 150), the student performance can be also very comparable, and the training of both teacher and distillation be reduced a lot? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject