Efficient Knowledge Distillation from Model Checkpoints
Authors: Chaofei Wang
Qisen Yang
Rui Huang
Shiji Song
Gao Huang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep learning
image classification
intermediate model
knowledge distillation
mutual information.

Abstract: Knowledge distillation is an effective approach to learn compact models (students) with the supervision of large and strong models (teachers). As empirically there exists a strong correlation between the performance of teacher and student models, it is commonly believed that a high performing teacher is preferred. Consequently, practitioners tend to use a well trained network or an ensemble of them as the teacher. In this paper, we observe that an intermediate model, i.e., a checkpoint in the middle of the training procedure, often serves as a better teacher compared to the fully converged model, although the former has much lower accuracy. More surprisingly, a weak snapshot ensemble of several intermediate models from a same training trajectory can outperform a strong ensemble of independently trained and fully converged models, when they are used as teachers. We show that this phenomenon can be partially explained by the information bottleneck principle: the feature representations of intermediate models can have higher mutual information regarding the input, and thus contain more ``dark knowledge'' for effective distillation. We further propose an optimal intermediate teacher selection algorithm based on maximizing the total task-related mutual information. Experiments verify its effectiveness and applicability. Our code is available at https://github.com/LeapLabTHU/CheckpointKD.

Efficient Knowledge Distillation from Model Checkpoints
Chaofei Wang', Qisen Yang", Rui Huang, Shiji Song, Gao Huang" Department of Automation, Tsinghua University, China wangef18, yanggs19, hr20@mails. wtw.ilsssssssiisi.. tsinghua. edu en shijis, sachuangetsinghhnn enu.cn
Abstract
Knowledge distillation is an effective approach to learn compact models (students) with the supervision of large and strong models (teachers). As empirically there exists a strong correlation between the performance of teacher and student models, it is commonly believed that high performing teacher is preferred. Consequently, practitioners tend to use a well trained network or an ensemble of them as the teacher. In this paper, observe that intermediate model, i.e., checkpoint in the middle of the training procedure, often serves as a better teacher com- pared to the fully converged model, although the former has much lower accuracy. More surprisingly, a weak snapshot ensemble of several intermediate models from same training trajectory can outperform a strong ensemble of independently trained and fully converged models, when they are used as teachers. We show that this phenomenon can be partially explained by the information bottleneck principle: the feature representations of intermediate models can have higher mu- tual information regarding the input, and thus contain more "dark knowledge" for effective distillation. We further propose optimal intermediate teacher se- lection algorithm based on maximizing the total task-related mutual information. Experiments verify its effectiveness and applicability. Our code is available at https//gghhbb.ooommmccpp/.t/tttp
Introduction
Knowledge distillation (KD) has been proved to be an effective technique to promote the performance of low-capacity model by transferring "dark knowledge" from a large teacher model. Empirically, there usually exists strong correlation between the performance of the teacher model and the student model. For this reason, it is standard practice to use well trained network or an ensemble of multiple well trained networks as the teacher 3 55 and some researches are attempted to improve distillation performance via boosting the ensemble performance [6] 7). The underlying assumption is that high performing teachers lead to better student models.
However, this viewpoint has been challenged by some recent works [8] has been observed that large model capacity gap between the teacher and student may have negative effect for knowledge transfer. To address this issue, researchers have proposed to employ an intermediate-sion size network [8] or an assistant network [9] to improve the distillation performance in such scenarios. In [10], "tolerant" teacher model is designed by using a softened loss function. In [T], Park et al. proposed learn student-friendly teacher by plugging student branches during the training procedure. Nevertheless, there is no clear theoretical explanation for the gap between teacher and student, and the search for substitute teacher is not straightforward.
[2]. in which
Equal contribution Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Porformance
Performance
Full Teacher
KKnesle Mag Dior larion
Intermediute Teucher
Khow Khowlpagee Qustitatioo
Distilled Student
Student
Training epochs of teacher
Ensemble tencher
(a) Intermediate Teacher vs. Full Teacher
(b) Snapshot Ensemble vs. Full Fisembbe
Figure I: A sketch map of the two counterintuitive observations. (a) A weak intermediate model can serve as a better teacher than the strong fully converged model. (b) weak Snapshot Ensemble [3 can serve better teacher than the strong Full Ensemble.
In this paper, we make an intriguing observation that further supports the viewpoint that high performing models may not necessarily be good teachers, but from novel perspective. Specifically, we find that an unconverged intermediate model from the middle of the training procedure, often serves as better teacher than the final converged model, although the former has much lower accuracy (as illustrated Figure Πa)). Moreover, a weak snapshot ensemble of intermediate teacher models along the same optimization path (denoted as Snapshot Ensemble, which is variant of [3 can outperform the standard ensemble of an equal number of independently trained teacher models (denoted as Full Ensemble). This surprising phenomenon is illustrated in Figure [[),, in which Snapshot Ensemble can have better distillation performance than a Full Ensemble, although the accuracy of the former is significantly lower.
To understand the above phenomenon, we show that there is a strong connection between KD and the information bottleneck (IB) theory [14). Therein, it has been observed that during the training procedure of deep neural networks, the mutual information between the learned features F and the target Y, denoted as I(Y; F). increases monotonically as function of the training epochs; while the mutual information between F and the input X, denoted as [(X; F), grows the early training stage, but then decreases gradually after certain number of epochs. We note that maximizing the mutual information T(Y; F) is helpful for improving the teacher model itself, but not always necessary for KD, because the ground truth target Y is already included in the KD objective function. In contrast, the mutual information I (X; F), to some extent, can be viewed as a type of dark knowledge that is desired for effective KD. For example, considering may be uniquely labeled into the "car" category, it still contains features of the "people" category. Such weak but nogligglie features extracted from the input (measured by I(X;F) are in fact the most valuable knowledge for distilling student models. Not surprisingly, most KD algorithms apply a high temperature soften the network prediction in order reveal these information from teacher model. However, as shown by IB theory, a fully converged model tends to be overconfident and may already have collapsed representations for non-targeted classes. Therefore, simply scaling the temperature can not effectively recover the suppressed knowledge. On the contrary, an intermediate model, although does not reach its top accuracy due non-optimal (Y: F)., may have a larger [(X;)) that benefits KD. This partially explains our observation that intermediate models can be better teachers. More detailed and formal analyses are provided in the following sections.
image with man driving car, although it
Further, we propose an optimal intermediate teacher selection algorithm based on the IB theory. From the perspective of entropy, the teacher model's representation be decomposed the information with respect to the input, the output and some nuisance [15]. The proposed algorithm aims to find the most informative intermediate teacher model which possesses the minimal part of nuisance on a training trajectory. Experiments verify its applicability in various distillation scenarios. Our contributions are summarized as follows:
By designing two exploratory experiments, we observe the phenomenon that intermediate models can serve as better teachers than fully converged models. This suggests that for In this paper.we adopt normal cosine learning instead of the cyclic learning rate.

---

effective KD one should not only focus on improving the teacher performance. Instead,
rethinking what the "dark knowledge" is and how to enrich it highly valuable.
We demonstrate the connection between our observations and the IB theory, providing new perspective for understanding KD and explaining the "dark knowledge". Based on our observations and analyses, novel simple but effective algorithm is pro- posed to find the optimal intermediate teacher and achieve better distillation performance. Experiments validate its effectivvness and adaptability.
2 Related Work
Knowledge distillation. Hinton al. proposed transfer "dark knowledge" of strong capacity teacher model to compact student model by minimizing the
divergence between the soft targets of the two models. Since then, many variants of KD methods were proposed to improve the distillation performance[1o, such as Fitnets [77] AT[ CCM FSP[20], SP 21], promising technique to improve model generalization, ensemble learn- ing is often combined with knowledge distillation to improve the distillation performance 3 In the online knowledge distillation framework (24)))rrr__4_____ efforts were made boost the distillation perfor- mance by increasing the diversity between multiple models to improve the ensemble performance 7 Most existing methods commonly assumed that a high performing teacher is preferred for KD, On the contrary, some researchers thought that the model capacity gap between strong teachers and small students usually degrades knowledge transfer & 9
CC KD 23
25). Some them have experimentally 12
verified that poor teachers compress this gap by introducing an assistant network &8 9 or designing a fenenntfriendly teacher [T] However, they did not explain theoretically why gap exists and how gap affects KD. In the seff-distillation framework [26] 27 it is essentially the intermediate model that is used as the teacher, but has theoretical explanation for why the intermediate model works. In this paper, we link KD and IB theory through extensive observations and experiments. From the perspective of mutual information, we explain why intermediate models serve as better teachers than full models, and how to select an suitable intermediate model to reduce the negative impact of model gap.
also perform KD tasks well 12 25 Several methods proposed to
Information bottleneck. Tishby et al. [28 firstly proposed the information bottleneck concept and provided a tabular method to numerically solve the IB Lagrangian (Eq. (3). Later, Tishby and Zaslavsky [14] proposed to interpret deep learning with IB principle. Following this idea, Shwartz-Ziv and Tishby [29] studicd the IB principle to explain the training dynamics of deep networks. It has motivated many studies to apply the IB principle to interpret and improve deep neural networks (DNNs) 30 31 32). Recently, some researchers introduced IB principle to deep reinforcement learning successfully [33] 34 35). As far know, are the first to introduce IB principle to interpret knowledge distillation.
3 Exploratory Experiments
In this section, we first formally describe the KD and ensemble KD methods used in the paper, then design two exploratory experiments to show how intermediate models are surprisingly valuable for KD, despite their lower accuracies due to incompletion of training.
3.1 Formulation
In the classical KD setting, fully converged teacher model (full teacher for short) T'flll is used to distill student model S. Define Prem as the softmax output of teacher model, Ps as the softmax output of student model and Yiue as the true labels. The student model is trained to optimize the following loss function:
LKD = (Yme, Ps) + (1 -a) (PF
(I)
where H refers to the cross cross-entropy, a is the trade-off parameter, T is the temperature. Conducting KD with an intermediate teacher model Tinter means using Tiinter instead of T in Eq. (1.
In the standard ensemble KD setting, there are M(M 2 2) full teachers (Tf'll) T ..... The ). which have the same network structure and training strategy but different initial parameters. The student
3

---

model needs to mimic the average softened softmax output of all teacher models. We call this method Full Ensemble KD. The loss function follows:
M
LEKD a.H Ps) (1 o)
Pisu, PS).
(2)
The Snapshot Ensemble aggregates MM 2 2) intermediate teachers {Timeer TT ,... TMr) from one training trajectory. Conducting KD with a Snapshot Ensemble, means using Timerr instead of T in Eq. 22. We call it Snapshot Ensemble KD
3.2 Experimental design and setups
To examine the common assumption "high performing teachers lead to better student models" and explore the value of intermediate models, we design two experiments. 1) The standard KD train a full teacher model to distill a student model. What if we adopt the intermediate teacher models instead? 2) The standard ensemble KD is to train multiple full teacher models independently and average their output to distill a single student model. What if we adopt the Snapshot Ensemble instead of the Full Ensemble? We name the first experiment as "Intermediate Teacher vs. Full Teacher", and the second experiment as "Snapshot Ensemble vs. Full Ensemble". For generality, we conduct experiments on the CIFAR-100 (36), Tiny-ImageNet 37 and ImageNet (38] datasets with various teacher-student pairs. The distillation loss functions follow Eqs. ① and (2). For fair comparison, we search the optimal hyperparameters (i.e., the loss ratio a and the Temperature T for each teacher-student pair. Top 1 accuracy is averagely evaluated in five independent experiments.
The "Intermediate Teacher vs. Full Teacher" experiment is conducted on the CIFAR-100 and ImageNet. On CIFAR-100, we adopt WRN-40-21 ( [3] and ResNet- (00Net-110[40] as teacher models, WRN- 40-1 139. ResNet-332(40) and MobileNetV2 41 (width multiplier is 0.75) as student models. We train each teacher model for 200 epochs to ensure convergence. We save the intermediate models the 20th 40 180th epochs as intermediate teachers, and the models the 200th epoch as full teachers. On ImageNet, we adopt ResNet-50 [40], and ResNet-34 [40] as teacher models, and MobileNetV2 41. and Re [40] as student models. We follow standard PyTorch practice but train teacher models for 120 epochs to guarantee convergence. We save the intermediate models the thth epoch as intermediate teachers, and the models
120th epoch as full teachers. The "Snapshot Ensemble vs. Full Ensemble" experiment is conducted CIFAR-100 100 and Tiny-ImaeNNet. We train models for 150 epochs on Tiny-Imageh to ensure convergence. We save the intermediate models at the 75 epoch as intermediate teachers, and the models at the 150th epoch full teachers. We adopt WRN-40-1 (39,, 3esNNet-32[40] and MobilenetV2[4]] as student models, and WRN-40- 2[3),, and ResNet- 10 40 teacher models. Due to the page limitation, include the introduction of the datasets and detailed experimental settings in the Appendix A.1.
3.3 Intermediate Teacher vs. Full Teacher
Firstly, we simply compare the half-way teach- ers with the full teachers on CIFAR-100 100 and ImageNet. It means that the intermediate mod- els at the 1h0th epoch are adopted as Tinter on CIFAR-100, the intermediate models the t0th
Test accuracy student
epoch are adopted as Tinter on ImageNet. The training cost of all intermediate teachers is only
half that of the full teachers. Table [ shows
69
the comparison results. Specifically, on CIFAR- 100, for WRN-40-2, the accuracy of the inter- mediate model is 13.54% lower than that of the full model, but its distillation performance is comparable (0.08% higher) and superior (0.96% higher). For ResNet- 10, the accuracy of the intermediate model is 13.98% lower than that of the full model, but its distillation performance still comparable (0.01% % higher and 161 higher). On ImageNet, the accuracy of the inter-
68
WRN 26 40-1
67
ResNett
WNNN-10-2/Mbblleeeee 2/000////oblleleee Nety2
66
Restlet 110MoaiieNeeV2
65
20
40
60
80
100
120
140
160
180
200
Training epochs of teacher
Figure Ablation experiments of the training epochs of Tinee CIFAR-100.

---

Table 1: Comparison results of "Full Teacher vs. Intermediate Teacher". T represents a full teacher. Timer represents a half-way intermediate teacher whose KD performance is underlined. Ti represents the best intermediate teacher whose KD performance is bold-faced. The numbers in brackets represent decreases (4) or increases (f) in accuracy.
Network structure
Accuracy reference
Accuracy
Daraset
7
7
Takr
T
WRN-40-1
70.38
72.68-00.10
72.76==0.24 (f (.08)
73.26+00003 (1 0.58)
WRN-10-2
76.53
62.99 (L 13.54)
MobileNNetV2
64.49
68.03:00344
68.99=0.12 0.96
68.99:0.122 (f 0.96)
CIFAR-100
ResNet-32
70.16
72.48
72.49::0.32 (1 0.01)
72.63:0.13 0.15)
ResNet- 110
73.41
59.43 (L 13,98)
MobileNerV2
64.49
68 08.63:0.355
68.79=0.177 0.16)
69.05=0.27 (1 0.42)
ResNet-34
ResNet-18
73.86
66.08 (L 7.78)
69.75
70.66
70.86 (1 0.20)
70.98 (1 0.32)
ImageNet
Re
MobileNevV2
76.93
69.14 () 7.79)
64.18
66.64
66.79 (t 0.15)
66.92 (1 0.28)
Table 2: Comparison results of "Full Ensemble vs. Snapshot Ensemble". EN1 denotes +2 EN₂ denotes TT ENg denotes + The best results of ensemble KD performance are bold-faced, while the best results of ensemble performance are underlined.
Network structure
Aocumcy of haseline
Accuracy of ensemble KD
Accurccy of ensemble
Dataset
T
KD
EN;
ENg
EN₂
EN:
EN₂
ENs
WRN-10-1
70.38
72.68:10.10
73,0520.16
73.80:0.31
73.70:0.04
WRN-10-2
76.53
79.44
76.90
76.29
MobileNet22
64.49
68 68.03:0034
68.6920.52
19.1470
69.20:0031
ResNet-32
72.48:10.22
72.91
73.03-:0.24
11sNet.110
73.41
76.92
74.28
73.23
MobileNetV2
64.49
08.63:10.35
69
69/6990:225
70.46:10.34
70.19::0.25
ImageemaeeNet
WRN-40-2
WRN-40-1
57.61
53.85
55.66
$5.96
56.27
56.37
62.55
61.53
60.33
MohilNNrV2
53.75
56.53
56.67
57.05
$7.28
Average
79.32
74.11
76.04
66.11
66.61
66.63
73.06
70.90
69.95
mediate teachers is about 7.8% lower than that of the full teachers, but their distillation performance is still comparable or even better.
To further explore the potential of the intermediate models, we conduct ablation experiments of the training epochs of Tinter on CIFAR- 100. Figure 2 shows some valuable information. 1) Good and bad teachers both improve the baseline of student model, which is consistent with (25] 2) The peaks of all curves are not the last points, which means that there is always an intermediate teacher that is better than the full teacher. Table I (the last column) shows the KD performance of the best Tinter 3) Generally, the accuracy curve of student first rises then decreases along with the teacher's s training epochs. Combing Table and and Figure 2 we find counterintuitive observation:
Observation The distillation performance of an intermediate teacher model can comparable with or even better than that of the fully converged teacher model, although the accuracy and truining cost of the former is significantly lower.
3.4 Snapshot Ensemble vs. Full Ensemble
First, we fix the ensemble size to to avoid introducing other factors. We train two full teacher models, Ifulll and fullll with 200 epochs independently, and save their intermediate models, 1a and Trparr at the thhth epoch. For the Full Ensemble, we construct an ensemble with T:fall and Zi For the Snapshot Ensemble, we build ensemble with one full teacher model Tfalll and its intermediate model Timtrr We use Tffill + Thaill and Tinter + T to represent the Full Ensemble and the Snapshot Ensemble, respectively. To further explore the intermediate and full models, we add an additional evaluation object, an ensemble with one intermediate model Timeer and one extra full teacher model Thull from another training trajectory. Such combination is represented as 7 Timter + T!olll
Table 2 shows two observations: 1) For the ensemble performance, Tfull + Thulll Timeer + T Trflll pinter is consistently true; 2) For the distillation performance, Tiitter + T-ulll and Timter + T:fill
5

---

have similar performance, but significantly outperform 71 + T In average, the accuracy of Full Ensembles is 3.11% higher than that of Snapshot Ensembles, but the distillation accuracy of the former is 0.52% lower the later. Comprehensively considering training cost and distillation performance, the Snapshot Ensemble (7f + T is the best choice.
To further verify this cognition, we conduct an ablation experiment of the ensemble size k CIFAR-100, We adopt WRN-40-2/WRN-40-- as the teacher-student pair. The Full Ensem- ble is composed of k full teacher models, while composed of full teacher model and its k - 1 intermediate mod- els. For simplicity, the training process full teacher model is averagely divided into k phases to obtain the evaluate the test accuracy of distilled students and the total training cost, which consists of training teachers and distilling the student. Fig- ure shows two phenomena. 1) The distillation performance of the Snapshot Ensemble is sig- nificantly better than that of the Full Ensemble, with the same ensemble size k. 2) The training cost of the Snapshot Ensemble is significantly lower than that of the Full Ensemble, with the same ensemble size Hence, Snapshot Ensem- bles can be more economical and efficient in the KD setting. Combining Table 2 and Figure 3 obtain the second counterintuitive conclusion:
W WRN--0-22WRN-0002
napshat Ensemble
Full Ensemble
74.5
Test accuracyy af student
the Snapshot Ensemble
74
73.5
intermediate models. We
2
4
12
14
Total training cost
Figure 3: An ablation experiment of ensemble size
(k: = 1,2, 4, 7, 10) on CIFAR-100, We estimate the training hours TITAN Xp.
Observation 2. A stronger ensemble does not always lead to better distillation. In particular, the Snapshot Ensemble has worse ensemble performance and lower training cost but better distillation performance than the Full Ensemble.
4 Knowledge Distillation and Information Bottleneck
We have observed that the student's s accuracy first increases and then decreases while the teacher's accuracy increases monotonically, as a function of the training epoch of the teacher (as shown in Figure In this section, we aim to understand this phenomenon using the IB principle proposed in 14. The effect of mutual information on distillation performance is analyzed theoretically and experimentally, and further verified with class correlation information.
4.1 Connection knowledge distillation with information bottleneck
Tishby and Zaslavsky proposed that layered neural networks form a Markov chain of successive representations of the input, and explained the optimization goal of DNNs with the IB principle. They claimed that DNNs tend to obtain an efficient representation of the input, capturing the features relevant to the output and compressing those irrelevant. Formally, for DNN, define the input variable X and desired output variable Y. Any representation the input F, is defined through an encoder X) and decoder PF) The optimization goal of the network can be described as the following IB trade-off optimization problem [29]:
min F
(T(::))BB(PFF)))
where [(X; F) represents the mutual information between X and F, I[F; Y) represents the mutual information between F and Y, is a positive trade-off parameter. From perspective information theory, knowledge transfer can be expressed as retaining high mutual information between the teacher and student networks (proposed in VID 42].. Following Eq. (3). we define the representation the teacher model F,, the representation of the student model Fg. The optimization goal of the student model KD setting can be described as follows:
min ([(:FF)) 3I(Y; (FF:
(4)

---

W WRN-0011
T: ResNNe 110: S: ResNit-32
50
Teacher
Teacher
70
Student
74
Student
S 1he
50
5. T7
5 Thtee
5. T
10335
1(W9
30
20
0.55
5.40
0.42
0.44
0.41 0.22
(X.F)
(OK:F)
(b)
Figure 4: The mutual information curves on the information plane. The red lines represent teacher models. The dark blue * represents T while the light blue represents Timler The gray, dark blue and light blue lines represent students, distilled students with 7 and distilled students with Tineer The yellow areas are the observation areas. A full teacher compresses I(X; F)) of student, while an intermediate teacher slows down the compression (a) or even amplifies [(X; F)) of student (b).
where y is positive trade-off parameter. [F;; Fs) represents the mutual information between the teacher and the student. F, and F, can be quantified by their coordinates: (T(X; Fr), T(Y; Ft)) and (T(X; F.), T(Y: Fs)) on the information plane. Hence, maximizing T(F;; F.) can evolve into minimizing [T(;; F.) T(X; F.)l (T(Y; Ft) T(Y; F.).. Then, we can reformulate Eq. 4 to
min - (1( + (1X F)) [(X;
(5)
+ + IXY;
where how to remove the absolute value signs depends on the relative location of the coordinates (I(X; P.), I(Y; F.)) and (T(X; F.), T(;; F.)) on the mutual information plane.
4.2 Mutual Information Analysis
In order to explore the role of mutual information in knowledge distillation, we quantitatively analyze the mutual information curves of different teacher-student pairs. In practice, following [43], we adopt the test accuracy to quantify {(Y; F") and a reconstruction loss to quantify 1 [(X; F). To get the reconstruction loss, we connect decoder following the last convolution layer of the network model to reconstruct the input X. We adopt WRN-44022WRR-440- and ResNetl 10/ResNet32 pairs on 100. We train each teacher model for 200 epochs to get full model Thhll and save the intermediate model Timer at the 100th epoch. Detailed settings are given in Appendix B.I. The mutual information curves of teacher model, student model, distilled student models with Thill and Trinter are shown in Figure 4
Some important observations and inferences are summarized.
1) The trend of all curves is similar to that shown in 22) and consistent with IB principle [1]. That is, T(X; F) first goes up and then goes down while T(Y; F) goes up monotonically. It can be interpreted as: for [X;; F)), networks first absorb information of the input X and later eliminate part of the information irrelevant to the target output Y; for (Y;;)) F), networks continuously accumulate information relevant to the target output Y.
2) The curve of large teacher model is usually on the right of the curve of small student model. means a large model generally has greater ability of information representation.
3) The full model fulll compresses model Tinter That is to say, a full model may discard more information of non-target classes from X.
T(X; F) but retains more T(Y; F) than the intermediate
4) The student models distilled with T further compress T(X; (the dark blue curves are on the left of the gray curves in Figure 4). while those distilled with Trinter slow down the compression of [(X; F') (the light blue curve in Figure or even amplify I(X; (the light blue curve in Figure (b)). Retaining more (X;; F)) seems to key factor to get a competitive performance from Tiiter
For full teacher with large I(Y; but a small I(X; F.). Eq.
5 is reformulated to
min ((1 (X;( (8 (0++
(6)
7

---

[R] Logits, Epoch
(b) Logits; Epoch
[4] Softmaa
5j Spoch 100
[4] Softmak r- 5: Epoch 200
The heatmaps of class correlation information on the intermediate model (at the 100 epoch) and the full model (at the 200th epoch) on the CIFAR-100 100 dataset. (a) and (b) are calculated with the logits output: (c) and (d) are calculated with the softened softmax output (temperature 5). More brightly colored areas mean that the intermediate model has more diversity than the full model.
Figure 5:
For suitable intermediate teacher model with large F(); and large ((x;Fe) Eq. (5) is reformulated to
min ((1 -1--)))( (x; (B + I(Y;
Comparing Eq. (6) and Eq. 77. the difference is that full teacher model accelerates the compression of [(;; F.) while the intermediate model alleviates the compression of I(X;F) Fz). This is consistent with our 4th observation. Therefore, we can reasonably assume that: more mutual information with the input data can be the main reason why an intermediate teacher model achieves better distillation performance than a full teacher model.
Compared with full model, a suitable intermediate model has more [(X; and less 1(Y;; which means that the intermediate model has more non-target classes information. To do further analysis, we use heatmaps of cosine similarity to visualize the class correlation information contained in the intermediate model and the full model. For CIFAR- 100, a heatmap where each element A: represents the similarity between corresponding two classes. Specifically, we represent a class 6; by its average logits or softmax output on the test set (a 100-dimensional vector Ve,)) and then calculate the cosine similarity Aij between G and ej with the following equation:
100 100 matrix A,
(Ve, Vo,) VVgll
A.j VV=
Figure 5 shows some heatmaps of cosine similarity on the intermediate model and the full model. The network structure is WRN-40-2. Comparing Figure 5(a) with (b), we can see that the logits output of the intermediate model has better diversity than that the full model. Comparing Figure 5)c) with (d), while using softened softmax output with higher temperature can increase some of the diversity between classes, the intermediate model still shows a wider range of class correlation information than the full model. Results of the class correlation information further support that suitable intermediate teacher has more non-target classes information than a full teacher, which is consistent with the mutual information analysis.
4.3 Label smoothing regularization and knowledge distillation
The previous work 225 proposed that label smoothing regularization (LSR) can be considered as an ad-hoc KD with pre-defined uniform distribution teacher. Mathematically for LSR, uniform distribution u is used in place of P in Eq. (T). We attempt to explore the connection between LSR and KD from the IB perspective. Obviously, the uniform distribution u has small I [(Y; Fi) and small Ft). Eq. (5) is reformulated to
min {(1 + (X; (8 ();
where B - must be positive to keep the optimization direction correct. Comparing Eq. (6) and Eq. 99) if we fix the coefficient the second term I(Y:F, to be then the coefficient of the first term is (1 + (B + 7) in Eq. (6) and (1 + 7)/(3 y)) in Eq. (9) It means that LSR accelerates the compression of T(X; F.). In order to verify it visually, keeping the same setting as Figure we add LSR to the student model and show its mutual information curve on the information plane. As shown in F both the dark blue curve (the normal KD) and the yellow curve (LSR) are on the left of the gray curve (the student model). It means that both the normal KD and LSR play a similar role to
8

---

WAN 40 40.1
RetNet S: ResNet
20
76
66
50
50
40
Epoch
Teacher
Student
Student
ffal
5 1-1M
with LSR
5 with LSR
047
(OGFF)
Figure 6: The mutual information curves on the information plane. The red curves represent the teacher model. The gray, dark blue and light blue curves represent the students, distilled students with full teacher and distilled students with intermediate teacher. The yellow curves represent student training with LSR. The yellow areas are the observation areas.
compress the student model's 's mutual information with respect to input, which supports the view in [25] However, there a big gap between the dark blue curve and the yellow curve, which means that the normal KD transfers more relevant information with input to the student model than the LSR does. For the distillation performance, 7 inneer > 1 full > u is consistent with the amount of / [(X; Ft)
of the teacher models, which further supports our view in Sec. 4.2
5 Optimal Intermediate Model Selection
As shown in Figure 2 and Figure 4 how to cap- ture the optimal intermediate model is trivial problem. An empirical conclusion that good intermediate model needs delicate trade- off between T(X; F) and T(Y; F). From the point of information entropy, the representation F of model consists of information concerning
T: WRN-40-2
T
the input X, the output Y and nuisance Z which is irrelevant to the task [15). A formal descrip-
I(Y:F)
0,5
tion of the entropy of F is:
0.4
H(F) = ((;; F) + I(Y;F)+
(10)
Teacher
0.
Search Lines
To reduce the nuisance Z and get more infor- mative intermediate teacher model, we solve the following optimization problem:
0,0
02
0.5
035
I(X:))
max (1(;; F) + F));
(1I)
Figure 7: An example of selecting the optimal in- termediate teacher Tinter The red line represents the normalized mutual information curve of the teacher model (WRN-40-2). The blue lines repre- sent the selecting process. The represents the optimal intermediate teacher 7" whose representa- tion maximizes (I(X: F) + I)Y:
where F belongs the set of representations in intermediate teacher models. Figure iis an example of the selecting process. The optimal solution is usually located in the upper right corner of the information plane. Based on the selecting strategy, we propose algorithm to find the optimal intermediate teacher for effec- tive distillation shown in Algorithm [ Note that the value of mutual information F) and (Y;F) is normalized to ensure the scale matching. Obtaining the optimal intermediate teacher model, standard KD or other mainstream KD methods can be conducted usual.
To verify the superiority of the optimal intermediate model selection strategy, we conduct comparison experiments on the CIFAR- 100 dataset. We keep the same experimental settings as in Sec. B.3 but save teacher's checkpoints every 10 epochs. For each checkpoint, the information theoretic quantities T(X; F)) and 7(Y; F) are estimated as in Sec. 4.2 Then, normalize T(X; F) and T(Y; F), and draw the mutual information eurves. By algorithm the optimal intermediate teacher models 7 of WRN-40-2 and ResNet-110 10 selected the 160 and 120th epochs respectively. An

---

Algorithm Distillation with the optimal intermediate teacher.
1: Train teacher model T from scratch and save its checkpoint T; every M epochs;
2: Calculating the normalized [(X; F) and I(Y; F)) of each T;;
3: Draw the mutual information curve of T;
4: Find the optimal 7 which has the representation that maximizes FI I(Y;)))
5:
Take 7 as the teacher model to do distillation.
Table 3:
KD Results of the optimal intermediate models on CIFAR-100. The intermediate teacher models are selected different epochs. The best results are bold-faced.
Network structure
Accuracy of T&S
KD accuracy of different intermediate teachers
T
7
7
7
Thsll
WRN-40-1
70.38
72.34=0.10
72.76:+0,24
73.0%+00.05
72.68=0.10
73.26++0.03
WRN-40-2
76.53
MobileNetV2
64.49
68.21=0.33
68.9911.112
68.54:10.07
68.03=0.34
68.58:10.34
ResNet-32
70.16
70.74=0.18
72.49:+:0322
72.46+0.30
72.48=0.22
72.63:00.13
ResNet-110
73.41
MohileNetV2
64.49
67.84=0.26
68.79++00117
69.01++0.20
68.63=0.35
68.99++0.33
Average
74.97
67.38
69.78
70.76
70.77
70.46
70.87
empirical conclusion is that the optimal intermediate model for each teacher model may be different. To evaluate the distillation performance T" pick out models at the 60th 100t; 140th 200 epoch, which are denoted by 7 T T and T as baselines. We adopt the same standard KD method [2] but different intermediate models teachers. Table that 7 has the best average performance, which validates the effectiveness and adaptability of our selection strategy. In addition, 70.7 and 70.5 also have better average performance than T If you think that selecting the optimal intermediate model is tedious, the half-way checkpoint may be your prefer.
6 Conclusion and Limitations
In this paper, we made observation that an intermediate model can have richer "dark knowledge" than fully converged model, and employed the IB principle to partially interpret this phenomenon. We argue that over-training of the teacher model results in the suppression of class correlation information, leading to degradation of the distillation performance. As a result, training fully converged teacher may not be the optimal choice, especially under essource-limited circumstances. To save training cost, we empirically suggest that the half-way teacher model can suffice. To achieve better distillation, further proposed an optimal intermediate model selection algorithm to find the appropriate intermediate teacher. Furthermore, this work implies a more economical and efficient way to construct snapshot ensemble with several intermediate models from the same training trajectory instead of the standard ensemble with independently full-trained models. This technique can significantly improve the ensemble model's distillation performance and reduce the training cost.
Our study also has some limitations. First, the selection of an optimal intermediate model considers the information entropy of the teacher but ignores the variation of the student structures, which can not ensure the optimal KD performance for all teacher-student pairs. Second, how to choose the best intermediate teacher model for specific structure of student is still challenging problem.
Acknowledgement
This work supported in part by the National Key R&D Program of China under Grant 2020AAA0005000 the National Natural Science Foundation of China under Grants 62022048, THU-Bosch JCMI. and Beijing Academy of Artificial Intelligence. We also appreciate the generous donation of computing resources by High-Flyer AI.
10

---

References
[1] Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. KDD '06, 2006.
[2] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/1503.02531, 2015.
[3] Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born-again neural networks, In ICML, 2018.
[4] Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR, 2018.
[5] Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In KDD, 2017.
[6] Defang Chen, Jian-Ping Mei, Can Wang, Yan Feng, and Chun Chen. Online knowledge distillation with diverse peers. AAAI, 2020.
[7] Zheng Li, Ying Huang, Defang Chen, Tianren Luo, Ning Cai, and Zhigeng Pan. Online knowledge distillation via multi-branch diversity enhancement. In ACCV (4). 2020.
[8] Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In AAAI, 2020.
[9] Mengya Gao, Yujun Shen, Quanquan Li, and Chen Change Loy. Residual knowledge distillation. CoRR, abs/2002.09 168, 2020.
[10] Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan L Yuille. Training deep neural networks in generations: A more tolerant teacher educates better students. In AAAI, volume 33, pages 5628-5635, 2019.
[11] Dae Young Park, Moon-Hyun Cha, Dacsin Kim, Bohyung Han, et al. Learning student-rrendlly teacher networks for knowledge distillation. NeurIPS, 34, 2021.
[12] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV, pages 4794 4802, 2019.
[13] Gao Huang, Yixuan Li. Geoff Pleiss, Zhuang Liu, John E. Hoperoft, and Kilian Q. Weinberger. Snapshot ensembles: Train 1, get M for free. In ICLR (Poster), 2017.
[14] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In ITW, 2015.
[15] Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. In ITA, 2018
[16] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey, International Journal of Computer Vision, 129(6):189-1811) 2021.
[17] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR (Poster), 2015.
[18] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR (Poster), 2017.
[19] Chaofei Wang, Jiayu Xiao, Yizeng Han, Qisen Yang, Shiji Song, and Gao Huang. Towards learning spatially discriminative feature representations. In ICCV, pages 1326-1335, 2021.
[20] Junho Yim, Donggyu Joo, Ji-Hoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017.
[21] Frederick Tung and Greg Mori. Similarity preservvn knowledge distillation. In ICCV,

---

[22] Baoyun Peng, Xiao Jin, Dongsheng Li, Shunfeng Zhou, Yichao Wu, Jiaheng Liu, Zhaoning Zhang, and Yu Liu. Correlation congruence for knowledge distillation. In ICCV, 2019.
[23] Chaofei Wang, Ke Yang, Shaowei Zhang, Gao Huang, and Shiji Song. Tc3kd: Knowledge distillation via teacher-student cooperative curriculum customization. Neurocomputiing 508:284- 292. 2022.
[24] Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-fly native ensemble. In NeurlPS, 2018.
[25] Li Yuan, Francis E. H. Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisit knowledge distillation: a teacher-free framework. CoRR, abs/ 1999 1173,, 2019.
[26] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance convolutional neural networks via self distillation. In ICCV, pages 3713-3722 2019.
[27] Chenglin Yang, Lingxi Xie, Chi Su, and Alan L Yuille. Snapshot distillation: Teacher-student optimization in one generation. In CVPR, pages 2859-2868, 2019.
[28] Naftali Tishby, Fernando C. N. Pereira, and William Bialek. The information bottleneck method. CoRR, physics/0004057, 2000.
[29] Ravid Shwartz- and Naftali Tishby. Opening the black box of deep neural networks via information. CoRR, abs/ 10/1000300011100 10, 2017.
[30] Ziv Goldfeld, Ewout van den Berg, Kristjan H. Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy. Estimating information flow in deep neural networks. In ICML, 2019.
[31] Andrew M. Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D. Tracey, and David D. Cox. On the information bottleneck theory of deep learning. In ICLR (Poster), 2018.
[32] Roman Pogodin and Peter E. Latham. Kernelized information bottleneck leads to biologically plausible 3-factor hebbian learning in deep networks. In NeurIPS, 2020.
[33] Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Hugo Larochelle, Matthew Botvinick, Yoshua Bengio, and Sergey Levine. Infobot: Transfer and exploration via the information bottleneck. In ICLR (Poster), 2019.
[34] Xingyu Lu, Kimin Lee, Pieter Abbeel, and Stas Tiomkin. Dynamics generalization via informa- tion bottleneck in deep reinforcement learning. CoRR, 0bs/2000.006144 2020.
[35] Jiameng Fan and Wenchao Li. Robust deep reinforcement learning via multi-view information bottleneck. CoRR,
2021.
[36] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. University of Toronto, 2009.
[37] Tiny-imagenet. ht ht immpge/eeiiyymmmgmeeee....m www.-iiaaagn..... hheraaeeppppaeeeeeee......u.... www.huapo.oocooomo com/
[38] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[39] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[40] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[41] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861. 2017.
12

---

[42] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Varia- tional information distillation for knowledge transfer. CVPR, pages 9163-9171, 2019.
[43] Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang. Revisiting locally supervised learning: an alternative to end-to-end training. In ICLR, 2021.
[44] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In ICLR (Poster), 2014.
[45] Iya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In ICLR (Poster), 2017.
[46] Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang. Revisiting locally supervised learning: an alternative end-to-end training. In ICLR (Poster), 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [No] We could not foresce any potential negative societal impacts of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yss]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
(b) Did you include complete proofs of all theoretical results? [Yes] See Section
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? (Yes) We provide the URL of code for reproducing the main results.
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes] See Section 3.2 Section 4.2 Section and the Appendix.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] In the paper, top accuracy is averagely evaluated in five independent experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Figure and Figure ??.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 3.2
(b) Did you mention the license of the assets? [No] The models and the datasets are open source.
(c) Did you include any assets either in the supplemental material We provide the URL of code for reproducing the main results.
URL? Yes]]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [No] We only use open source datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] We only use open source datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "oYAYWWHmKuC", "reviewer": "Reviewer_encu", "summary": "This paper observes that the intermediate models (checkpoints in the middle of the training procedure) can serve as better teachers than fully trained teacher models in knowledge distillation (KD). \nMeanwhile, this paper proposes the Snapshot Ensemble, which ensembles several intermediate models and one full model, and observes that the snapshot ensemble has better KD performance and low training cost than the full teacher ensemble. \nFurthermore, this paper applies the Information Bottleneck (IB) theory to understand KD and “dark knowledge”. Moreover, this paper proposes an algorithm to find the optimal intermediate teacher based on IB theory. \n", "questions": "The idea of applying IB theory to KD is in general novel. Nevertheless, the above weaknesses refrain me from giving a better score. I would be happy to improve my rating as long as the authors address the above weaknesses in the rebuttal.   \n", "limitations": "Yes, this paper does mention some limitations in Section 6. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths.  \n- The idea of applying mutual information to understand KD and “dark knowledge” is very novel to the community.  \nThe curves of mutual information between the input and learned features clearly reveal the training process of a neural network and dark knowledge.  \n\n- Experiments are sufficient. \n\n- This paper is in general well written and easy to follow. \n\nWeaknesses\n\n- The findings of the intermediate checkpoints are not very exciting to the community.  \nMy most concern is about the novelty of the intermediate model part. This paper spends a lot of effort demonstrating the advantages of the intermediate checkpoints. However, previous works [36] already suggest that the Reversed KD (“student teach the teacher”) or even Defective KD (“poorly-trained teacher teach the student”) can achieve comparable results with normal KD. Thus, the observations on intermediate models in this paper should be naturally correct as well and not very exciting. Although this paper claims that the training cost of a teacher is significantly lower, the defective teacher [36] can further reduce training effort. To this end, I would suggest this paper compare their methods with methods in [36], such as defective-KD, reversed-KD, and teacher-free KD. Specifically, this paper can compare the accuracy and total training cost with [36] on the experiments in Tables 1, 2, and 3. With these additional experiments, this paper can convincingly claim the advantages of the intermediate checkpoints. Otherwise, I would suggest this paper turns the tune down on the intermediate models and put more effort into the IB theory section. \n\n- The error bar is missing.   \nAlthough this paper claims that the results is evaluated in five independent experiments in CheckList 3(c), this paper only reports the average value, while the standard deviation (error bar) is missing. Since the improvement in many experiments is relatively small (less than 0.5), the experimental results can not be significant until this paper demonstrate that the improvement is large than the error bar. \n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XuRnXiClpmQ", "reviewer": "Reviewer_6Nks", "summary": "The authors propose a new model distillation method utilizing teacher's training checkpoints, showing performance gain over a handful of benchmarks. The authors connect their methods with information bottleneck theory and hypothesize that the performance gain partly comes from the fact that checkpoints save a lot of representation information about inputs, without overfitting to the task. Although the findings are somewhat interesting, I have some concerns with the core methods I discuss below.", "questions": "- **Standard deviation is really useful for gaining real insights about performance improvements**\nThe authors show that multiple experiments over 5 random seeds are conducted. Can you include the standard deviation in all main results tables? Since the performance increment looks small numerically, grounding them with SD could be very useful in interpreting the results.\n\n- **Instead of showing the T^{0.3} T^{0.5} and T^{0.7}, would it be possible to evaluate all checkpoints as in Fig. 4**\nIt would be still interesting to observe that IF curves completely correlate with the downstream distilled model performance.\n\n- **Worth exploring checkpoint distillation with language models**\nIf time permitted, I would be interested to see if this works with language models as well.", "limitations": "The authors adequately addressed the limitations and potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**:\n- The connections between the information bottleneck theory and model distillation seem intriguing. However, I am not convinced that this is the real cause of the performance gain.\n-  It is interesting to see how checkpoints (i.e., not overfitting the training data) generalize better in model distillation.\n\n**Weakness**:\n- **The fact that the information bottleneck theory shows correlated behavior with distilled model performance does not mean IF is the cause or the explanation behind the result**\n\nThe core contribution of the paper is providing a checkpoint selection method based on IF for the teacher model. The core claim behind this contribution is that IF offers a way to measure how good a checkpoint is in preserving \"dark information\" which leads to better image representations for model distillation. However, it is not convincing to me that there is a causal link between IF and checkpoint distillation. Fig.4 is somewhat trivial to any pretrain-then-finetune paradigm, i.e., in the initial phase of finetuning, the pretrained model begins with poor in task performance but with a good representation to reconstruct outputs; at the end, the model becomes overfitting to the task with sacrificed reconstruction accuracies. The fact that IF seems to correlate with downstream distilled model performance does not necessarily prove IF is the reason behind this phenomenon. I layout another very simple theory which could be another explanation behind this, which is in fact, proved in the literature as well.\n\n\n- **What if it is just the fact that the pertaining objective helps in model distillation? i.e., earlier checkpoints simply provide closer weights to the pretrained model, which provides more information about the pertaining objective**\n\nUsing the checkpoints to distill can simply be viewed as a way of preventing the student to overfit the downstream finetuning task. Specifically, it asks the student model to essentially forget task-specific information and asks the student model to recover pretrained information. This is indeed proved by some recent works in language model distillation, i.e., while distilling with the task objective, the student model is also co-trained with the teacher's pretraining objective. With this, there could be many other baselines to run, what if we simply distill with lower learning rates, or with larger learning rate decays? What if we do weighted distillation loss by combining task-specific loss as well as pertaining loss?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "IOxq5X5tgVP", "reviewer": "Reviewer_CyDK", "summary": "**A brief summary**: The paper makes an interesting observation that an intermediate checkpoint of the (teacher) model is just as good for the purposes of knowledge distillation (i.e., training a student model). The paper provides ample evidence to support their observations. They explain this observation using information theory: intermediate checkpoints retain more mutual information (I(X, F)) between the input and the representations than the converged models, which have compressed I(X, F) to maximize target-related information (I(Y, F)). Put simply, intermediate models retain more information about the inputs and other non-target classes. The paper further provides a simple scheme to identify intermediate checkpoints most suited for distillation and demonstrate the efficacy of checkpoints identified through their protocol.\n\nEDIT: Updated my score from 6 $\\rightarrow$ 7, based on the responses during the discussion phase. ", "questions": "How would the results in Table 1 and Table 2 look like if T_{full} is a model based on early stopping rather rather than a model after 120 or 200 epochs?\n\nPlease also see the main review for other suggestions on presentation and optimal model-selection algorithm. ", "limitations": "(Copy pasting my concerns from my main review).\n\nSome of my minor concerns—largely concerning the presentation of the paper—are as follows:\n\nThe figure and table captions in the paper are often incomplete and sow confusion. For instance, after reading Figure 2 the impression a reader gets is that we are looking at teacher performance over epochs. Only later in section 4, it becomes clear that the y-axis is the distillation performance of student models for various intermediate teacher models trained for that many epochs (in x-axis). Similarly, Figure 1 is a cartoon representation of the key result and can be misleading (the KD performance is not necessarily as high as figure 1 would have you believe. Further, it is never clarified if the figure stems from real values or is just a high-level cartoon depiction). Similarly, Figures 4 and 5 leave out important details.\n\nThe paper title, albeit catchy, doesn’t really concern the paper’s key observations, methods or results and is rather a snappy message which could apply to several papers.\n\nThis is more of a suggestion than a concern: it would be great to have an intermediate model-selection process that doesn’t require training the teacher model to convergence (which means that we don’t save any training cost). This could be future work, and therefore could be discussed in the conclusions and limitations section.\n\nSome typos that I noticed and need fixes:\n\n- Line 73, Rethinking → rethinking\n- Line 171, comparable or even better (the word or is missing)\n- Line 329, s in missing from the limitations\n- Line 345, We → we\n\n\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**My assessment**: Overall, I enjoyed reading the paper and think that the paper would be of interest to other members in the research community. Their results also suggest that one could significantly reduce training cost of the teacher models. The key result is intriguing: an intermediate teacher model—checkpoint corresponding to 50% of full convergence—is less accurate by about 8-14 points, **yet yields comparable distillation performance**. This result is supported through a number of experiments on various student-teacher pairs. Their protocol to select the optimal checkpoint is also simple, reasonable and effective.\n\nDespite my overall positive assessment, I have one major and a few minor concerns. My biggest concern is that selecting 200 or 120 epochs upfront is unsettling. In practice, most training runs use an early stopping criterion, where models are trained to a point till they improve validation performance. I would be more confident in the results and my recommendation if the T_{full} is a model based on early stopping rather rather than a model after 120 or 200 epochs. I am curious to see how the key results presented in Table 1 and 2 change upon making this change?\n\nSome of my minor concerns—largely concerning the presentation of the paper—are as follows:\n\nThe figure and table captions in the paper are often incomplete and sow confusion. For instance, after reading Figure 2 the impression a reader gets is that we are looking at teacher performance over epochs. Only later in section 4, it becomes clear that the y-axis is the distillation performance of student models for various intermediate teacher models trained for that many epochs (in x-axis). Similarly, Figure 1 is a cartoon representation of the key result and can be misleading (the KD performance is not necessarily as high as figure 1 would have you believe. Further, it is never clarified if the figure stems from real values or is just a high-level cartoon depiction). Similarly, Figures 4 and 5 leave out important details. \n\n\nThe paper title, albeit catchy, doesn’t really concern the paper’s key observations, methods or results and is rather a snappy message which could apply to several papers. \n\n\nThis is more of a suggestion than a concern: it would be great to have an intermediate model-selection process that doesn’t require training the teacher model to convergence (which means that we don’t save any training cost).  This could be future work, and therefore could be discussed in the conclusions and limitations section. \n\n\nSome typos that I noticed and need fixes:\n\n\n- Line 73, Rethinking → rethinking\n- Line 171, comparable or even better (the word or is missing) \n- Line 329, s in missing from the limitations\n- Line 345, We → we\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AGRNAXj1rzh", "reviewer": "Reviewer_So7a", "summary": "The paper deals with knowledge distillation, which is the training of smaller “student” models from larger “teacher” ones in an effort to reduce the model size while preserving as much performance as possible.\n\nIn this paper, the authors challenge the notion that better performing (and closer to convergence) models make better teachers, instead showing that checkpoints sampled throughout training make better teachers. This counterintuitive finding is explained using the information bottleneck (IB) principle, which suggests that the mutual information between the parameters of a neural network and the model input increase at the start and then decrease. Informally, this corresponds to the observation that models begin by learning correlates at the start, then learn to discard information about class correlates that would be helpful for knowledge distillation. The authors show that teachers are improved by constructing them from “intermediate” models (i.e. checkpointed versions of a model sampled as it is trained) instead of using converged models.\n\nFurther, they use the IB principle to identify the optimal intermediate model for knowledge distillation. They show that selecting a model that maximizes the sum of the mutual information between the parameters and input and the mutual information between the parameters and output yields better intermediate teachers than simpler schemes that sample models at fixed training intervals.\n", "questions": "How significant are the results? (Equivalently, if you were to randomly initialize your experiments and run them again and again, how often would $T^*$ be better than the alternatives.)", "limitations": "Improvement is shown in percentage points, but there is no information presented on the standard deviation of these results. This means that we cannot conclude that the change in results is statistically significant. To make this concrete: in Table 3, $T^*$ has an average performance of 70.87%, which is presented as better than $T^{0.7}$ at 70.77%. It is not clear to me that this difference of 0.1% is better, particularly on the CIFAR-100 dataset.\n\nThe way to fix this is to run each experiment multiple times, present the result and standard deviation, and bold the results that are within some confidence interval of the top result. I suspect that repeating the analysis with this addition will show that the results are less significant than claimed.\n\nUnfortunately, the experiments showing that the use of snapshot teachers/ensembles is better than full, and the experiments establishing the usefulness of their proposed algorithm are also vulnerable to this criticism. I believe this is a fatal flaw in the paper, and needs remediation before publication.\n\n(Similar changes should also be incorporated into the construction of Figure 4. It is not clear to me how significant the shift in the blue vs gray lines are unless some measure of deviation can be plotted in the chart.)\n\nConsider running these experiments again to establish that they are significant. If you cannot show that, consider investigating how a smaller training budget for the teacher network is enough to give good performance in student networks. That may be a useful result.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Originality\nThe work is reasonably original. It is the first application of the IB principle to Knowledge Distillation (KD), yielding an observation that improves the state of KD.\n\n# Quality\nThe interpretation of IB presented in the paper is compelling, but the experiments don’t provide enough information to confirm the significance of the results. I explain this in the Limitations section of the review.\n\n# Clarity\nThe writing is clear and proceeds logically from section to section. I particularly like the intuitive introduction to IB (lines 44-63), which clearly gives readers sufficient intuition to understand the work. I also really appreciate how the related work section summarizes the rather extensive KD literature as a coherent narrative.\n\nMinor points:\n - In Figure 2, the yellow line (ResNet-110/MobileNetV2) is unreadable. Consider sampling colors of different hues with a fixed saturation and value/lightness to prevent such issues.\n - Avoid floating figures on one side of the page, use the default setting of a full-width figure instead. This greatly improves the flow of text.\nFootnote 2 (link to tiny-imagenet) can be replaced by a citation – bibtex supports URL citations. This should also save you a little vertical space for the publication-ready version of the paper.\n - Line 152 “Fistly” should read “Firstly”\n\n# Significance\n\nThe results are sufficiently significant, as it is likely to spawn a line of follow-up work expanding our understanding of KD using IB principles. This is conditional on the results being statistically significant, though.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept