Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: **A brief summary**: The paper makes an interesting observation that an intermediate checkpoint of the (teacher) model is just as good for the purposes of knowledge distillation (i.e., training a student model). The paper provides ample evidence to support their observations. They explain this observation using information theory: intermediate checkpoints retain more mutual information (I(X, F)) between the input and the representations than the converged models, which have compressed I(X, F) to maximize target-related information (I(Y, F)). Put simply, intermediate models retain more information about the inputs and other non-target classes. The paper further provides a simple scheme to identify intermediate checkpoints most suited for distillation and demonstrate the efficacy of checkpoints identified through their protocol.

EDIT: Updated my score from 6 $\rightarrow$ 7, based on the responses during the discussion phase. 
strengths_and_weaknesses: **My assessment**: Overall, I enjoyed reading the paper and think that the paper would be of interest to other members in the research community. Their results also suggest that one could significantly reduce training cost of the teacher models. The key result is intriguing: an intermediate teacher model—checkpoint corresponding to 50% of full convergence—is less accurate by about 8-14 points, **yet yields comparable distillation performance**. This result is supported through a number of experiments on various student-teacher pairs. Their protocol to select the optimal checkpoint is also simple, reasonable and effective.

Despite my overall positive assessment, I have one major and a few minor concerns. My biggest concern is that selecting 200 or 120 epochs upfront is unsettling. In practice, most training runs use an early stopping criterion, where models are trained to a point till they improve validation performance. I would be more confident in the results and my recommendation if the T_{full} is a model based on early stopping rather rather than a model after 120 or 200 epochs. I am curious to see how the key results presented in Table 1 and 2 change upon making this change?

Some of my minor concerns—largely concerning the presentation of the paper—are as follows:

The figure and table captions in the paper are often incomplete and sow confusion. For instance, after reading Figure 2 the impression a reader gets is that we are looking at teacher performance over epochs. Only later in section 4, it becomes clear that the y-axis is the distillation performance of student models for various intermediate teacher models trained for that many epochs (in x-axis). Similarly, Figure 1 is a cartoon representation of the key result and can be misleading (the KD performance is not necessarily as high as figure 1 would have you believe. Further, it is never clarified if the figure stems from real values or is just a high-level cartoon depiction). Similarly, Figures 4 and 5 leave out important details. 


The paper title, albeit catchy, doesn’t really concern the paper’s key observations, methods or results and is rather a snappy message which could apply to several papers. 


This is more of a suggestion than a concern: it would be great to have an intermediate model-selection process that doesn’t require training the teacher model to convergence (which means that we don’t save any training cost).  This could be future work, and therefore could be discussed in the conclusions and limitations section. 


Some typos that I noticed and need fixes:


- Line 73, Rethinking → rethinking
- Line 171, comparable or even better (the word or is missing) 
- Line 329, s in missing from the limitations
- Line 345, We → we

questions: How would the results in Table 1 and Table 2 look like if T_{full} is a model based on early stopping rather rather than a model after 120 or 200 epochs?

Please also see the main review for other suggestions on presentation and optimal model-selection algorithm. 
limitations: (Copy pasting my concerns from my main review).

Some of my minor concerns—largely concerning the presentation of the paper—are as follows:

The figure and table captions in the paper are often incomplete and sow confusion. For instance, after reading Figure 2 the impression a reader gets is that we are looking at teacher performance over epochs. Only later in section 4, it becomes clear that the y-axis is the distillation performance of student models for various intermediate teacher models trained for that many epochs (in x-axis). Similarly, Figure 1 is a cartoon representation of the key result and can be misleading (the KD performance is not necessarily as high as figure 1 would have you believe. Further, it is never clarified if the figure stems from real values or is just a high-level cartoon depiction). Similarly, Figures 4 and 5 leave out important details.

The paper title, albeit catchy, doesn’t really concern the paper’s key observations, methods or results and is rather a snappy message which could apply to several papers.

This is more of a suggestion than a concern: it would be great to have an intermediate model-selection process that doesn’t require training the teacher model to convergence (which means that we don’t save any training cost). This could be future work, and therefore could be discussed in the conclusions and limitations section.

Some typos that I noticed and need fixes:

- Line 73, Rethinking → rethinking
- Line 171, comparable or even better (the word or is missing)
- Line 329, s in missing from the limitations
- Line 345, We → we
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper observes that the intermediate models (checkpoints in the middle of the training procedure) can serve as better teachers than fully trained teacher models in knowledge distillation (KD). 
Meanwhile, this paper proposes the Snapshot Ensemble, which ensembles several intermediate models and one full model, and observes that the snapshot ensemble has better KD performance and low training cost than the full teacher ensemble. 
Furthermore, this paper applies the Information Bottleneck (IB) theory to understand KD and “dark knowledge”. Moreover, this paper proposes an algorithm to find the optimal intermediate teacher based on IB theory. 

strengths_and_weaknesses: Strengths.  
- The idea of applying mutual information to understand KD and “dark knowledge” is very novel to the community.  
The curves of mutual information between the input and learned features clearly reveal the training process of a neural network and dark knowledge.  

- Experiments are sufficient. 

- This paper is in general well written and easy to follow. 

Weaknesses

- The findings of the intermediate checkpoints are not very exciting to the community.  
My most concern is about the novelty of the intermediate model part. This paper spends a lot of effort demonstrating the advantages of the intermediate checkpoints. However, previous works [36] already suggest that the Reversed KD (“student teach the teacher”) or even Defective KD (“poorly-trained teacher teach the student”) can achieve comparable results with normal KD. Thus, the observations on intermediate models in this paper should be naturally correct as well and not very exciting. Although this paper claims that the training cost of a teacher is significantly lower, the defective teacher [36] can further reduce training effort. To this end, I would suggest this paper compare their methods with methods in [36], such as defective-KD, reversed-KD, and teacher-free KD. Specifically, this paper can compare the accuracy and total training cost with [36] on the experiments in Tables 1, 2, and 3. With these additional experiments, this paper can convincingly claim the advantages of the intermediate checkpoints. Otherwise, I would suggest this paper turns the tune down on the intermediate models and put more effort into the IB theory section. 

- The error bar is missing.   
Although this paper claims that the results is evaluated in five independent experiments in CheckList 3(c), this paper only reports the average value, while the standard deviation (error bar) is missing. Since the improvement in many experiments is relatively small (less than 0.5), the experimental results can not be significant until this paper demonstrate that the improvement is large than the error bar. 




questions: The idea of applying IB theory to KD is in general novel. Nevertheless, the above weaknesses refrain me from giving a better score. I would be happy to improve my rating as long as the authors address the above weaknesses in the rebuttal.   

limitations: Yes, this paper does mention some limitations in Section 6.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new model distillation method utilizing teacher's training checkpoints, showing performance gain over a handful of benchmarks. The authors connect their methods with information bottleneck theory and hypothesize that the performance gain partly comes from the fact that checkpoints save a lot of representation information about inputs, without overfitting to the task. Although the findings are somewhat interesting, I have some concerns with the core methods I discuss below.
strengths_and_weaknesses: **Strength**:
- The connections between the information bottleneck theory and model distillation seem intriguing. However, I am not convinced that this is the real cause of the performance gain.
-  It is interesting to see how checkpoints (i.e., not overfitting the training data) generalize better in model distillation.

**Weakness**:
- **The fact that the information bottleneck theory shows correlated behavior with distilled model performance does not mean IF is the cause or the explanation behind the result**

The core contribution of the paper is providing a checkpoint selection method based on IF for the teacher model. The core claim behind this contribution is that IF offers a way to measure how good a checkpoint is in preserving "dark information" which leads to better image representations for model distillation. However, it is not convincing to me that there is a causal link between IF and checkpoint distillation. Fig.4 is somewhat trivial to any pretrain-then-finetune paradigm, i.e., in the initial phase of finetuning, the pretrained model begins with poor in task performance but with a good representation to reconstruct outputs; at the end, the model becomes overfitting to the task with sacrificed reconstruction accuracies. The fact that IF seems to correlate with downstream distilled model performance does not necessarily prove IF is the reason behind this phenomenon. I layout another very simple theory which could be another explanation behind this, which is in fact, proved in the literature as well.


- **What if it is just the fact that the pertaining objective helps in model distillation? i.e., earlier checkpoints simply provide closer weights to the pretrained model, which provides more information about the pertaining objective**

Using the checkpoints to distill can simply be viewed as a way of preventing the student to overfit the downstream finetuning task. Specifically, it asks the student model to essentially forget task-specific information and asks the student model to recover pretrained information. This is indeed proved by some recent works in language model distillation, i.e., while distilling with the task objective, the student model is also co-trained with the teacher's pretraining objective. With this, there could be many other baselines to run, what if we simply distill with lower learning rates, or with larger learning rate decays? What if we do weighted distillation loss by combining task-specific loss as well as pertaining loss?
questions: - **Standard deviation is really useful for gaining real insights about performance improvements**
The authors show that multiple experiments over 5 random seeds are conducted. Can you include the standard deviation in all main results tables? Since the performance increment looks small numerically, grounding them with SD could be very useful in interpreting the results.

- **Instead of showing the T^{0.3} T^{0.5} and T^{0.7}, would it be possible to evaluate all checkpoints as in Fig. 4**
It would be still interesting to observe that IF curves completely correlate with the downstream distilled model performance.

- **Worth exploring checkpoint distillation with language models**
If time permitted, I would be interested to see if this works with language models as well.
limitations: The authors adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper deals with knowledge distillation, which is the training of smaller “student” models from larger “teacher” ones in an effort to reduce the model size while preserving as much performance as possible.

In this paper, the authors challenge the notion that better performing (and closer to convergence) models make better teachers, instead showing that checkpoints sampled throughout training make better teachers. This counterintuitive finding is explained using the information bottleneck (IB) principle, which suggests that the mutual information between the parameters of a neural network and the model input increase at the start and then decrease. Informally, this corresponds to the observation that models begin by learning correlates at the start, then learn to discard information about class correlates that would be helpful for knowledge distillation. The authors show that teachers are improved by constructing them from “intermediate” models (i.e. checkpointed versions of a model sampled as it is trained) instead of using converged models.

Further, they use the IB principle to identify the optimal intermediate model for knowledge distillation. They show that selecting a model that maximizes the sum of the mutual information between the parameters and input and the mutual information between the parameters and output yields better intermediate teachers than simpler schemes that sample models at fixed training intervals.

strengths_and_weaknesses: # Originality
The work is reasonably original. It is the first application of the IB principle to Knowledge Distillation (KD), yielding an observation that improves the state of KD.

# Quality
The interpretation of IB presented in the paper is compelling, but the experiments don’t provide enough information to confirm the significance of the results. I explain this in the Limitations section of the review.

# Clarity
The writing is clear and proceeds logically from section to section. I particularly like the intuitive introduction to IB (lines 44-63), which clearly gives readers sufficient intuition to understand the work. I also really appreciate how the related work section summarizes the rather extensive KD literature as a coherent narrative.

Minor points:
 - In Figure 2, the yellow line (ResNet-110/MobileNetV2) is unreadable. Consider sampling colors of different hues with a fixed saturation and value/lightness to prevent such issues.
 - Avoid floating figures on one side of the page, use the default setting of a full-width figure instead. This greatly improves the flow of text.
Footnote 2 (link to tiny-imagenet) can be replaced by a citation – bibtex supports URL citations. This should also save you a little vertical space for the publication-ready version of the paper.
 - Line 152 “Fistly” should read “Firstly”

# Significance

The results are sufficiently significant, as it is likely to spawn a line of follow-up work expanding our understanding of KD using IB principles. This is conditional on the results being statistically significant, though.

questions: How significant are the results? (Equivalently, if you were to randomly initialize your experiments and run them again and again, how often would $T^*$ be better than the alternatives.)
limitations: Improvement is shown in percentage points, but there is no information presented on the standard deviation of these results. This means that we cannot conclude that the change in results is statistically significant. To make this concrete: in Table 3, $T^*$ has an average performance of 70.87%, which is presented as better than $T^{0.7}$ at 70.77%. It is not clear to me that this difference of 0.1% is better, particularly on the CIFAR-100 dataset.

The way to fix this is to run each experiment multiple times, present the result and standard deviation, and bold the results that are within some confidence interval of the top result. I suspect that repeating the analysis with this addition will show that the results are less significant than claimed.

Unfortunately, the experiments showing that the use of snapshot teachers/ensembles is better than full, and the experiments establishing the usefulness of their proposed algorithm are also vulnerable to this criticism. I believe this is a fatal flaw in the paper, and needs remediation before publication.

(Similar changes should also be incorporated into the construction of Figure 4. It is not clear to me how significant the shift in the blue vs gray lines are unless some measure of deviation can be plotted in the chart.)

Consider running these experiments again to establish that they are significant. If you cannot show that, consider investigating how a smaller training budget for the teacher network is enough to give good performance in student networks. That may be a useful result.
review_title: null

Meta-review: metareview: While this paper has 4 accept recommendations among the four reviewers, I have serious misgivings about the content of this paper. The main experimental insight, that a less trained teacher sometimes performs better, is already known and unsurprising. In fact it's the very point of KD that makes that result interesting --- why should it be better to aim for a noisy target than the true target? On any dataset, if we train the teacher for long enough, we will eventually recover the exact labels to arbitrary precision. In that sense, all KD is with an "intermediate" trained model, and the only question is (has always been) just how early to stop. The next issue with the paper is that the authors claim to "explain" theoretically why KD works from the perspective of information bottleneck theory, however what they offer falls short. The “theory” is more like a story, with significant gaps. Most significantly, there is no logic to carry the leaps from stories of how mutual information evolves to why knowledge distillation should work. Moreover what the authors call "mutual information" in their experiments is not actually mutual information and the surrogates they use seem odd choices that are not consistent. For I(F;Y) the authors look at the output of the teacher model but for I(X;F) the authors look at an intermediate layer of F, training a decoder to predict X from the last convolutional layer of F. Why should the information contained in this middle layer of the teacher model matter when the student only accesses the teacher's output? My ambivalence with this paper is two-fold: (i) that the experimental findings are the main contribution and they are by themselves not sufficient for publication and (ii) that the IB component of this paper is misrepresented as a theoretical explanation of the efficacy of KD but actually it falls short. 

Unfortunately I’m discovering these concerns and expressing them after the discussion, hence my recommendation to accept the paper on the basis of the reviewer's initial recommendations. If the work is accepted, I expect the authors to edit it responsibly to remove all misleading claims that suggest that the paper provides a propert theoretical account for why KD works (they certainly have not), versus a speculative intuition, and to be much more careful to disambiguate the quantities that they track from actual mutual information. 
confidence: Less certain
Decision: Accept