Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Knowledge distillation (KD) is to improve a low-capacity student network's performance with a high-capacity teacher network. Current works focus on how to define KD loss. This work proposes to improve KD from data augmentation (DA). This work proposes an entropy-based data-mixing DA to improve two KD methods: KD and CRD.
strengths_and_weaknesses: Strengths:
1. The method is simple and clearly presented.
2. The experiments are done on different data augmentation techniques.
3. Results on CIFAR-100 and TinyImageNet show increased performance.

Weaknesses:
There are three contributions in the method part: (1) data augmentation (2) longer training and (3) cutmixpick. (1) and (2) are already investigated in works [1][2]
[1] Cui, Wanyun, and Sen Yan. "Isotonic Data Augmentation for Knowledge Distillation." IJCAI2021
[2] Beyer, Lucas, et al. "Knowledge distillation: A good teacher is patient and consistent." CVPR2022.
The improvement from (3) is very minimal.


After rebuttal:
The authors have addressed my concerns. I increased my score to boardline accept after considering other reviewrs' comments.
questions: The main concern is the novelty of the proposed methods (see weakness). 
limitations: The authors have addressed the the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper establishes a connection between data augmentation (DA) and generalisation of student-teacher training, namely showing theoretically that given two subsequences of the original dataset D (where the subsequences differ by some data augmentation scheme), the subsequence that performs better is the one that induces less variation in the predicted probabilities of the teacher network.

**Edited review**

In light of the authors _active discussion and contributions_ during this phase, as well as their addressal of my own concerns, I will tentatively raise my score from 4 to 7. Obviously this is conditional on some changes to the manuscript, namely:
- Making the math more digestible (we already discussed this in great detail)
- Preferring plots to large tables where applicable, or having plots supplement them. I leave this to the authors' discretion since space constraints will mean decisions will have to be made as to what should be a table vs what should be a plot, and what should be relegated to the appendix.
- Further work on plotting Equation 8 vs its approximation vs both 'bad' and 'good' students. The authors have acknowledged this and even provided preliminary results.

Thanks, and good work.
strengths_and_weaknesses: # Strengths

- The efficacy of any DA scheme for student training could in principle be evaluated before the training of any student model, since computing the variance term $V[\Delta]$ only requires the pretrained teacher model.

- Experiments support the proposed theory, and are numerous, run over many different architectural variants as well as data augmentation schemes.

- Proposal of an entropy-based scheme to improve results.

# Weaknesses

Some of the notation in Section 3.1 and 3.2 is incredibly confusing, and looks to be rushed. Firstly, I would recommend simply using $\log$ instead of $\mathcal{L}$ to denote the logarithm. Alternatively, if you want to be more abstract, you can simply define the cross-entropy loss as $\mathcal{L}_{CE}(x, y; f) = y^T f(x)$ and it would make equation (6) more readable. In fact, you have already defined it as such in Equation (1).

In Section 3.2, I have several concerns. In Equation (6), in line 1 you have an expectation $S \sim D^N$, yet in line 2 it confusingly turns into an expectation $x \sim D$, yet you still have the inner summation. Why isn't it just this instead:

$E_{x \sim D} [ p(x) \log f(x) ] + \text{const}$

i.e. the expectation implies the inner summation + multiplication over $1/N$, so having both seems redundant (and incorrect). Furthermore, in the inner multiplication you're missing the transpose, so it should be $p(x_n)^{T} \log f(x_n)$, since that is how it is defined in equation (4).

In Equation (7), I am also not sure why the last line expands out into a variance + covariance term. Covariance to me implies that you're computing the variance between two sets of random variables; don't we only have one random variable? (That random variable being $\Delta$, the empirical risk over whatever random draw $S \sim D$.) Maybe I am missing something crucial, but it seems like one could just remove the last line of Equation (7), leaving us with:

$V_x[ p(x)^T \log f(x) ]$,

then say that the variance is going to be proportional to the $p^{(t)}$ and that $f$ is held constant. Actually, it seems like from Equation (9) that the covariance is over the $p$'s for the different classes. What is confusing is that, for a given $x$, you are indexing into its predicted probability for class $i$ as as $p^{(t)}(x_i)$, and $x_i$ is apparently the $i$'th example as you have implied in equations (3) and (4). If $p^{(t)}$ outputs a probability distribution over x, then you should be using $p^{(t)}(x)_i$ instead. 

"Each element in S is drawn the same underlying distribution D." - I don't follow this, since it says earlier that $S_1$ and $S_2$ are elements from $D$ but not necessarily iid. If so, how can $E_{x \sim S}$ turn into $E_{x \sim D}$?

Table 1: I appreciate that the results are averages over three runs, but please indicate the standard deviation. Since space seems a little tight in this table, maybe just indicate the stdevs below each number in parentheses. Also such information can be nicely condensed into a scatterplot with one axis denoting test accuracy and the other denoting mean stdev. When necessary, prefer plots over big tables, and leave big tables to the appendix.
questions: My questions mainly lie in the aforementioned weaknesses.
limitations: n/a
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a theoretical development of a straightforward to use measure of effectiveness of data augmentation techniques for distillation. It proposes that this measure need only be based on an analysis of the teacher, without consideration of the student. The paper also presents an enhanced version of the cut-mix data augmentation technique, which performs a distillation of the augmented dataset to maximize the information content of the data.
strengths_and_weaknesses: The method yields an informative measure that, empirically, can be reliably used to judge the efficacy of different data augmentation schemes, at least for the methods considered.

The justification and derivation of the proposed measure is reasonable, but it (possibly) depends on an overly strong assumption on the student to be "well-trained". This may not be the case in some applications where the student has a much lower capacity than the teacher. However, the proposed measure may still be effective, but would need more detailed theoretical justification to show this.

The cut-mix-pick data augmentation technique demonstrates improved performance over the standard cut-mix method.

questions: Non-data manipulations, such as drop-out, can result in the same behavior as shown in figure 2a (of spreading different soft targets while keeping the same hard targets).
The paper (Bouthillier, X., Konda, K., Vincent, P., and Memisevic, R. Dropout as data augmentation. arXiv preprint arXiv:1506.08700, 2015.) even considers the dropout as a type of data augmentation. So, could your approach be made even more general (than the few types of augmentation you consider)?

In lines 127-129, why is a script L used to indicate the log function, rather than the more usual log() notation? Script letters are conventionally used to indicate spaces or sets. In line 139 the script L is used to indicate a loss function, adding to the confusion.

In line 154-156 the assumption is made of a "well-trained" student, that gives softmax outputs "pretty equal" to those of the teacher. However, in most practical applications of distillation, the student can be much smaller than the teacher (e.g. for network compression for edge devices). What happens to your conclusions in that case? This assumption is used to make the claim that we need only look at the teacher to judge the effectiveness of a DA method. So, is this claim still true if the assumption does not hold?
In table 1, (as was done in table 3) you should give the teacher and student test accuracies to see how well the "well-trained" assumption holds. For CIFAR-100 and the student/teacher networks used, I would expect a significant difference in softmax outputs (e.g. 74% for ResNet56 and 69% for ResNet20).

It seems surprising that an analysis of the student has no role to play in determining the effectiveness of a data augmentation technique that is being used to train that student. If no distillation was being done at all (just training using cross-entropy on the hard targets), DA still would be of some use, no? So, this suggests that one cannot just ignore the student in determining the effectiveness of the DA method. Perhaps, in light of the results in table 1, the teacher's effects dominate the student's effects in terms of judging DA performance, but the student may still have some influence. It would be nice to have some theoretical insight into why this is, rather than just throwing away the student based on the "well-trained" assumption.

Does the "pick" aspect of the Cut-Mix-Pick help with the other DA methods? Presumably doing the data distillation on the other augmentations would also improve their results.

In section 5.1 the authors say that they do not look at accuracy, as the derivations are based on error, and sometimes a network can give high accuracy but also high error. But accuracy is typically what people are interested in when using networks, not the value of the loss function. It may be that by focusing on loss (error) one is biasing the student to become more confident, like the teacher, and perhaps overly confident, leading to reduced accuracy. In any case, table 3 shows accuracies, so I am not sure why this was not done in section 5.1.
limitations: There is no discussion of limitations and potential negative societal impacts. However, distillation techniques, to the extent they can be used to train smaller networks, may positively affect societal impacts related to high power usage during training and inference time.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The major contributions of this paper are 2-fold:
1) The authors suggest that given a fixed teacher model, a good data augmentation (DA) scheme is characterized by a lower variance of the teacher’s mean output probability. This claim is supported by statistical analysis and empirical results (image classification).

2) An entropy-based sample picking scheme is introduced to synthetically reduce the variance of the teacher’s mean probability. This scheme (CutMixPick) further improves the results.
strengths_and_weaknesses: **Strengths:**
1) Setups with lower variance (although it is not clear how $u$ is calculated, see weaknesses below) obtains acceptable improvements in CIFAR-100 and Tiny-ImageNet experiments.


**Weaknesses:**

Though this work is executed well and the image classification results show improvements, the novelty and significance of this work are limited. See below:
1) The clarity of Section 3.2 can be improved. Though proposition 1 seems interesting, I have questions regarding the Std of the teacher’s mean output probability. It is unclear as to how $u$ in equation 9 is calculated. Particularly:
- What is $S^*$ and how is it determined? Why is $K=640$?
- Would it be reasonable to calculate $m$ using the variance of the output probabilities of the teacher **for the correct class** over all samples?
2) Experiment coverage / results are limited: 
- Particularly, ImageNet results are not discussed and analyzed. Not that I have anything against CIFAR-100, Tiny-ImageNet analysis, but I tend to think that ImageNet results are critical to add merit to the findings (1000 classes, higher resolution:224 x 224).  Most KD works report results on ImageNet-1K ([1, 2, 3, 4, 5]). If compute is a problem, authors can consider using a smaller dataset for experiments: ImageNet-100 (A subset of ImageNet popular in the self-supervised learning community). I believe the authors can also use publicly available pre-trained ImageNet models if required: https://pytorch.org/vision/stable/models.html
- What is the reason for not including Flip, Flip+Crop, Cutout, Auto-Augment, Mixup results in Tables 3, 4?
3) For ‘Std of mean prob’ in Tables 1, 2, is it reasonable to use the training set as the training set is used for distillation (not the test set)?
4) For CutMixPick experiments, what is the percentage of the training set used for distillation (after 'picking' / filtering)?

Overall I enjoyed reading this paper. In my opinion, the weaknesses of this paper outweigh the strengths. But I’m willing to change my opinion based on the rebuttal. 


=====================

[1] Müller, Rafael, Simon Kornblith, and Geoffrey E. Hinton. "When does label smoothing help?." Advances in neural information processing systems 32 (2019).

[2] Shen, Z., Liu, Z., Xu, D., Chen, Z., Cheng, K. T., & Savvides, M. (2021). Is label smoothing truly incompatible with knowledge distillation: An empirical study. In ICLR

[3] Chandrasegaran, K., Tran, N. T., Zhao, Y., & Cheung, N. M. (2022). Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?. ICML

[4] Heo, Byeongho and Kim, Jeesoo and Yun, Sangdoo and Park, Hyojin and Kwak, Nojun and Choi, Jin Young (2019). A Comprehensive Overhaul of Feature Distillation. ICCV

[5] Tang, Jiaxi, et al. "Understanding and improving knowledge distillation." arXiv preprint arXiv:2002.03532 (2020).

questions: Please see Weaknesses section above for a list of all questions.

Remark : I managed to only study the main text. Unfortunately, I only skimmed through the supplementary, but didn't study in detail due to its length / my time-constraints.
limitations: The authors have discussed limitations / potential societal impacts in Supplementary Section 6.
review_title: null

Meta-review: metareview: After a lively and interactive author discussion period all reviewers ended up recommending to accept this paper.
The work examines the ways in which different data augmentation schemes can increase knowledge distillation performance, providing some theoretical analysis with actionable insights and experiments to back it up. The work focuses on the generalization gap of the student under different sampling schemes, and asserts that their study leads to the conclusion that a good data augmentation scheme should reduce the variance of the empirical distilled risk between the teacher and student. Reviewers were generally positive about the clarity of the manuscript after some changes during the author discussion.

The AC recommends acceptance.
confidence: Certain
Decision: Accept