Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Distillation is very popular technique both for preparing smaller student model for inference or doing some sort of pseudo-labeling including usage of unlabeled data. Current paper continues study in distillation topic and proposed a new theoretically motivated loss reweighting method to improve distillation of the teacher. The method is mainly studied in context of distillation on huge amount of unlabeled data. Here first student model is trained on supervised data, then the weights for every unlabeled sample are computed once from the results on validation data (we take KNN for distillation samples among validation data to tell what weight value will be), and finally we finetune model with weighted loss on mix of labeled and unlabeled data. Idea behind the weights is to have unbiased estimation for the risk minimization in distillation setting and there is an interesting discussion in the paper on noisy labels relation. Authors perform experiments on various datasets for image classification both with hard labels and soft labels with standard distillation or with a proposed method. Additional ablations on the supervised dataset size are conducted. Proposed method is hyper-parameter free, data-agnostic, and simple to implement. Experimental results are supported by theoretical analysis which rigorously justifies the method in certain settings.
strengths_and_weaknesses: **Strengths**
- A new reweighting method for knowledge distillation with unlabeled data which is based not on the uncertainty estimation (as people often do in pseudo-labeling) but on unbiased reweighting. 
- The method is simple to implement 
- Strong experimentation results across different datasets for image classification task (including ImageNet and smaller datasets): both for hard and soft distillation / with different amount of supervised data proposed method has statistically better results. [Sometimes however improvement is still marginal though statistically better]. 
- Interesting connection to noisy labels but here authors formulate the problem that noise is very specific and we can play with it.
- Detailed instructions on reproduction experiments which are simple to follow.
- Theoretical justifications of the proposed method for simpler forms of functions and standard regularity conditions: (i) it is information-theoretically optimal; and (ii) SGD optimization of the reweighted objective learns a solution with nearly optimal generalization.

**Weaknesses**
- Some mistakes in the proofs and formulations are found from Appendix section, however these findings should not affect the result and main statements.
- Additional results on distillation on supervised data and training curves for current results would be great to have for the full picture (details see in the Questions section).
- Ablation on recomputing the weights during the training (with some simple schedule at least, say every 50 epochs).
- Ablation on the temperature for soft distillation as it could be the main contribution for the generalization and maybe proposed approach doesn't change much. Temperature usage is very standard setting in distillation works.
questions: **Main paper**

The paper is very well written, thanks authors for this careful formulations and style of writing! Please find below my small suggestions on improving the text and typos as well as some extra questions on the results.
- I would suggest authors to include [1] into introduction / related works section and discuss this paper in the context of usage unlabeled data and getting unbiased predictions from teacher. Also would be a bit simpler to track references if they are ordered by their appearance in the text.
- line 27 "overparametrized networks" - here I wonder about this formulation (and do not agree with it) as student model is often much smaller (and authors also mentioned this before), so hard to say how this small student models will be overparametrized. Maybe reformulate this a bit?
- In the whole paper (including appendix) there is a bit mess with usage of space only for inputs or for pair (input, target), e.g. lines 129-130, equation (3), etc. Would be simpler to have a unified way on these definitions and usages in expectation formulas. Maybe having notation of $X$ and $\tilde{X}$ for input data saying that these spaces are the same for original data and adversarial.
- line 182 formula for $k$: fix $2 / 2+D$ to $2 / [2+D]$
- With respect to recent results in [1] I wonder what is happening with proposed method in case we have only labeled data and distillation is done as the original knowledge distillation paper (on training data of teacher model only)? Does reweighting scheme work? Also does unlabeled data improves distillation compared to distillation on supervised data only? [1] showed interesting results in this direction. 
- [1] also introduced discrepancy between training quality (how we transfer knowledge from teacher to student) and generalization (about which we care). Would be interesting to see training curves (training loss and accuracy) for student models with standard distillation and weighted one as maybe proper weights change the optimization and simplify it even.
- Could authors provide what is approximate amount of time on computing weights for every sample? I guess it is negligible overall compared to the training, but curious to see confirmation.
- Do authors do any ablations on recomputing the weights for every sample say every 50 epochs? At least to know if it changes a lot or not during the training.
- Could authors perform ablation with temperature for the soft distillation case? Here we know (see e.g. in [1]) that temperature is giving a lot for optimization and generalization, even outweights other techniques to improve distillation. I wonder how it can change the optimization procedure and affect the proposed method. Also in practice we most likely use temperature not equal to 1.
- Could authors provide numbers on the teacher performance pre-trained on supervised data before starting the distillation?
- In Section 4 notation on spaces and distributions is not consistent with Section 2.

**Appendix**

I went through almost all appendix except last two proofs for sgd. General comment - Appendix needs a revision, it is less clear and careful prepared compared to the main text. For the rest of the text comments are below
- Appendix A plots are strange: teacher accuracies between hard and soft distillation are not consistent (e.g. Fig 8 leftmost top and bottom plots). Why are they different? We start distillation from the same teacher for both hard and soft distillation so teacher accuracy should be exactly the same.
- line 547 "if the sample" -> "is the sample"
- line 558-559 usage of $M$ while it should be $\mathcal{M}$
- Equations (7) and (8): what about variance estimation? Do we have any assumption on it? Or it is bounded for some reason?
- line 572: I repeated the math several times and got another range for $p$. It is $[0, (1-2d) / (1-d)^2]$ where $d$ is distortion. This does not affect the main conclusion and restricting weights to $[0, 1]$.
- C3 section should be moved earlier before C1
- in Equations after line 586 I don't understand how $l(f_{true}, f)$ disappears in $Bias(f)$. I think $Bias(f)=E_{x \sim\mathbb{P}} \\,p(x) (d - 1) * l(f_{true}, f)$ where $d$ is distortion.
- line 577 type "than" -> "that"
- notation is Section C3 and beginning of D should be fixed as again it is messed up with notation for input only and for (input, target) pairs.
- lines 600-606: usage of notation $z$ should be changed. It is used as $x$ context and later as $y$ context.
- line 629 - seems English language should be fixed here.
- It will be simpler to read if all proofs from Section D.* will be placed in the first statements formulation in section D itself, not reformulating this several times.
- Lemma D.7 - please add proper reference to the proof of this property (or via which theorem it can be proved).


[1] Stanton, S., Izmailov, P., Kirichenko, P., Alemi, A.A. and Wilson, A.G., 2021. Does knowledge distillation really work?. Advances in Neural Information Processing Systems, 34, pp.6906-6919.
limitations: Limitations are listed in the conclusion section.


**Discussion period updates**: I changed my rating score from 7 to 8 during the discussion period, details see in the thread below.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper deals with the problem of few-shot knowledge distillation and proposes a method for weighing unlabeled examples based on the gaps between the true and distilling teachers. The obtained weights are employed for computing the loss function.
strengths_and_weaknesses: [Strengths]

S1. The proposed method is simple and intuitive, which would draw the attention of people who are interested in applying it to their own tasks or applications.

S2. The current manuscript is basically well written and easy to follow.

[Weaknesses]

W1. The current experimental evaluation fails to demonstrate the effectiveness of the proposed method due to the following two reasons:

  (1) It lacks empirical comparisons with other related methods. In particular, [Dehghani+ ICLR2018] and [Kimura+ BMVC2018] incorporated example weighting for knowledge distillation. 

  (2) According to the experimental setup presented in Section 3.1, both the teacher and student models have access to whole the training dataset, which is not a typical setting in the context of knowledge distillation. The teacher is usually trained with the full size of supervised training examples and the student is able to employ the same dataset as the teacher (fully supervised), a part of the supervised dataset plus additional unsupervised examples (few-shot) or only unsupervised examples (zero-shot). Due to this experimental setting, it is very difficult to understand whether the current baseline sufficiently works well or not.

W2. The proposed method requires additional supervised validation examples for computing the weights for unsupervised examples, which is a critical limitation of the proposed method. Existing methods for weighing examples do not rely on the availability of validation examples. If they are available for training, it seems better to employ them as additional training examples.
questions: I have no questions about this paper. 
limitations: See the Strengths and Weaknesses part.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a knowledge distillation technique for semi supervised settings. For the labeled data, they uniformly assign equal weights for the cross entropy loss between the predicted output and the supervision. On the other hand, the proposed method distributes adaptive weights for each data, which is theoretically and rigorously justified. The experimental results also support their theoretical analysis.

strengths_and_weaknesses: Strengths: 
1. The method is well-motivated and theoretically justified. 
2. Even though the accuracy improvements are marginal compared to the baselines, the proposed method consistently outperforms them. 

Weaknesses:
1. The method is hard to follow due to the missing details and confusing notations. For example, D has different meanings in algorithm 1 and line 183.
2. The experiments were insufficient. The previous standard knowledge distillation techniques such as [1,2,3] can still be applied for the problem settings used in the main paper. Could you compare with the previous algorithms [1,2,3]?

[1] Wasserstein Contrastive Representation Distillation, CVPR 2021.
[2] Contrastive Representation Distillation, ICLR 2020.
[3] Variational Information Distillation for Knowledge Transfer, CVPR 2019.  
questions: 1. It is hard to understand k-NN(U, Query) in the algorithm part 1. Could you explain it in detail? I can not find the definition of the  function.  
limitations: The authors adequately addressed the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors proposed a reweighting method to improve the performance of data distillation on the unlabeled dataset. Experiments on various datasets (SVHN, CIFAR-10/100, CelebA, and ImageNet) demonstrate the performance of the proposed method.
strengths_and_weaknesses: Advantages:
1. The research topic, distillation with unlabeled examples, is very close to many real-world applications.
2. The proposed method has some provable guarantees.

Disadvantages:
1. The proposed method, especially the procedure for weight estimation is quite complex and computationally expensive. If the labels from the teacher model are not reliable (can be regarded as label noise), why not apply some label noise methods to tackle this problem?
2. In the experiments, the performance gain is not that much (e.g., on CelebA and ImageNet) when compared with the unweighted method.
3. The paper does not show how and why using K-nn to learn the mapping between label confidence and the estimates of p(x) and the distortion(x). Is there any other method that can be used? 
questions: I'm wondering why or how likely the pseudo labels are not reliable in the considered setting of the paper. In many cases in distillation with unlabeled data, the unlabeled dataset is different from the labeled data (or comes from a different distribution) used to train the teacher model. Then, it is likely that the pseudo labels given from the teacher model are unreliable. But when the labeled data and the unlabeled data come from the same distribution (i.e., the setting used in the experiments), this may be not a big issue.
limitations: It seems that the authors haven't addressed the potential negative societal impact of their work.
review_title: null

Meta-review: metareview: This paper proposes to learn to reweight data samples in the distillation process to deal with potential noisy labels from the teacher. The writing is clear, and the empirical and theoretical results are satisfactory in general. At the beginning, the paper receives a mixture of positive and negative scores. The reviewers raise a number of questions, mostly about empirical comparisons with related works. The authors didi a good job in the rebuttal by providing many comparisons requested by the reviewers. The rebuttal resolves most of the concerns from the reviewers. Finally, the strong supporter remains strong support and the most negative one agrees that his concerns are also resolved, and urges that the authors should carefully revise the paper by incorporating all the results from the rebuttal into the paper. I standby the reviewers. Authors, please make sure to incorporate the extra results and clarifications into the final revision.
confidence: Certain
Decision: Accept