Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper investigates knowledge distillation using a data set consisting of a single or a few images which are heavily augmented. The paper finds that this type of knowledge distillation is as effective as using the training set, at least for small scale data sets such as CIFAR-10/100. Different design choices such as student and teacher architecture, the effect of the teacher training set as well as the image(s) chosen for distillation and the augmentations used are explored in-depth. Furthermore, the method is also shown to be effective for other modalities such as audio and videos, albeit these are less explored than images.
strength_and_weaknesses: Overall I feel that the paper provides a comprehensive investigation of single/few image based distillation. Nevertheless, I have a few suggestions on how to improve the paper.

*Strengths:*
- The observation that a single or a few images are enough for distillation is surprising and interesting (at least to me).
- The paper covers a lot of datasets, architectures, and modalities, and provides many insightful ablations, and as such answers many important questions on the phenomenon.

*Weaknesses/suggestions for improvement:*
- One important open question: What happens to transfer and robustness properties of the distilled models? The output confidence scores seem to change significantly on the testing set (Figure 3), and the paper also delivers a potential hint why this could be happening: The output probabilities on the samples used for training are very flat for the teacher (Figure 8). Given this observation, the models might be mis-calibrated or have reduced OOD robustness, which would merit more investigation.
- A related analysis would be to look at examples for which are misclassified by the student but correctly classified by the teacher. Are there some patterns.
- It would be interesting to see if more sophisticated synthetic data than uniform noise can lead to better transfer, such as the synthetic data sets considered in Baradad et al. 2021.
- The observation that the method works less well for transformer models could be discussed in the main paper. Given that transformers are harder to train on small data sets than ConvNets this is not very surprising.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. It provides a comprehensive investigation of the phenomenon and covers most of the important aspects. The experiments are well-documented and the details should suffice to reproduce the results. To my knowledge, distillation based on a few or a single image has not been explored before, so the contributions are novel as far as I can tell.
summary_of_the_review: The paper provides an interesting and comprehensive investigation of knowledge distillation based on a single or a few images. I think the paper should be published, but I also believe the points I raised above regarding calibration/robustness should be answered before publication, which is why I chose my rating defensively.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper uses a single image along with its augmentation and a pretrained classification model (teacher) together to train a student classifier. Through experiments on different datasets including images, audios, and videos, the authors find that the student classifier can achieve reasonably good classification performance.
strength_and_weaknesses: Strength:
In order to verify its effectiveness, the authors have conducted experiments on different datasets and also conducted ablation studies on different components of this proposed method. 


Weakness:

1) The motivation is not clear. Since this proposed method only uses a single image and its augmentations as its training data (it also uses a “teacher” model which should also be considered as an input information source), thus it might be acceptable that it will sacrifice some performance/accuracy. But the question is do we really want to do that? I do not see any benefits from doing this. If we think there are some domains where obtaining training data is expensive, then we say this model may be helpful. However, the thing is we still need a “good” teacher which should be trained on a large amount of dataset in this domain.

2) Using a single image to train a classifier and gain reasonable accuracy sounds surprising; however, there are several recent papers finding that even a “randomly initialized neural network with frozen network” can also yield good classification performance (I list several references below). Given this background, the finding may not be as impressive as it sounds since in any way it has updated its network weights and take the advantage of the knowledge from the “good teacher”.

Ref1: K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun, “What is the best multi-stage architecture for object recognition?” 2009 IEEE 12th International Conference on Computer Vision, Sep. 2009, pp. 2146–2153, ISSN: 2380-7504.

Ref2: H. Zhou, J. Lan, R. Liu, and J. Yosinski, “Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask,” in Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc.,
2019.

Ref3: K. Sreenivasan, S. Rajput, J.-y. Sohn, and D. Papailiopoulos, “Finding Everything within Random Binary Networks,” 2021.

Ref4: S. Baek, M. Song, J. Jang, G. Kim, and S.-B. Paik, “Face detection in untrained deep neural networks,” Nature Communications, vol. 12, no. 1, p. 7328, Dec. 2021.

Ref5: G. Kim, J. Jang, S. Baek, M. Song, and S.-B. Paik, “Visual number sense in untrained deep neural networks,” Science Advances, vol. 7, no. 1, p. eabd6127, Jan. 2021.






/********************************** After Rebuttal******************************/

Thanks for your response. After reviewing the responses and the discussions, most of my major concerns have been resolved. I would slightly raise my rating. 

clarity,_quality,_novelty_and_reproducibility: The overall presentation and writing sound good to me. The authors also provided the code, so it might be possible to reproduce the results.
summary_of_the_review: Overall, this paper is easy to follow and understand. It also has a lot of experiments. The big issue is it is not clear what benefits this method can bring to the community; also “random neural network with frozen weights” can also achieve impressive classification accuracy, then the finding in this paper does not sound as surprising as it is.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a distillation framework that makes a network learn with crops and augmentations from a single image. The learning target comes from a pre-trained teacher's outputs. The results obtained is of decent level (69% ImageNet top1). Various design options and internal properties of the resulting network are analyzed.
strength_and_weaknesses: Strengths:
1 ) It is interesting to know with a teacher and a single large image, what level of accuracy a student can achieve on certain images.

2 ) It demonstrates (vanilla) knowledge distillation's immense power. It means KD can be done with extremely small amount of data. The paper may consider citing related few-shot KD methods such as [1].

3 ) The paper's ablation study is comprehensive and reveals many interesting points. 
	

Weaknesses

1 ) The teacher is supervised pre-trained. It contains information from a full dataset. Thus it may not be fair to say this study helps to understand the mechanism of NNs when provided with the minimum data. Moreover, the semantic categories are explicitly  modeled with teacher's final classifier, which is clearly learned from large data instead of the single image. 
	
Therefore, I don't think the question in "we go far beyond these and instead ask what the minimal data requirements are for neural networks to learn semantic categories" can be answered with "only one image and augmentations" without mentioning the teacher.

The conclusion sentence in the discussion section may add "together with a supervisory signal provided by a teacher pretrained on large data with many semantic categories, provide sufficient …"
	
There is another way to frame the conclusion and finding: KD can be done with one image and augmentations to reach decent accuracy. I think this one represents the finding more accurately and concisely.
	
2 ) As the paper suggested in the intro, the use case of the method is not obvious.
	
3 ) It may be an exaggeration to say that this is learning from a single datum also because the image used is very high resolution. Small crops of CIFAR-size (32x32) with data augmentations easily surpass thousands. The "image space" doesn't specify image size so in extreme I could tile all ImageNet images into one and still call it one "image". 

[1] few sample knowledge distillation for efficient network compression

clarity,_quality,_novelty_and_reproducibility: The paper is well written. Details are documented in appendix.
summary_of_the_review: The paper presents some interesting findings but I think there are exaggerations in claims as discussed in weaknesses, so both considered my recommendation is "marginally above".
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this work, the authors demonstrate the ability to semantically classify images based on a single example by leveraging heavy data augmentations. In particular, the authors employ a single image along with its augmentations and a pretrained classification model to train a student classifier with knowledge distillation. The authors find that a single or a few images are sufficient to achieve good performance on CIFAR-10 and CIFAR-100. The authors show additional results on audio and video datasets. 

The reviewers applauded the author’s comprehensive experiments and the clarity of presentation. In addition, some reviewers commented that they were surprised about the quality of results given the limited amount of data. In follow up discussion, the authors offered some new results suggesting that the few image distillation were sufficient to even produce non-trivial results on transfer learning and robustness experiments. The reviewers did find some concern about the interpretation of the results and asked the reviewers to clarify their writing accordingly. These issues, while important, have been addressed and provide for a reasonable interpretation of the contributions of this work and the placement of this work within the literature. Given the interesting findings based on convincing experiments that are clearly explained, this paper will be accepted in this conference.

justification_for_why_not_higher_score: Unclear interpretation of results.
justification_for_why_not_lower_score: Interesting result. Clear presentation.
Decision: Accept: poster