Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors study knowledge distillation in a scenario where the data on which the teacher is trained is no longer available. They propose to use other stimulus (in the form of other datasets or noise) to train the student model, and evaluate the performance on the original dataset. 

The main finding seems to be that using natural images as stimulus is better than using random noise. This is not very surprising. 

From a theoretical standpoint, the fact that random stimulus allows for knowledge transfer is interesting. But this was already covered by Papamakarios (2015). More practically, I cannot see how this kind of technique would be useful in real situations (i.e. I can envision a scenario in which we no longer have the data on which the teacher was trained, but why would we ever be interested in training a student model to perform well on that data?)
review_title: review

final_score: rating: 3: Clear rejection
review_content: This paper tackles semi-supervised learning leveraging Knowledge Distillation.  In particular, they investigate the setting where a teacher has been pretrained but the training data are not available anymore. 

The authors show that distilling knowledge using samples from another distribution/dataset achieves reasonable results and outperforms knowledge distillation with noisy inputs.


- Questions:
Given that you don't have access to the training data, what is your stopping criteria when distilling knowledge from the teacher to the student model?
Don't you think that CIFAR and Tiny ImageNet are both very similar as they both contain natural images? Did you try to leverage MNIST for distilling knowledge in the CIFAR experiment.


- Novelty:
The novelty is somewhat incremental.   [1]  already tried to distill knowledge from 80 million tiny image datasets for solving the CIFAR task. Although, they do use the CIFAR training set as well.

- Clarity:
The paper clarity could be improved overall (citation format looks strange, some typos, what is W_s in equation 1). However, the experimental part is rather clear.

- Significance:
I would be nice to add in the paper a compelling example where you have a pretrained model, but don't have access to the data to train a student model.  I don't really see how the approach developed would be useful in real application.

In summary:
Pros:
- Rather clear experiment section 
- Clearly shows the benefits of distilling knowledge from 'true data' rather noisy inputs.

Cons:
- low novelty
- low significance


[1]  Do network really need to be deep?, Ba, Jimmy and Caruana, Rich, NIPS 2014
review_title: Novelty and Significance?

Decision: Reject