Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper investigates Knowledge Distillation for network compression.  In their approach, the authors propose to match feature vector of the sofmax preactivation. In addition, they introduce a new loss function for training the teacher, i.e. they add a regularisation term so that class-wise clusters of feature vectors are more dense.

Authors evaluate their approach on the CIFAR10 dataset using Resnet for both teacher and student. Contrary to previous approaches using Knowledge Distillation, they show that their approach is able to leverage the Teacher to improve the Student performances with such network architectures.

Questions:
- What is the performance of the Teacher on CIFAR10?
- Did you compare with  knowledge distillation baseline (that matches softmax logits of teacher and student networks) ?
- Do you use Batch Normalization in your residual networks?

Pros:
- The paper is clear an easy to follow
- Authors show that Knowledge Distillation is useful for recent network architecture (Resnets).
Con:
- Experiences on only one dataset.


I recommend acceptance.
review_title: Promising Work

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: In this work, the authors propose to transfer knowledge from a teacher model to a smaller student model with two variations:
1) knowledge is transferred by matching the feature vector before the softmax 
2) the teacher is trained with an additional regularization term to make the feature vectors more dense within the same class.

This is a solid piece work that should be accepted. One question:
- Does the TF-baseline refer to student model trained with traditional cross-entropy knowledge transfer? or the feature vector transfer? If the latter, can you please have additional baseline numbers for student models trained with (standard) cross-entropy loss transfer?

Minor comments:
- Citations: Should really cite Bucila et al. 2006 for knowledge distillation and LeCun et al. 1990 (Optimal Brain Damange) for model compression, as these predate some of the more recent work (Hinton 2015, Han 2016, Jaderberg 2014, etc.)

- "Mimic learning": probably best just to stick to "knowledge distillation"
review_title: good paper

Decision: Accept