Like What You Like: Knowledge Distill via Neuron Selectivity Transfer

Abstract: Despite deep neural networks have demonstrated extraordinary power in various applications, their superior performances are at expense of high storage and computational costs. Consequently, the acceleration and compression of neural networks have attracted much attention recently. Knowledge Transfer (KT), which aims at training a smaller student network by transferring knowledge from a larger teacher model, is one of the popular solutions. In this paper, we propose a novel knowledge transfer method by treating it as a distribution matching problem. Particularly, we match the distributions of neuron selectivity patterns between teacher and student networks. To achieve this goal, we devise a new KT loss function by minimizing the Maximum Mean Discrepancy (MMD) metric between these distributions. Combined with the original loss function, our method can significantly improve the performance of student networks. We validate the effectiveness of our method across several datasets, and further combine it with other KT methods to explore the best possible results. Last but not least, we fine-tune the model to other tasks such as object detection. The results are also encouraging, which confirm the transferability of the learned features.

Reviews and author responses:

This submission proposes a novel loss function, based on Maximum Mean Discrepancy (MMD), for knowledge transfer (distillation) from a teacher network to a student network, which matches the spatial distribution of neuron activations between the two.

The proposed approach is interesting but there is significant room for improvement. In particular:

*Clarity*

It is not clear how the distribution of neuron activation are matched between the teach and student network. The C_T and C_S are not defined specifically enough. Does it include all layers? Or does it only include a specific layer (such as the last convolution layer)?

*Interpretability*

Section 4.1. tries to interpret the approach but it is still not clear why matching distribution is better. The MMD loss proposed could run into problem if the classification task does not involve "spatial variation". For example, for a extremely simple task of classifying three classes "R", "G" and "B" where the whole image has the same color of R, G and B respectively, the spatial distribution is uniform and the proposed MMD loss would be 0 even if the student network's channels do not learn discriminative feature maps. Another example is when a layer has H=W=1.

*Significance*

The experiment shows that polynomial-two kernel gives better result, but Sec. 4.3.2. mentions that it is equivalent to Li et al. (2017b) in this case.

*Practical usefulness not justified*

In the experimental section, the student network's number of parameters and FLOPS are not detailed, so it is unclear how much efficiency gain the proposed method achieved. Note in practice small networks such as MobileNet and ShuffleNet have achieved significantly better accuracy-efficiency trade-off than the teacher networks considered here (either for CIFAR10 or for ImageNet1k).

*Improvement not significant*

The results obtained by the proposed approach is not very significant compared to "KD" along.

This paper targets knowledge distillation of a large network to a smaller network. The approach is summarized by equations (3) and (4), which in short proposes that one should use the maximum-mean-discrepancy (MMD) of the network activations as a loss term for distillation.  

When considering CIFAR image classification tasks, it is shown that only when using a specific quadratic polynomial kernel (which as described in https://arxiv.org/pdf/1701.01036.pdf is tantamount of applying neural style transfer) the proposed approach is able to match the performance of the seminal paper of Hinton et al.  When embarking to imagenet, the proposed approach is only able to match the performance of standard knowledge distillation by adding the quadratic term (texture in neural style synthesis jargon). This is actually a sensible proposal. Yet, the claims about MMD as a way of explaining neural style transfer has appeared in the paper cited above, which the authors mention.

The idea of transferring from one domain to another using MMD as a regularizer appeared in https://arxiv.org/pdf/1605.06636.pdf by Long et al --- indeed equation (3) of this paper matches exactly equation (10) of Long et al. Note too that Long et al also discuss what kernels work well and which work poorly due to vanishing gradients and propose parametrised solutions. This is something this paper failed to do.

The two works cited above make me wonder about the novelty of the current paper.  In fact, this paper ends us being an application of the neural style transfer loss function to network distillation. As such this could be useful, if not already done by someone else previously.

I find that the paper is poorly written, with many typos, and lacks focus on a single concrete story. The CIFAR experiments fail to use KD+NST (ie the thing that works for imagenet - neural style transfer) and section 5.3 appears trivial in light of the cited works. For all these reasons, I am inclined to reject this paper.


















This paper proposes a simple method for knowledge distillation. The teacher and student models are matched using MMD objectives, the author demonstrates different variants of matching kernels specializes to previously proposed variants of knowledge distillation.

- The extensive evaluation suggests that the MMD with polynomial kernel provides better results than the previously proposed method.
-  It is interesting to see that MMD based transfer has more advantage on the object detection tasks.
- Can the author provides more insights into the behavior of different kernels, for example visualizing, the gradient map might help us to understand why certain kernel works better than another one?
- Did you consider translation invariance or other spatial properties when designing your kernels?

In summary, this is an interesting paper with good empirical results. The technique being used generalization is quite straightforward, but the paper also includes a good amount of discussion on why the proposed approach could be better and I think that really helps the reader.


Hi, the ResNet-1001 was first proposed by kaiming he in [1].  Please refer [1] for more details. Thanks!

[1] He, Kaiming, et al. "Identity mappings in deep residual networks." European conference on computer vision. Springer, Cham, 2016.

Hi, I noticed that you used ResNet-1001 as the teacher model. It's astounding, considering its massive parameters. Could you please disclose more details on how you trained this extremely deep network? 

Meta-review:  The paper presents a sensible algorithm for knowledge distillation (KD) from a larger teacher network to a smaller student network by minimizing the Maximum Mean Discrepancy (MMD) between the distributions over students and teachers network activations. As rightly acknowledged by the R3, the benefits of the proposed approach are encouraging in the object detection task, and are less obvious in classification (R1 and R2).
 
The reviewers and AC note the following potential weaknesses:
(1) low technical novelty in light of prior works “Demystifying Neural Style Transfer” by Li et al 2017 and “Deep Transfer Learning with Joint Adaptation Networks” by Long et al 2017 -- See R2’s detailed explanations; (2) lack of empirical evidence that the proposed method is better than the seminal work on KD by Hinton et al, 2014; (3) important practical issues are not justified (e.g. kernel specifications as requested by R3 and R2; accuracy-efficiency trade-off as suggested by R1); (4) presentation clarity.  
R3 has raised questions regarding deploying the proposed student models on mobile devices without a proper comparison with the MobileNet and ShuffleNet light architectures. This can be seen as a suggestion for future revisions. 

There is reviewer disagreement on this paper and no author rebuttal. The reviewer with a positive view on the manuscript (R3) was reluctant to champion the paper as the authors did not respond to the concerns of the reviewers. 
AC suggests in its current state the manuscript is not ready for a publication. We hope the reviews are useful for improving and revising the paper. 

Decision: rejected-papers