Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Overall the method proposed in this paper is simple but effective, and adequate experimental results are given to show its performance improvements.  However, the literature survey of this paper is not satisfactory.

1. To reduce model size, there are several different ways including efficient architecture design, parameter pruning, quantization, tensor decomposition and knowledge distillation. The authors forgot to mention tensor decomposition and mixed it with efficient architecture design. As for parameter pruning and quantization,  many important papers are missing.

2. Utilizing the "soft targets" to transfer knowledge from teacher to student model is not first proposed by Hinton et al. (2015). To the best of my knowledge, it is first proposed in  
J. Li, R. Zhao, J.-T. Huang, Y. Gong, “Learning small-size DNN with output-distribution-based criteria,” Proc. Interspeech-2014, pp.1910-1914.

3. Leveraging back part of teacher model's guidance to improve student performance has been investigated by other researchers on OCR tasks in 
Ding H, Chen K, Huo Q. Compressing CNN-DBLSTM models for OCR with teacher-student learning and Tucker decomposition[J]. Pattern Recognition, 2019, 96: 106957.
They combine student's CNN with teacher's DBLSTM to learn better representations.

In conclusion, I will give a weak reject currently, unless the authors improve their literature survey and modify their claims.




review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: 
This paper proposed a new method for knowledge distillation, which transfers knowledge from a large teacher network to a small student network in training to help network compression and acceleration. The proposed method concatenate the first a few layers of the student network with the last a few layers of the teacher network, and claims the gradient directly flows from teacher to student, instead of through a KL or L2 similarity loss between teacher and student logits. 

The experimental results look good, and extensive experiments have been done on CIFAR, ImageNet and PASCAL VOC. 

However, the description of the proposed method looks rather unclear. First, the ‘front’ and ‘back’ part of networks are very vague. I have to guess that is the first a few layers of student and last a few layers of teacher. And it is still unclear how many layers in student and teacher are concatenated to form the ‘collaboration network’. How could the authors connect the two subnetwork with different structures?

It is unclear to me why proposed method is better than AT, FT or FitNets. It looks to me the proposed method use an ad-hoc selected layer to transfer knowledge from teacher to student, and the transfer is indirect because it has to go through the pre-trained subnetwork in teacher.

Minor issue, FT and AT are not defined when they first appear in page 1. 

Could the authors show the student and teacher accurayunder standard supervised training in the result tables?

Several related works are not discussed, such as
Xu et al. 2018 https://arxiv.org/abs/1709.00513
Belagiannis et al. 2018 https://arxiv.org/abs/1803.10750
Wang et al. 2018 https://papers.nips.cc/paper/7358-kdgan-knowledge-distillation-with-generative-adversarial-networks


============ after rebuttal ================
I updated my rate to weak accept. Though it is a borderline or below paper to me. The paper has really good empirical results. However, I cannot understand the intuition behind the paper why concatenating teacher to student is better than use l2 for intermediate layers. The choice of the transferring layer seems to be rather ad-hoc, and it is hard to say how much tuning needed to get the empirical benefits.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The paper suggests a new method for knowledge transfer from teacher neural network to student: student-teacher collaboration (STC). The idea is that teacher not only provides student with the correct answer or correct intermediate representation, but also instructs student network on how to get the right answer. The paper suggests to merge the front part of the student network and back part of the teacher network into the collaboration network. In this way, the weights in the student subnetwork are learned from the loss backpropagated from the teacher subnetwork. The target labels for collaboration network are outputs of teacher network. The method is adapted for different tasks, classification and bounding box regression are presented in the experiments. It outperforms previous methods on various datasets. Furthermore, the method shows good results when integrated with traditional knowledge distillation.

Overall, the paper is a significant algorithmic contribution. The related work section provides thorough review of different methods to decrease computational costs, including not only knowledge distillation, but also pruning, compressing and decomposition approaches. The idea is elegant and, to the best of my knowledge, has never been suggested in other works. Considering the theoretical part, it is clearly shown how the gradient signal from the teacher sub-network guides the student network on which part of the weights should be paid attention on. All derivations are easy to follow. The paper also considers how the suggested idea is aimed to solve the problems of previous knowledge transfer methods. The experimental section is consistent and clearly shows the advantage of the suggested method. Teacher and student networks used are different sizes of ResNet, Wide ResNet and VGG. The paper presents classification experiments on CIFAR-10, CIFAR-100, ImageNet and object detection experiment on PASCAL VOC 2007 dataset. STC outperforms previous methods, both with KD integration and without. The performance is always better than pure student training (which was not always the case for previous methods) and sometimes the results are even better than teacher performance. Finally, the choice of teacher output as target over soft target and ground truth, which was previously motivated in the theoretical section, is shown to be superior in the experiment.

Possible improvement of the paper is the instruction on how to choose the intermediate layer from where to teach the representation, i.e. where the student sub-network ends and teacher sub-network begins. For object detection experiment the choice of the border is naturally defined by the architecture of the network in Faster-RCNN approach. Could the choice be different? May be somewhere inside the BackBone part of the networks? For classification, it could be interesting to study how this choice influences the results. However, this question didn’t affect my score of the paper, and, as far as I know, it is also not considered in the previous works on knowledge distillation.

Minor comments
1.	In the context of accelerating the models using decomposition, Lebedev et al., ICLR 2015 could be cited.
2.	Page 2: difference tasks -> different tasks
3.	Page 2 the first bullet point: additionally utilizes -> additionally utilizing/which additionally utilizes
4.	Page 2 the third bullet point: brings good generalizability -> which brings good generalizability
5.	Page 5: “training strategy is more accelerate than” – odd phrase
6.	Page 6: while KT has conflicts with KD in some cases -> while FT has conflicts with KD in some cases.

review_title: Official Blind Review #3

Meta-review: This paper has been assessed by three reviewers scoring it as follows: 6, 3, 8. The submission however attracted some criticism post-rebuttal from the reviewers e.g., why concatenating teacher to student is better than the use l2 loss or how the choice of transf. layers has been made (ad-hoc). Similarly, other major criticism includes lack of proper referencing to parts of work that have been in fact developed earlier in preceding papers. On balance, this paper falls short of the expectations of ICLR 2020, thus it cannot be accepted at this time. The authors are encouraged to work through major comments and resolve them for a future submission.
Decision: Reject