Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary & Pros
- This paper proposes a well-principled distillation method based on contrastive loss maximizing the mutual information between teacher and student models.
- This paper provides extensive experiments that demonstrate the effectiveness of the proposed method. The performance gap compared to the existing distillation approaches seem to be significant.

Major Concerns:
- The authors claim that "none of the other methods consistently outperform KD on their own". I feel that this claim is somewhat aggressive because some of them outperform KD without combining with KD, e.g., Table 1 in FT (Kim et al., 2018) and Table 2 in SP (Tung & Mori, 2019). Since the distillation (or transfer) methods are typically sensitive to hyperparameters (e.g., architecture types, transfer connections between layers), I also wonder how to set the hyperparameters for baselines, especially in Table 2, because choosing the transfer connections between different architectures is very important when using feature-based methods such as FitNet and AT.
- Moreover, the baselines are developed for improving distillation performance, not replacing KD. Especially, feature-based methods (e.g., FitNet, NST) are easily combined with logit-based ones (e.g., KD). So I think the compatibility between the proposed and exisiting methods should be checked. However, in this paper, only Table 4 shows the compatibility with KD (CRD+KD).
- The authors compare the proposed method with only KD, AT, FitNet except Table 1-3. For example, when evaluating the transferability (Table 4), why other baselines are not compared?
- VID also maximizes MI between penultimate layers. What is the key difference and why CRD perform better? I think detailed verfication should be provided in the paper.

Minor Comments
- Comparison with Online-KD is unfair because it does not use pre-trained ResNet32.
- Why only use penultimate layers? CRD between intermediate layers is also available like VID.
- A result in Table 1 is missing (FSP WRN40-2 -> WRN40-1).
- In Section 4, both CKD (contrastive knowledge distillation) and CRD (contrastive representation distillation) are used, so one of them should be removed.
- In Section 4.1 Transferability paragraph, "Table 3" should be changed to "Table 4".
- On page 4, "space" after "before the inner product." should be removed.

I think the proposed method is well-principled and provides meaningful improvements on various distillation settings, thus this paper seems to be above the borderline. It would be better if additional supports for the above concerns is provided in a rebuttal.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper combines a contrastive objective measuring the mutual information between the representations learned by a teacher and a student networks for model distillation. The objective enforces correlations between the learned representations. When combined with the popular KL divergence between the predictions of the two networks, the proposed model shows consistently improvement over existing alternatives on three distillation tasks. 

This is a solid work – it is based on sound principles and provides both rigorous theoretical analysis and extensive empirical evidence. I only have two minor suggestions.

1, From Section 3.2 to Section 3.4, it is not clear to me that on the model compression task, are both the proposed contrastive loss and the loss in Eq. (10) used?

2, The “Deep mutual learning”, Zhang et al, CVPR’18 paper needs to be discussed. I’d also like to see some experiments on the effects of training the teacher and student networks jointly from scratch using the proposed loss. 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: I do not necessarily see something wrong with the paper, but I'm not convinced of the significance (or sufficient novelty) of the approach. 

The way I understand it, an independent assumption on internal representation is relaxed by capturing correlations between them and higher order dependencies between them using a different objective function(distance measure), the problem then becomes nothing but trying to minimize another distance metric between teacher and student networks on an intermediate layer. 

Comparing with original distillation method (KD) I'm not sure how significant the improvement is. And technically this is just a distance metric between the internal representation or output of the student and teacher. Sure it is a more involved distance metric, however it is in the spirit of what the distillation work is all about and I do not see this as being fundamentally different, or at least not different enough for an ICLR paper.

The experimental results also suggest only a marginal improvement compared to other methods. It would be helpful to also include the variance of each experiment i.e., it was mentioned that the results were averaged by repeating 5 experiments to make sure the proposed method consistently better than others.  Rightnow, It is hard to compare with other approaches. The paper gives off a feeling that this method is not novel. Why this particular distance metric between the representations? Why not just L2?

review_title: Official Blind Review #3

Meta-review: This paper presents a new distillation method with theoretical and empirical supports.

Given reviewers' comments and AC's reading, the novelty/significance and application-scope shown in the paper can be arguably limited. However, the authors extensively verified and compared the proposed methods and existing ones by showing significant improvements under comprehensive experiments. As the distillation method can enjoy a broader usage, I think the propose method in this paper can be influential in the future works.

Hence, I think this is a borderlines paper toward acceptance.
Decision: Accept (Poster)