Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: = Summary
This paper presents a new deep mutual learning (i.e., online peer-teaching) method based on Knowledge Distillation (KD) in a feature map level. The target task is similar with the original KD in the sense that the a network is taught by another network as well as groundtruth labels, but different with the KD in the sense that the networks are not a (frozen) teacher and a student but teaching each other in an online manner. Most approaches in this relatively new line of research rely on logit-based KD for transferring knowledges between networks, and the paper demonstrates that by an additional feature map level KD the performance can be further improved.


= Decision
The current decision is borderline in my mind, but officially weak accept. Although the proposed method is simple and consists of known ideas, it is designed convincingly and enhances performance practically. Also, I believe the target task itself is worth to be introduced as a next direction of KD. However, the submission is weak in terms of novelty and the manuscript should be polished carefully.


= Comments
1) Weak clarity
- The main motivation and advantages of the deep mutual learning is not well introduced in Section 1. Although the two papers (i.e., ONE and DML) are cited here, it would be much better to explicitly describe the main idea and motivation of peer-teaching, what is the difference between the task and the original KD, and the achievements in the previous work. Without these, readers, including me, may get confused why the online KD is required and why there is no clear teacher-student relationship between networks. 
- In a similar context, the motivation of introducing more than two student networks should be given.
- The architecture of the discriminator seems not described even in the appendix. 
- The meaning of various arrow types in Figure 1 is not clearly described.

2) Insufficient experiments
It would be better to report the performance of vanilla and (offline) KD in Table 1 to show more clearly that the feature map alignment is useful and that online KD is better than its offline counterpart.

3) Limited novelty and performance improvement
- The main idea is already introduced in previous work on the task and the feature map level KD has been studied widely for various applications, their combination is somewhat new though.
- The performance improvement by the proposed feature map level KD seems marginal as shown in Table 2.
- The performance gap between DML and the proposed model seems also marginal.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: A new online knowledge distillation is investigated by utilizing feature map information next to the logits via GAN. Instead of direct feature map alignment, the algorithm tries to transfer the distribution of the feature maps. There is no teacher per se, but the big and small nets are trained via an adversarial game where 2 discriminators try to minimize the distributions of the two nets. The idea is understandable but some issues remain:
1-	Training GAN is by itself an expensive task and optimization is difficult, so how computationally expensive is this online KD compared to the offline one?
2-	It is not clear form the paper feature maps from which layers are being used? If multiple layers are considered, how did you choose which ones are better. Also the smaller model has a different structure, h ow did you choose to pair feature maps in the big model and the small model?
3-	What would be the performance difference compared to offline knowledge distillation? For example in Table 1 can you please add a column with offline KD?
4-	Mutual training brings some generalization, and when you compare the results in Table 3 with vanilla model, I am wondering if you made sure this is not only due to a better generalization.
5-	In cyclic learning framework, it is not well-motivated why someone wants to train multiple networks that mimic each others’ behavior; also the complexity increases in that case which makes me wondering wouldn’t it be better to do it offline then?

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors study the online knowledge distillation problem and propose a method called AFD (Online Adversarial Feature map Distillation), which aims to transfers the knowledge of intermediate feature map (first propose) using adversarial training. Then, a cyclic learning scheme is proposed to train more than two networks simultaneously and efficiently. Ablation study on CIFAR100 shows that the adversarial training in AFD can improve the accuracy significantly, while the direct method such as using L1 distance is worse. The comparison experiments with several online distillation methods also show the effectiveness of proposed method.

Some comments or suggestions:
(i) The theoretical analysis is lacking. For example, some formulas proofs can be added to illustrate that the adversarial feature map distillation is more advantageous than the direct feature map alignment.
(ii) The details of the experiments such as parameter configurations are missing, which makes the results not easy to be reproduced.
(iii) Tab.1 and Tab.2 can be combined. 

review_title: Official Blind Review #1

Meta-review: The paper received scores of WR (R1) WR (R2) WA (R3), although R3 stated that they were borderline. The main issues were (i) lack of novelty and (ii) insufficient experiments. The AC has closely look at the reviews/comments/rebuttal and examined the paper. Unfortunately, the AC feels that with no-one strongly advocating for acceptance, the paper cannot be accepted at this time. The authors should use the feedback from reviewers to improve their paper. 
Decision: Reject