Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a new online distillation technique enabled by a novel bi-level formulation that is based on a closed form expression. The bi-level formulation results in a weighting of the peers according to importance thus enabling optimized training of the entire network of peers unlike prior works that depended on approximations. The proposed method enables systematic pruning of the networks which results in significant improvement over the state of the art in terms of accuracy given a certain number of flops.
strengths_and_weaknesses: Strengths:
1. Originality and Significance: The paper's proposed closed form approach is a clear advancement of the state of the art and is thus both original and significant.
2. Quality: The proposed closed form approach provides a compact and systematically tunable framework that achieves much better than the state of the art performance. The quality is therefore high.
3. Clarity: The paper is very well written. It provides a broad overview and then covers the state of the art in an insightful way and motivates the proposed technique. The technique is itself well described and the results are presented clearly as are the conclusions. The clarity is high.

Weaknesses
1. Having a closed form expression is of course impressive and handy, but in most cases a Taylor approximation provides a powerful approximation. So what accounts for the great improvement over the state of the art? 
2. How exactly the closed form expression is  being used needs to be explained better.
3. The above are minor weaknesses. The overall quality of the paper far outweighs these "weaknesses."
questions: Having a closed form expression is of course impressive and handy, but in most cases a Taylor approximation provides a powerful approximation. So what accounts for the great improvement over the state of the art? 
limitations: The authors have pointed out that to maintain a bounded investigation they have refrained from incorporating conventional pruning techniques which they say they will adopt in future work to get even better results.
That is the one discussion of limitations that I could find.

There is no mention of social impact, positive or negative. I would suggest that the proposed fine-grained optimization will have only positive social impact because it will help reduce complexity in a systematic manner, thus more energy efficient and compact devices can be constructed which is a net positive for society.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposed an online-distillation framework called Weighted Mutual Learning with Diversity-Driven Model Compression (WML). The authors Introduce a hierarchical structure in which peer networks share different parts to reduce memory consumption. A structured pruning method based on SNIP is adopted for the diversity of peer students. They use a dynamic weighting method to determine the importance of the pruned peers and formulate it as a bi-level problem. 
strengths_and_weaknesses: -- The paper is clearly presented, and the main ideas are easy to follow.

-- The two difficulties of online distillation: memory consumption and model diversity, as the proposed framework tries to solve, are important and meaningful. The authors conduct several experiments to prove the effectiveness.

-- Structured network pruning is used to generate peer networks in the hierarchical structure. This combines pruning and online distillation. And to dynamically decide the weight of each peer rather than take the average, the authors formulate it as a bi-level optimization problem.
questions: --- The contributions are the memory-friendliness and diversity of student peers; they respectively benefit from a hierarchical structure where peers share different parts and structure pruning. As mentioned in the original paper, the hierarchical architecture is adopted from [1] and the pruning criterion is SNIP[2]. Neither of these points is original, and the innovation of this paper needs more clarification.

-- Though self-distillation can be regarded as a special case of online distillation, it is different from the mutual learning method in this paper since it only uses its own structure as a teacher. Instead of self-distillation, the comparison with online distillation using multiple peers should be of more concern. In this paper, only DML[3] and ONE[4] are compared without more updated methods. This makes a claim in line 66 "significantly outperforms existing online distillation and self-distillation methods" less convincing. I think Table 3 should be expanded and highlighted with more comparisons with the latest online distillation approaches.

-- The authors mention the use of structured pruning to enhance diversity, but due to the use of a partially shared hierarchical structure, diversity is limited compared to other online methods that can use completely different networks. It seems to me that the advantage of structural diversity here is only in comparison with self-distillation methods or [1].

-- The diagram in Figure 1 (b) is ok, but it could easily be confused with self-distillation. It would be more representative of online distillation if it were changed to different peer networks.

[1] G. Song and W. Chai. Collaborative learning for deep neural networks. NeurIPS, 2018.
[2] N. Lee, T. Ajanthan, and P. Torr. Snip: Single-shot network pruning based on connection sensitivity. ICLR, 2019.
[3] Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu. Deep mutual learning. CVPR, 2018.
[4] X. Zhu, S. Gong, et al. Knowledge distillation by on-the-fly native ensemble. NeurIPS, 2018.
limitations: Yes，it's oK.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper tackle the problem of online distillation using weighted mutual learning, where structured network pruning is leveraged to reduce the memory consumption. Extensive experiments on several neural networks and three datasets show this method outperforms existing online distillation and self-distillation methods.
strengths_and_weaknesses: Strengths:
1. The idea is interesting and can be applied in many tasks.
2. Experimental setups are proper and there are sufficient experiments.


Weaknesses:
1. The motivation of the method is not described clearly. For example, why do you propose to use several peer models to share different parts in a hierarchical structure in Figure 1? If the motivation does not described clearly, the method seems to be incremental.
2. The improvements are marginal. In Table 1,2,3, most of the accuracy gaps between WML and SCAN/SD are within 1%, which is marginal. Did the authors run the experiments for several different times and take the means and standard variances?
3. Tables need to be improved. More detailed descriptions should be added in the table caption. For example, the values in Table 1 and Table 3 are accuracy (%) right? The authors can clearly mention that.
questions: Q1: There is nothing in the supplementary material. Did the authors submit a wrong file?

Typo:

Table 1 ResNet 101->WML->Model 1: "79.6" -> "79.60".
limitations: See weaknesses.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces the structured network pruning and bi-level formulation to online distillation and shows its effectiveness in performance and efficiency. On the one hand, this paper proposes to prune unimportant channels in peer models, reducing the memory requirements. On the other hand, this method leverages a new formulation to estimate the relative importance of peer models, and improve the effectiveness of online distillation. Experimental results show that the proposed method outperforms a variety of methods in knowledge distillation and channel pruning. 
strengths_and_weaknesses: Pros:  

- The paper is very clearly written and easy to understand.  

- The proposed method reduces the memory cost of online knowledge distillation and maintains competitive performance on a variety of deep neural networks.

- This paper makes a comprehensive ablation study on the proposed method.   

-----------
Cons:  

- This paper proposes to have improved the diversity of peer models but does not analyze how the proposed method impacts the diversity of all peer models deeply. It is necessary to discuss it in the experiment section.   

- To my understanding, SNIP is an unstructured pruning method that prunes out the unimportant connections. I think the authors should consider other competitive structured pruning methods rather than just borrowing SNIP for structured pruning.  

- Most of the selected competitors have been proposed for a long time and have a better alternative. The superiority presented in the experiments is not very convincing. 
questions: - How does the proposed method compare with the newly proposed methods in knowledge distillation and channel pruning?

- How does channel pruning contributes to the diversity of peer models? 

- Why choose SNIP as the pruning method? Does it perform much better than others?
limitations: Yes
review_title: null

Meta-review: metareview: This paper proposes an online distillation technique: Weighted Mutual Learning with Diversity-Driven Model Compression (WML). It uses a novel bi-level formulation that estimates the importance of the peers thus enabling optimized training of the entire network of peers. The proposed method got significant improvement over the state of the art in terms of accuracy. The proposed idea is novel and could be used in many tasks.


confidence: Certain
Decision: Accept