Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose to diversify the models in an ensemble forming the teachers in distillation. The specific setup the authors consider is  on-line distillation (where the teachers are trained in parallel with the student) and peer-based (where the models share parts of their weights). 

The proposed approach to achieve the aforementioned diversification of the teachers is based on varying the prior distributions of the labels in the train data used for training each teacher. Then, as an efficient approach to this, importance sampling is employed to avoid sampling from the data separately for each teacher. Then, before averaging the teacher outputs for the student supervision, the "Post-Compensated Softmax" loss correction is employed from a previous work.

The comparison against selected baselines shows improvements. Ablations are provided where the authors measure the effect of the teacher's diversity and the number of teachers within the proposed framework. 
strength_and_weaknesses: Strenghs:
- The paper is mostly clear. 
- The Figure 1 is a very nice summary of the work.

Weaknesses:
- One limitation is that the setup seems very specialized (on-line distillation, peer networks). I don’t see why the technique is specific for that setup - diversifying the teachers (which is the main idea behind the authors’ method) could be well applied to off-line distillation or non-peer networks.
- The baselines are not explained. It is not clear why these are chosen over other options (there are a lot of distillation methods one could compare to).
- The ablation section is very useful to have, but I would like to see ablations for why using each of the steps suggested by the authors’ is useful. For example, does PC-Softmax  help? Is importance sampling actually good compared to sampling the data instead? (checking this even on a small dataset would be useful) What about the effect of the peer network choice in the light of your method, as opposed to using completely separate networks?
- When averaging the teachers, why not weigh them differently depending on what labels they were trained on? Since each teacher focuses on a different set of labels, they should be experts with respect to such labels. It seems that averaging the teacher's predictions is naive and ignores useful information about the teachers.
- As an application of label prior shift for diversifying the teachers, Why not try it in non-on-line distillation? Is there anything special about on-line distillation that diversifying the teachers is particularly interesting compared to off-line distillation? 

clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear and the method is explained well so it should be reproducible. The novelty is somewhat limited as methods from previous works are combined without a clear explanation why using them is the best option.

Some specific questions:
- What is the motivation for the label prior shift for training models in an ensemble? MoEs are mentioned, but I missed an explanation if that particular technique was employed there and what its success was. 
- “notably expected calibration error” – why is this finding notable? 

summary_of_the_review: The use of the methods described by the authors looks interesting and promising. I suggest conducting more ablations and adding explanations for the decisions to strengthen the paper before resubmission.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a framework for online knowledge distillation from an intentionally diversified ensemble of teachers. A common hypothesis for the success of distillation from an ensemble is that diverse teacher models increase the efficacy of the ensemble. The paper provides a formal framework to enhance teacher diversity by introducing a prior shift - effectively turning each teacher head into a “specialist”. The paper includes a robust empirical analysis and demonstrates that the proposed method significantly improves student performance with respect to both accuracy and calibration.
strength_and_weaknesses: Strengths:
+ The paper is written and structured well.
+ While many of the component ideas have been introduced before, incorporating such specialists in a complete framework of online distillation is novel and interesting.
+ The empirical analysis is thorough.

Weaknesses:
- The paper has a number of typos that are sometimes distracting. e.g. 
  - On page 3: “... both categories, using the extra modules as ever.”
  - On page 4: “... target distribution while only with samples …”
  - On page 18: “... optimal exposure differs from models…”

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and clear. As mentioned earlier in the review, while some / most of the component ideas such as label shift, PC-softmax, and the principle of enriching diversity in an ensemble for distillation have been observed before - the framework to put these together for online distillation is novel.
The experimental section is well-written and provides enough details for reproducibility.

summary_of_the_review: The paper proposes a new framework for online distillation with the intent to increase diversity among the teacher heads and demonstrates significant improvements over prior approaches in benchmark datasets.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a online knowledge distillation framework with specialized ensembles. It first uses a label prior shift to generate different teachers for student to learn from. Then it uses PC-Softmax to post-compensate teacher logits and an averaged classifier manner to aggregate teacher predictions. And the experiments part show the improvement over previous method on CIFAR and ImageNet.
strength_and_weaknesses: The paper is fairly well-written and the high-level problem is interesting. The writing is clear and easy to follow. And the experiment part especially ablation study is well designed to show the improvement over previous methods.
However, there seems lack of comparison on CIFAR with more current works like CGL [1] or PCL [2]. Also there is a most recent one L-MCL [3] that has better performance. And the performance over those method seems not very significant. (It seems more common to report acc than error rate. Using error rate makes it hard to efficiently compare with the unlisted models.)

[1] Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online
knowledge distillation via collaborative learning. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR), 2020
[2] Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. In
Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
[3] Yang C, An Z, Zhou H, et al. Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition[J]. arXiv preprint arXiv:2207.11518, 2022.
clarity,_quality,_novelty_and_reproducibility: The paper is clear in most parts---some ambiguities are discussed above.
summary_of_the_review: Overall, the paper proposes a new approach that uses specialized ensembles to do online KD. I would be happy to raise my score if the authors conducted the additional evaluations discussed in my review above.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Towards the ensemble-based KD, this paper first presents a label prior shift to induce evident diversity among the same teachers. Then the authors propose an aggregation strategy that uses post-compensation in specialist outputs and conventional model averaging. Experiments on several datasets validate its effectiveness.
strength_and_weaknesses: Strength:

(1) The motivation is clear, where this paper mainly aims to solve the issue of diverse teachers' construction.

(2) The organization is good, which makes the paper easy to follow.

(3) The authors also provide detailed proof for theorems and corollaries.

(4) Sufficient experimental analysis is given for better understanding.

Weaknesses:

(1) My main concern lies in the experimental results. Several methods achieve much better results than the proposed method on all these datasets. The authors compared with PCL on ImageNet, and it achieves a 0.05% improvement over PCL. I wonder why the authors do not compare with it on CIFAR10/100. According to the results on PCL, the results on CIFAR under various backbones are better than the proposed method. Moreover, one important reference[1] is missing, which achieves much better results than the proposed method, including ImageNet.

(2) As introduced by the authors, label prior shift has been extensively discussed by existing methods. Please analyze the difference between the adopted strategy and existing methods. Is this a simple combination with KD?

(3) Sampling has also been well studied. Please also compare the importance sampling with existing methods.

(4) Based on the above analysis, I think the novelty could be improved.

[1] MULTI-VIEW CONTRASTIVE LEARNING FOR ONLINE KNOWLEDGE DISTILLATION, ICASSP, 2021

clarity,_quality,_novelty_and_reproducibility: Please refer to my detailed comments above.
summary_of_the_review: The presentation and writing are good. But the novelty could be improved. And the experimental results are not convincing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a distillation strategy that leverages an ensemble of multiple teachers. These teachers are trained with different marginal label distributions. The resulting predictions are thus adjusted prior to ensembling, and the resulting average prediction is fed to a student model.

Reviewers were somewhat mixed on the paper, with concerns mostly on the empirical results and novelty. The response featured some new results, following which reviewer scores leaned more positive. The opinion of an additional domain expert was sought, which was a little more negative owing to the novelty and technical depth concerns. In my reading, I concur with these. The basic idea is interesting, and the paper is generally well written. The technical depth is a bit limited, with Section 3 essentially formalizing the strategy mentioned above. The ideas of importance sampling to construct the loss, and adjusting the logits to account for a label shift, are natural and fairly standard. It is also not immediately clear from the exposition what advantages the proposed scheme have over other means of constructing an ensemble of teachers; certainly the proposal generalizes training models on different subsets of classes, but are there deeper reasons to favour the proposal over, say, input clusters? 

Overall this is certainly a decent paper, but is just a bit below the bar at present.

Minor comments:
- The idea of constructing specialist models in distillation is also explored in Section 4 of Hinton et al., 2015.
- Theorem 1 is an elementary consequence of the importance weighting identity. I would just state the result rather than wrap it in a theorem. Certainly Theorem 1 is overkill, and potentially a bit misleading unless one clearly states this is a standard result.
- Equation 2, use \left[ and \right].
- "manually adjusting logit values (Ren et al., 2020)" -> the cited paper changes the loss function. It does not appear to adjust logit values as stated.
- Appendix C.3 seems to present a new theoretical result that is not referenced in the body. This is not advisable.

justification_for_why_not_higher_score: Limited technical depth -- while a simple and intuitive idea, the key components are fairly standard in studies on label shift. The novelty appears to be in their application to the distillation setting, for which a more detailed discussion of why this is preferable to other ensemble generation strategies would be useful.
Decision: Reject