Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper takes a step toward understanding ensemble and knowledge distillation. The authors consider the challenging setting where the teacher model is an average of several models of the same structure, or even the teacher has an identical structure as the student model. The authors developed a theory that the distillation from several independently trained neural networks can improve the performance when the data has a "multi-view" structure. 
strength_and_weaknesses: Strength:

1. The authors present theoretical results showing that a single model is guaranteed to have 0 training error while having a high testing error with high probability; the ensemble can provably improve the testing accuracy.

2. The author shows that the ensemble can be efficiently distilled into a single model. This understanding is fundamentally different from the standard NTK settings.

3. The idea of "multi-view data" is intuitive and provides a natural and convincing explanation for various empirical observations for distillation.

Questions and weaknesses:

1. One major concern is that the supplementary material cannot be opened due to a file format error, and I'm not able to see the results of self-distillation and empirical evaluation, which seem to be important aspects of this work.

2. In the example of Section 2.3 and for a single model, why the network can only pick up one of the features in $v_1, v_2$ with the 90% of the data, and only memorize the remaining 10% of the data, instead of learning from the other relevant feature in $v_1, v_2$?


clarity,_quality,_novelty_and_reproducibility: The idea of "multi-view" is interesting, and provides an intuitive explanation for various observations in distillation.
summary_of_the_review: This paper presents an interesting and intuitive explanation for distillation. However, one major concern is that I'm not able to open the supplementary material due to some file format error. I'm not able to fully assess this paper.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: This paper studies the theoretical explanation of why the ensemble and knowledge distillation works for the deep learning model. It shows the different behavior of classic parameterization of neural networks and the random feature model (e.g. NTK). It then explains via the idea of multi-view data that the network infers via generalizable learning of a subset of features and memorizing another subset of data. Such analysis differs from the classic analysis that decomposes optimization and generalization.

Overall, I find the insight of the paper quite interesting, and such intuition is well supported by the empirical evidence using the practical network. The construction of the multi-view data for analysis is closely related to the practical settings.

strength_and_weaknesses: Strength:

1. The novelty of the paper is quite strong. I believe the such analysis is new and sound.

2. There is quite a good match between theory and practice, especially the construction of the multiview dataset is an exciting and meaningful proxy for the practical case while being theory-friendly.

3. The writing of the main context is well organized. It is easy for the reader to get to know the most critical intuition of the theory.

4. I like the experiment used in this paper. It is concise but well supports the theory.

Weakness:

1. I am not able to open the supplementary of the paper but reviewing this paper, reading the appendix is very necessary and thus my reading of the appendix is via the arxiv. The structure of the appendix is good but I feel a more simplified overall technical/intuition is desirable.

2. To obtain such a theory, there are still a lot of 'artificial' in the problem setup such as the use of smoothed ReLU, and specific restrictions of the data distribution. Those slightly increase the gap between theory and practice. But this point does not downgrade my point, understanding deep learning in theory is very hard and this paper has done excellent work toward that.

clarity,_quality,_novelty_and_reproducibility: Clarity: Good
Quality: Good
Novelty: Godd
summary_of_the_review: Questions:

1. Almost all the result suggests a 100% training accuracy but this is different from almost 0 cross-entropy loss. I feel all the results will not hold if the model is large and trained with almost zero cross-entropy loss? At least, in this case, the distillation via logits seems to be no more useful. Pls correct me if I am wrong. 

2. Following 1, how do the results look like if we consider regression rather than classification?

3. Following 1 and 2, I was curious if we do distillation on the latent feature space rather than logit space, will the result become very different? From the feature learning perspective, distill on the feature space seems more reasonable?
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper studies the theories of ensemble and knowledge distillation. Insightful studies regarding NTKs and deep neural networks are performed and novel theoretical results based on the multi-view structure of data are established. I get many new insights after reading this paper.
strength_and_weaknesses: # Strength
- The writing and presentation are good. It is a pleasure to read this paper. 
- The empirical results on neural tangent features and deep neural networks are novel and interesting. The authors then provide thorough explanations for them, which is appreciated.
- The theoretical results are rigorous.

# Weaknesses
- We know new theories usually motivate new algorithms. I want to know if there are some potential future works to better exploit these theories. 
clarity,_quality,_novelty_and_reproducibility: All aspects are good.
summary_of_the_review: This is an outstanding paper in multiple aspects, so I suggest acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigated the superior performance of ensemble, knowledge distillation and self-distillation in the field of deep learning. As previous principles of ensemble on traditional machine learning algorithms do not apply to deep learning methods, a novel perspective of "multi-view" data was proposed to explain the phenomenon. Both empirical and theoretical analysis support multi-view theory for ensemble in deep learning.
strength_and_weaknesses: Strengths:
- The paper analyzed an underexplored problem of how ensemble works in deep learning methods. It pointed out contradictions when applying previous principles of traditional algorithms and shed light on understanding deep learning.
- For the theory side, the analysis was rigorous with clear definitions and concrete examples to help better digest the theorems.

Weaknesses:
- I am just curious whether the hypothesis of multi-view data structure can be extended to different data structures such as texts and graphs. Those data are very different from image data discussed in this paper and 'multi-view' might not be applicable to them.
- It is not a good practice to include "self-distillation" in the title but defer all details about it to the appendix, which was somewhat misleading even though it was due to page limit. 
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is clearly written and well organized. As it was primarily a theory paper, it was fine to include few empirical results in the main paper. The multi-view data structure was novel to the community and made the first attempt to interpret how ensemble and knowledge distillation works in the field of deep learning. In addition, the theoretical analysis was rigorous except for some minor typos in Contradiction 2 in page 2, where $f_i$ and $g_i$ were used mixedly.
summary_of_the_review: The paper made an attempt to explain the superior performance of ensemble, knowledge distillation and self-distillation with the theory of "multi-view" data structure and provided insights on understanding those techniques in deep learning. The analysis is rigorous and thus I tend to accept it.
review_title: null

Decision: Accept: notable-top-5%