Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: The paper attempts to provide insights on how superior model distillation is related with strong correlation between classifier layers of student model and the teacher model. The claims and insights are well validated using references and experiments.
strengths_and_weaknesses: Strengths:
1. The paper is well written.
2. It shows a strong correlation between the student classifier and the final classifier layer of the teacher model.
3. Interestingly, it shows that the strong correlation is very specific to the distillation and doesn't exist for normal training.
4. The insights obtained may be helpful in developing better distillation strategies.

Weaknesses:
1. The novelty is very limited, specially as we already knew that the teacher classifier is pivotal for achieving effective distillation (from previous studies Yang et al. (2021) and Chen et al. (2022) ).
2. The experiments and evaluations may be limited to reach a concrete conclusion as samples from only one dataset CIFAR 100 are used.
3. Different architectures could have been tried (ViT, InceptionNet etc.) rather than using two variants of ResNet to be more confident about the obtained results and conclusions.
suggested_changes: 1. There are a lot of typing errors (e.g. $||w_t-w_t||$ will probably be $||w_t-w_s||$) making it hard to interpret the meaning at times. Please correct them.
2. If authors can repeat the experiments for a different dataset and a different teacher model architecture, it may provide indication if the results are generalizable.
review_title: null

Meta-review: metareview: This paper investigates the effect of knowledge distillation on the alignment between weights of student and teacher classifier models, inspired by recent work showing that directly matching representations can lead to strong distilled models. The authors train models using KL-divergence knowledge distillation at a variety of temperatures, then compute alignments between the weights of the last layer of the teacher and student models.

As Reviewer gNx2 notes, the paper is well written, and the findings about increased alignment over time are interesting. The authors also provide details about how to reproduce their experimental results.

The main weakness of this submission is that the experimental results may not be strong enough to fully support the authors claims.

- Reviewer gNx2 notes that the results are only presented for a single dataset and model architecture, and may not translate to other settings.
- Additionally, although the authors report a strong correlation between the weight difference $\mu$ and the training loss $\mathcal{L}$, Figure 1 shows that the size of the effect itself is very small; the L2 error between the weights only seems to change by about 5%, and plateaus well before reaching strong alignment between teacher and student weights. The direction of the effect also seems to change depending on the temperature.
- The authors note that $\mu$ fluctuates before epoch 120, and thus drop these epochs from the comparison. It's possible this may lead to an overestimate of the size of the effect, since dropping outliers is a "researcher degree of freedom" that can lead to false-positive results if used to confirm a previous hypothesis

There are also a number of minor typos and writing errors. In addition to the one noted by Reviewer gNx2, I noticed a stray "=" on page 1, and the caption of Figure 2 seems incorrect.

Overall, this seems like a clear and well written paper, but I think it's claims are a bit too strong at the moment. I'd suggest weakening the empirical claims and acknowledging some of the limitations of the analysis to ensure that the claims in the paper are fully supported.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Invite to archive