Learnable Relational Knowledge Distillation Through Orthogonal Projection
Authors: hufeng, Kai Zhang, Ye Liu, Meikai Bao, Xukai Liu, Junyu Lu, Linbo Zhu, Qi Liu


Reviewer paper summaries: This paper proposes a straightforward method for feature-based distillation by introducing the orthogonal projection. It can keep the similarity of features within a batch without distorting the underlying student representation. In the training stage, 
The orthogonal projection autonomously learns the relational features between the teacher and student model. A theoretical analysis and the projection across different layers are provided. Extensive experiments are conducted to verify the performance of the proposed loss function.
Summary of strengths: - This paper works on a practical problem in the relational knowledge distillation.
- The performance of this paper looks promising.
- This paper is clear and easy to follow.
Summary of weaknesses: - Similar ideas of using orthogonal projection in knowledge distillation has been proposed.
- The motivation of using the orthogonal projection is not well-motivated.
- Typos and grammar issues.
Comments, suggestions and typos: Comments:

- One major concern about this paper is that similar ideas of using orthogonal projection in knowledge distillation have been proposed in [1]. This paper should distinguish itself with the mentioned paper, and also conduct experimental comparison.

- The motivation of using the orthogonal projection is vague. More illustrations are supposed to be provided to demonstrate the motivation. Some toy examples are also preferred.

- This paper does not compare to the most recent paper in knowledge distillation like [1] and [2].

[1] Miles et al., Improving Knowledge Distillation using Orthogonal Projections, CVPR’24.

[2] Sun et al., Logit Standardization in Knowledge Distillation, CVPR’24.

Typos and grammar issues.
1. In the title of 4.2, “The” should not be capitalized.
2. On Line 444, there should be a comma ahead of “respectively”.
3. On Line 448, the value of the hyper-parameter should be put into the brackets.
Overall assessment scores: 2.5