Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors apply knowledge distillation on vision transformer to learn a more efficient one, so that it could be  deployed on edge devices like cell phones and smart watches. 

Considering the knowledge distillation has been fully investigated in neural networks,  the authors utilize the patch-level information and propose a fine-grained manifold distillation method. 

To achieve it, the authors use a kind of manifold learning-based KD method, which supports mismatched embedding dimensions. And to utilize the patch-level information in VIT , the authors propose a fine-grained manifold distillation method. The aim is to teach the student layer to output features having the same patch-level manifold structure as the teacher layer.

And in order to simplify the computation, the authors decouple a manifold relation map into three parts, which are intra-image relation map,  inter-image relation map and  randomly sampled relation map.

Together with the path merging strategy, the computation complexity can be further reduced.
strengths_and_weaknesses: First of all, the motivation of this paper is clear. The authors want to do knowledge distillation to learn a smaller and more efficient vision transformer. And as it is applied in the vision transformer architecture, the authors consider to preserve the relationships among patches.
So, the afterwards proposed schemes are just reasonable. There are mainly two strategies, both of which are proposed to reduce the computation complexity. One is to use three kinds of maps to replace the manifold relation map. The other is to merge patches.

So the weakness of this paper comes, that it doesn't surprise me a lot. It is just looked like a normal solution. In distillation, the manifold map is like a kind of structure information in the pixel-level tasks.

To be more specific, I wonder:
1. If we only use the patch merging strategy, it could reduce the computation complexity a lot. So what is the necessarily to uncouple the map?
2. And when uncoupling the map, what is the exact meaning of the random sampled relation map. I think that the first two kinds may work.
3. And in equation 10, there introduces three weights for balancing each item. When it is applied in each block, I think choosing the weights for each block is a complex task.
4. In line 120-121,  the authors say the teacher-student layers are manually selected, is it reliable enough?
questions: see weakness
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper works on improving manifold learning-based knowledge distillation when using lightweight vision transformer models. The authors proposed a patch-level fine-grained manifold distillation method by matching the intermediate features between student and teacher in a manifold space and further decoupled the matching loss to reduce the computational complexity.

The proposed approach is evaluated on ImageNet-1K classification task and COCO2017 object detection task and shows better performance than other baselines. Ablation studies are done on the four design components.
strengths_and_weaknesses: Strengths: The evaluation and ablation study is relatively comprehensive, covering different downstream tasks and proposed design components.

Weakness: The efficientness of the proposed method is not well supported due to the lack of discussion/details/evaluation on common public benchmarks. 
questions: Is there a trade-off between the scheme for reducing computational complexity and the performance on downstream tasks? More explanation and reasoning should be done here.
limitations: More quantitative evaluation and examples should be provided in the Complexity analysis section.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors found the existing vanilla distillation in ViT can not work with different embedding dimensions between the teacher and student model. Moreover, the authors argue previous ViT distillation process is conducted at a coarse level.
To resolve the above issues, they propose a fine-grained patch-level manifold distillation method. 
By decomposing the distillation process into inter and intra parts, the authors found the results have better accuracy with negligible computational overhead. 
strengths_and_weaknesses: 1. originality:
Manifold learning-based KD, which does not require matching dimensions among teacher and student networks, was first implemented in ViTs.
To reduce the computational complexity, the relation maps are decomposed into three parts for efficiency.
2. quality:
The paper is well written, with detailed experiments and ablation studies with other methods and the proposed variants.
3. clarity:
The paper consists of an illustration of the decoupled manifold relation map, text explanations of the proposed methods.
4. significance:
The paper solves the problem that distillation of different hidden dimensions in ViTs, reduces computational cost than its baseline, and improves the distilled accuracy of previous ViT+distillation methods.
questions: 1. Why does the Soft Distillation (3rd row) in Table 3 improve only 0.3% compared with the 2nd row?
2. The proposed manifold-based KD still belongs to a subset of KD, but why the authors separate the proposed MD from the original KD, in the caption of Table 2?
3. I appreciate the authors admit the failure of distillation with L_inter, but why do not you re-run the experiment and report a new number? 
4. Why missing semantic segmentation experiments as an extension of the downstream tasks for the proposed new KD?
limitations: The authors should add discussions about societal impact. 
For other limitations, please refer the question part.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a fine-grained patch-level distillation method for vision transformers. Specifically, the proposed method matches the student and teacher models in the patch-level manifold space. The proposed method further decouples the relation map into three terms to solve the high computational complexity brought by the patch-level distillation. Experimental results on various vision tasks and benchmarks demonstrate the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths:
1. The authors propose a manifold learning-based knowledge distillation method for vision transformers to match patch-level manifold information between teacher and student models.

2. The authors decouple the relation map into three terms, reducing the computational complexity in computing the patch-level matching loss.

3. Extensive experiments demonstrate the effectiveness of the proposed method.

Weaknesses:
1. The contribution of the paper is limited. The proposed soft distillation has been proposed in [1]. Relying on the proposed patch-level distillation loss, the authors simply combine [1] to learn a compact vision transformer. Moreover, the proposed fixed depth seems to be an empirical trick to adjust the regularization strength.

2. The proposed method belongs to the relation-based knowledge distillation [2]. What is the difference between the proposed method and existing relation-based knowledge distillation methods [3][4]? It would be better for the authors to provide more discussions.

Reference:
[1] Distilling the knowledge in a neural network. arXiv 2015.
[2] Knowledge Distillation: A Survey. IJCV 2021.
[3] A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. CVPR 2017.
[4] Self-supervised knowledge distillation using singular value decomposition. ECCV 2018.

questions: 1. Some notations are confusing. For example, in Section 3.3, the authors use the same notation $F$ for the feature maps before and after reshaping/merging operations. Please make the notations more distinguishable.

2. In Table 2, for different students, the authors only compare the proposed method with either the “Hard” or “KD”. It would be better for the authors to compare the proposed method with both “Hard” and “KD”, which would strengthen the paper.

3. The authors manually select teacher-student layers as distillation pairs. However, it is unclear how the number of selected layers would affect the performance. Please provide more discussions and experimental results.

4. The authors introduce a pair of hyper-parameters $(H^{\prime}, W^{\prime})$ for patch merging. However, the effect of different values of $(H^{\prime}, W^{\prime})$ is unclear. It would be better for the authors to provide more discussions and experiments, which would strengthen the paper.

5. In Table 5, the authors state that training with only inter-image patch-level distillation loss fails because of a nan loss value. In this case, how to obtain the Top-1 accuracy? Why does the model fail to converge? It would be better for the authors to provide more explanations.

6. Minor issues

(1) In Line 3 of Page 1, “... their ...” should be “... them ...”.

(2) In Line 42 of Page 2, “... result ...” should be “... results ...”.

(3) In Line 84 of Page 3, “train ...” should be “trained ...”.

(4) In Line 123, the notations for features of the student and teacher layer should be $F_S$ and $F_T$, respectively.

(5) In Line 143, the dimension of $F_r$ should be $K \times D$.

(6) In Line 172 of Page 6, “... that and a ...” should be “... that a ...”.

(7) In Line 198 of Page 6, “... model.” should be “... models.”.

(8) In Line 208 of Page 7, “... grad ...” should be “... grid ...”.

(9) In Line 228 of Page 8, “... demonstrates ...” should be “... demonstrate ...”.

(10) In Line 232 of Page 8, “... result ...” should be “... results ...”.

(11) In Page 8, the reference position of Tables 5 and 6 seems reversed.
limitations: The authors do not provide the potential negative societal impact of their work.
review_title: null

Meta-review: metareview: Four experts in the field reviewed the paper and recommended Borderline Accept, Weak Accept, Accept, and Borderline Accept. The reviewers generally liked the approach, though some commented that it is straightforward. The reviewers' questions about experiments and clarifications were well addressed by the rebuttal. Hence, the decision is to recommend the paper for acceptance. We encourage the authors to consider the reviewers' comments and make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
confidence: Certain
Decision: Accept