Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation
Authors: Zhiwei Hao
Jianyuan Guo
Ding Jia
Kai Han
Yehui Tang
Chao Zhang
Han Hu
Yunhe Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: vision transformer
knowledge diatillation
manifold learning

Abstract: In the past few years, transformers have achieved promising performance on various computer vision tasks. Unfortunately, the immense inference overhead of most existing vision transformers withholds them from being deployed on edge devices such as cell phones and smart watches. Knowledge distillation is a widely used paradigm for compressing cumbersome architectures into compact students via transferring information. However, most of them are designed for convolutional neural networks (CNNs), which do not fully investigate the character of vision transformers. In this paper, we fully utilize the patch-level information and propose a fine-grained manifold distillation method for transformer-based networks. Specifically, we train a tiny student model to match a pre-trained teacher model in the patch-level manifold space. Then, we decouple the manifold matching loss into three terms with careful design to further reduce the computational costs for the patch relationship. Equipped with the proposed method, a DeiT-Tiny model containing 5M parameters achieves 76.5\% top-1 accuracy on ImageNet-1k, which is +2.0\% higher than previous distillation approaches. Transfer learning results on other classification benchmarks and downstream vision tasks also demonstrate the superiority of our method over the state-of-the-art algorithms.

Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation
Zhiwei Hao Jianyuan Guoi, Ding Jia?.3, Kai Han", Yehui Tang2.3,
Chao Zhang', Han Hu Yunhe Wang"*
School of information and Electronics, Beijing Institute of Technology.
Huawei Noah's Ark Lab.
KKey Laboratory of Machine Perception (MOE), School of Intelligence Science and Technology. Peking University.
(haozhw, hhu]-Obit. edu. en, Kjianyuan.gun guo, kai.han, yunhe. wang) @huawei. com, j jiadingdstu. pku. edu. cn, lyhtang, c. r.zhang)@@krr edu. cn
Abstract
In the past few years, transformers have achieved promising performance on various computer vision tasks. Unfortunately, the immense inference overhead of most existing vision transformers withholds them from being deployed on edge devices such as cell phones and smart watches. Knowledge distillation a widely used paradigm for compressing cumbersome architectures into compact students via transferring information. However, most of them are designed for convolutional neural networks (CNNs), which do not fully investigate the character vision transformers. In this paper, we fully utilize the patch-level information and propose fine-grained manifold distillation method for waasformer-bassee networks. Specifically, we train a tiny student model to match pre-trained teacher model in the patch-level manifold space. Then, decouple the manifold matching loss into three terms with careful design to further reduce the computational costs for the patch relationship. Equipped with the proposed method, DeiT-Tiny Tiny model containing 5M parameters achieves 76.5% top-1 accuracy on ImageNet-1k, which is +2.0% higher than previous distillation approaches. Transfer learning results other classification benchmarks and downstream vision tasks also demonstrate the superiority of method over the state-of-the- algorithms.
Introduction
The past decade has witnessed the rise of attention-bbssd models in the field of natural language processing (NIP) 2 Such models belonging to the transformer family can effortlessly build long-range dependencies and have achieved remarkable performance. Inspired by the success in NL.P, researchers have made great efforts introducing the transformer-baa architectures to vision domain and achieved promising results. In early attempt, Dosovitskiy et al. [4] proposed transformer- based vision model termed ViT, which takes split image patches as the input. ViT obtains comparable performance when compared to the state-of-the-art convolutional neural networks (CNNs), demonstrating the immense potential of applying transformers to vision tasks. Inspired by the design of splitting the whole image into patches as input, various vision transformers have been proposed, including Swin (5), T2T 6). Twins 77. and TNT g
Although transformers have shown the excellent capability for various vision tasks, they require large amounts parameters, resulting heavy computational burden. For example, ViT-B 44] with 86M parameters pretrained on JFT-300M can only achieve comparable performance with CNN-based EfficientNet [9] while the latter trained on ImageNet and contains only 5M parameters.
"Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) Image-level manifold space
(b) Patch-level manifold space
Figure 1: Comparison between (a) image-level manifold space and (b) patch-level manifold space. The patch-level manifold space containing fine-grained information facilitates knowledge transfer. The requirement for inordinate computing resource and storage prevents them from being deployed on memory-boundee
edge devices such as cell phones and smart watches. To facilitate the challenges above, a series of methods have been proposed to investigate compact deep neural networks, such as network pruning [10 M low-bit quantization [22 13
and knowledge distillation [4 [5].
However, smaller models usually lead to performance degradation. Knowledge distillation (KD) 14 is promising approach for inheriting information from perfoorefrreece teacher to compact student and maintaining the strong performance.
Touvron et al. [16] first proposed a KD-based compression approach for vision transformers. They trained the student transformer to match hard labels provided by pre-trained CNN teacher. Although this approach obtains satisfying results, they ignore the intermediate- information inherited in vision transformers. There have been works [17 proving the effectiveness of learning from intermediate layers for transformers in NLP, but these methods require teachers and students to have exactly the same embedding dimension at corresponding lavers, which is a fairly tight constraint and cannot always be satisfied. Manifold learning- based KD 19 20 support layers with mismatching dimensions and make use of inter-sample information concurrently. However, existing manifold distillation approaches
designed for CNNs and cannot utilize the patch-level information of vision transformers. As shown in Figure I patches depict a manifold space in a more fine-grained way. Such information can facilitate knowledge transfer remarkably.
Based on this consideration, we propose a fine-graines manifold distillation method in both patch- level and batch-level. In particular, we regard vision transformers projectors mapping inputs into multiple manifold spaces layer by layer. At each layer, we collect embeddings of patches to build their manifold relation map and train the student to match the relation map of the teacher. Since the computational complexity is high, we decouple the relation maps into three parts, which reduces the complexity by two orders of magnitude approximately. We evaluate the proposed method on the ImageNet-16 1k classification task. The proposed manifold KD outperforms the distillation method in [16] by +2.0% top-1 accuracy DeiT-Tiny. We also conduct transfer learning experiments on CIFAR- and evaluate our method on downstream tasks such as object detection and semantic segmentation. The proposed method outperforms its counterparts on both tasks. Our contributions are summarized as follows:
We propose fine- mne-grained manifold distillation method, which transfers patch-level and batch-level manifold information between vision transformers.
We use three decoupled terms to describe the manifold space and simplify the computational complexity significantly.
We conduct extensive experiments verify the effectiveness of the proposed method. The results also demonstrate the importance of soft-label distillation and fixed-depth students.
2 Related works
Vision transformer. Transformer is originally designed for NLP tasks 21 Inspired by its remarkable performance, researchers have made efforts to adopt transformer-basee models in CV tasks [2 23) Among them, Dosovitskiy et al. 4 proposed to divide an image into patches and used embeddings of the patches as model input. Based on their proposed input processing scheme for images, many variants of vision transformers have been proposed. For example, Han et al. [8] proposed TNT to model in-patch attention, Zhou el al. (24 and Touvron et
(25) built deeper vision transformers, and some researchers [26] 5 27 adopted the experience in CNN to guide the design of vision

---

transformers. However, Most well-performed vision transformers are extremely resource-consuming and should be compressed for deployment.
Knowledge distillation. KD a model compression method proposed by Hinton et al. 14 which trains lightweight student model match soft labels given by large pre-trained teacher model [28]. Moreover, there are also works using structure of a information flow in the teacher model knowledge [29 30 31,. combining self-supervised training with KD [32], or training student
with only few samples [Л]. Due to its excellent performance, KD has been adopted in various
research fields, such CV II 32 NLP T7]. and recommendation systems [33]. To compress
vision transformers, Touvron et al. [16] proposed KD-baser method termed DeiT. They added
distillation token into the student and trained the student with hard labels provided by the teacher.
Although DeiT has achieved remarkable performance, KD for vision transformers has not been well explored yet.
Manifold learning. Manifold learning is an approach for non-linear dimensionality reduction. learns smooth manifold embedded in the original feature space construct low-dimensional features (34 35). Recently, several works introduce manifold learning to KD [19 20 These methods train the student preserve the relationships among samples learned by the teacher. For vision transformers, these primary attempts are coarse and can be further improved because the basic input elements are patches not images.
Method
3.1 Preliminaries
Vision transformer. Vision transformers are sitention-bbaed models inherited from NLP, and each layer of transformer consists of multi-head self-attention (MSA) block and multi-layer perceptron (MLP) block. These models take split images as input. i.e. the patches. In particular, assuming the patch of size P x P, an input image x E H& X of size H W and channel number C' reshaped into flattened patches Xp E Nx(' for processing, where N = HW/ p2, The patches then projected into D D-dimension embedding space and added positional embeddings. We denote the summation result as X E RRN N Each model layer works as follows:
X₂ MSS((L((x.)) )) + +X::
(1) (2)
MLP((L((x.)) Xe:
where LN denotes the layer normalization operation.
Knowledge Distillation. KD widely used model compression method, where we predictions of a large pre-trained teacher model as the learning target of a tiny student model. Given a sample x corresponding label y, representing the prediction of the student and the teacher as ʃa fx(x and fx(x), respectively, the loss function of KD can be formulated as:
Led (1 (J v) + AT
where HCE is the cross-entropy function, HKL is the Kullback-Leiie divergence function, T is label smoothing
termed temperature, and À is balancing hyper-parameter.
Sometimes the intermediate features of the teacher can also be used for knowledge transfer. For example, Romero al. (36] trained the student output features similar the teacher at intermediate layers. However, such methods require the teacher and the student to have the same embedding dimension. Otherwise, additional mapping layers are required for aligning, making the distillation process non- non-transparent. Manifold learning-bast based KD methods [20] [9] support mismatched embed- ding dimensions. They train the student learn sample relationships predicted by the teacher but ignore patch-level information in vision transformers.
3.2 Fine-grained manifold distillation
To utilize the batch-level and patch-level information, we propose a fine-grained manifold distillation method. Unlike existing KD methods [16] for vision transformer only distilling with logits, our method distills batch and patch level manifolds at intermediate layers. [ an overview of the proposed method.

---

Patch Embedding
Teacher (fixed)
MD loss
Block N
Classifie
Block
Block
Matching in Manifold Space
MLP LN
MLP
Manifold Distilaaionn (MD) loss
loss
Image
Patch Emnbediing
Black M
Classiffer
Block
Block
&
LN
LN
Patches
Student
Teacher Block
Student Block m
(a) Overview
(b) MD loss illustration
Figure 2: The fine-grained manifold distillation method. (a) An overview of the method. When transferring knowledge from the teacher to the student, a manifold distillation loss adopted together with the orignal KD loss. (b) Computing of the manifold distillation loss. The loss computed by matching feature relationships between each pair of selected teacher-student layers in manifold space.
In the fine-grained manifold distillation method, regard vision transformer feature projector that embeds image patches into series of smooth manifold space layer by layer. At each pair of manually selected teacher-student layers, aim to teach the student layer output features having the same patch-level manifold structure as the teacher layer. In particular, given samples of batch size B, we denote the feature of the student layer and the teacher layer as Fss E R N Dy and Fr IR Rx Nx Dr, respectively, where Ds and Dr are embedding dimensions. We first normalize the feature at the last dimension and then compute the manifold structure, or manifold relation map. as follows:
M(N(Fs)) = (Fss)
where vi I. DDxx + R Da Da is tensor reshape operation. The manifold relation map of Fr can be obtained in a similar way. After that, we train the student to minimize the gap between (Fr) and M(}(FFs) with the following loss:
Cmf [LM()(Fs)) M(
(5)
However, the computation of manifold relation maps is resource-consumiing Its computational complexity is OB2NND) and a memory space of B 2 size required to save the result. Taking the settings B 128, N 196 and D = 192 in the DeiT-Tiny model [16] example, requires more than 240GFLOPs to compute and 2.5GB of memory space to save a single manifold relation map. The remarkable resource consumption limits the fine-grained manifold distillation method scaling up to multiple layers. Hence we must simplify the computation.
Inspired by the orthogonal decomposition of matrices, we decouple manifold relation map into three parts: an iatre-image relation map, an inter-image relation map, and a randomly sampled relation map. Figure B illlustrates the decoupling, We compute the intra-image patch-level manifold distillation loss as follows:
1 [IM(Fsli :.:) -M(MMF(........ I...
Lintra =
(6)
B
2=1
Similarly, the inter-image patch-level manifold distillation loss is computed by:
N
Linter
Σ
(7)
J=U
Moreover, relieve the information loss caused by the decoupling, we relate the intra- intra-image and the inter-image manifolds via relation map computed across randomly sampled patches. Specifically. we sample K rows in the reshaped feature (F) to obtain Fr KRD and compute the randomly sampled patch-level manifold distillation loss as follows:
Lrandom [LM(F)) MMFF)
(

---

patch
image
(a) Intra-image
(b) Inter-image
(c) Random
Figure 3: Illustration of the decoupled manifold relation map (4 images with 4 patches in each image): (a) intra-image relation map: (b) inter-image relation map; and (c) randomly sampled relation map. Manifold relation maps are computed across each group of patches filled with the same color.
Hence, the overall loss function of our proposed method is:
C = Lkdd + Efms- -ems-decouple:
(9)
L. decooppee Ochintra -Linnte + "Lrandoo:
(10)
where (t, B3 and are hyper-parameters. The summation over 7 means that the manifold relation map matching is performed multiple pairs of teacher-student layers. We will discuss the layer selecting scheme in Section 4.3
Complexity analysis. The computational complexity of the decoupled manifold relation map is and the memory space requirement is BN 2 B where each one is reduced by nearly (B/ + .) times when ignoring the lower order terms. Still taking DeiT- Tiny as an example, the floating Moating-point operations and memory space requirement reduce to 3GFLOPs and 32MB, respectively. The sampling number K is set to 192, the same as experiments. With the decoupling scheme, fine-graine fine-nrained manifold distillation across multiple layers becomes feasible.
B'
3.3 Optimization
Patch merging. Although the decoupling reduces the computation and memory space remarkably, the computing and storing overhead is still unaffordable when the patch size too small. For instance, the patch size the first stage of SwinTransforne (5) is 4, indicating that the total patch number N is 3136 under the input size of x24x 224. Such a large patch number significantly increases the computational complexity and memory space requirement of the intra-image patch-level manifold loss Lintra. To remedy this drawback, we merge adjacent patches and view them as single patch to further simplify the computation. In particular, given feature map F E DN we first reshape where the height H and the width W are obtained following the original patch splitting operation. Then we adopt a merging setting (H', W') merge every (H H") (W/W') adjacent patches in a non-overlapping manner. Zero-pane petro-paddngg is adopted when H/H H' or W/W' / is not an integer. After merging, the feature map becomes Fro E R XW' (HWDD/'' and is finally reshaped into Fm E IR. ("' (HWD) H' W') By adjusting the merging setting (H' W'), we easily strike the trade-off between the complexity and the granularity manifold relation maps.
into F" E TR. XW >
Soft distillation. Previous work [16 adds an additional distillation token into the student, and uses this token learn hard labels provided by a CNN teacher. However, when teacher and student are both vision transformers, we propose that it is better to teach the student only with soft labels of the teacher. This design is based on the assumption that a larger model can learn more knowledge than a smaller one, and models of the same family share the same knowledge pattern, i.e., student can learn most knowledge of a teacher. In our work, unless the student is larger than the teacher, we set the typer--aramedte À in Equation] 3 to
Fixed depth. Stochastic depth 377 regularization method that has becomes an infrastructure in training vision transformers [6 5]. We propose that the student should not adopt this regularization when the teacher trained with stochastic depth, since the soft labels already contain knowledge about this regularization. Otherwise, the repeated regularizations may harm the student performance. Hence, we adopt a fix-depth student in our method, i.e., the stochastic depth regularization is not used for training the student.
5

---

Table 1: Details of used teacher and student models. The input resolution is set to 224x224.
Model
Embedding
Heads
Layers
#params
FLOPs
Throughput(im/3)
RegNetY- 16GF 38
84M
15.9G
334.7
CaiT-S24 25
384
24
47M
9.4G
573.6
CaiT-XXS24 25
192
24
12M
2.5G
1012.8
DeiT-Small
16
384
12
22M
4.6G
940.4
DeiT-Tiny 16
192
12
5M
1.3G
2536.5
Swin-Small S
96
24
50M
8.7G
436.9
Swin-Tiny [
96
12
29M
4.5G
755.2
Experiments
We evaluate our fine-grained manifold distillation method on ImageNet-Ik 1k [39 classification task, CIFAR- 10/10 [40] transfer learning task, COCO [41] object detection task, and ADE20K [42
semantic segmentation task. Our implementation based on Pytorch framework MindSpore Lite tool [44].
43 and the
4.1 Setup
Datasets. We evaluate the proposed method mainly on the ImageNet- Ik image classification task.
ImageNet-1k is subset of the ImageNet dataset [39], which consists of more than 1.2M training images and 50K validation images from 1000 classes. we conduct transfer learning experiments on CIFAR-10 and CIFAR-100 datasets [40] test the generalization performance of student models. These two datasets both contain 50K training images and 10K testing images, which are categorized into 10 classes and 100 classes, respectively. Moreover, we conduct experiments
the object detection downstream task with COCO dataset 41. We use the COCO 2017 split, which consists of 118K training images and 5K validation images containing objects from 80 categories.
Baselines and models. We compare the proposed method with DeiT [16], which adds an additional distillation token in the student model to learn hard labels from the teacher. SwinTransforme students containing no distillation token, so we compare with the original KD method [4] on these models. Table I summarizes the used models in experiments.
4.2 Distillation results on ImageNet-Ik
Implementation details. On the ImageNet-1k 1k image classification task, we train DeiT students with CaiT teachers and Tranforaaffrmme students with Swin Transforrrr teachers. We slightly modify the architecture of DeiT students by removing the distillation token and only using the class token. in the KD loss is set to 1, i.e., the real label is not used to train the student. When the teacher is smaller than the student, to prevent the performance degradation caused by the weak teacher, we set À to 0.5. In the fine-grained manifold distillation loss, hyper-parameters a, 3, and are set to 4, 0.1, and 0.2, respectively. The sampling number K in loss term Grandom is set to 192. We set the above prper-ppraaeters one by one with the grid search method, indicating that their combination may not be optimal. We select the first 4 layers and the last 4 layers of the student and the teacher to conduct manifold distillation. Note that the class token in DeiT is ignored when computing manifold relation maps. Moreover, to train SwinTrannfoomrr students efficiently, we adopt a patch merging setting of (14, 14). Other training settings follow those in DeiT [16] and Swin' Transformer (5), except the stochastic depth regularization, which not used in our experiments. Each student is trained for 300 epochs with 8 Tesla-V100 GPUs
The syper-parameter
Results. Table 2 presents classification results ImageNet-1k. When compared with the hard logits distillation proposed in DeiT [16), our manifold method achieves remarkable performance improvements. For example, Deit-Tine Tiny distilled via manifold method outperforms Deit by +2.0% top-1 accuracy, and Deit-Small outperforms Deit by + +0.9% with the CaiT-S24 teacher. Although the "soft logits distillation" can boost the result by 0.4% compared to the "hard logits distillation" our proposed manifold still obtains better performance by +0.6%. When the teacher is CaiT-XXS24, a much weaker architecture, the corresponding improvements are +1.6% DeiT-Tiny and +1.2% on DeiT-Smmal, respectively. Note that we report the RegNetY-16GG teacher results DeiT for comparison, but do not conduct fine-grained manifold distillation experiments with this CNN model because the proposed method is designed for distillation between vision transformers.
*Mindspore: http:///minddoooeeccoc/coooo//// Pytorch code: distillation and httpps///gttubb.cc////awwweeweee

---

Table 2: Distillation results on ImageNet- Ik with 224 input. the first column. "Hard" indicates the hard-label distillation strategy, "Soft" indicates the soft-label based KD method.
Distillation method
Teacher
Top-1(%)
Student
Top-I($8
DeiT-Tiny
72.2
Hard 16
RegNetY- 16GG
82,9
Dei'--Tiny
74.5
Hard [6
CaiT-XXX24
78.5
Deiy-Tiny
73.9
Hard [6
CaiT--24
83.4
DeiT-Tiny
74.5
Soft 14
CaiT--22
83.4
DeiT-Tiny
74.9
Manifold (ours)
CaiT-XXS24
78.5
DeiT-Tiny
75.5
Manifold (ours)
CaiT-S24
83.4
DeiT-Tiny
76.5
DeiT-Small
79.9
Hard 16
RegNerY-16GF
82.9
DeiT-Small
81.2
Hard 16
CaiT-XXS24
78.5
DeiT-Small
80.1
Hard 16
CaiT-S24
83.4
DeiT-Small
81.3
Manifold (ours)
CaiT-XXS24
78.5
DeiT-Small
81.3
Manifold (ours)
CaiT-S24
83.4
DeiT-Small
82.2
Swin-Tiny
81.2
Soft 14
Swin-Small
83.2
Swin-Tiny 81.7
Manifold (ours)
Swin-Small
83.2
Swin-Tiny 82.2
Table 3: Ablation study of main components. The "V" mark indicates whether we adopt the corresponding training strategy. The "Hard" logits label the hard-label distillation scheme in DeiT. We adopt a CaiT-S24 teacher and a DeiT--Tny student.
Manifold Distillation
Logits label
Fixed Depth
Top-1(%)
Hard
74.5
Soft
x
Hard
V 75.5
V
Soft
75.8
Hard
V
75.9
Soft.
76.5
Moreover, we conduct experiments based on Swin Transformer, one of the state-of-the-arr vision transformer architecture. When the teacher is Swin-Small and the student is Swin-Tiny. our proposed method surpasses the original student by +1.0% and achieves +0.5% performance improvement compared with the original soft logits label based KD method, demonstrating the effectiveness of our fine-grained manifold distillation.
4.3 Ablation and parameter comparison
Ablation of main components. We design experiments to verify the effectiveness of our proposed fine-graines manifold distillation method, the fixed student depth, together with the "hard" and "soft" logits label in previous distillation methods. As shown in Table soft logits label can bring +0.5% top-1 accuracy compared hard logits label, and fixed depth can improve the baseline by +0.8-1.0%. which is a practical strategy to distill transformer. Moreover, when combined with the -raiinai manifold distillation, student performance can be further improved by +0.7%.
Layers for fine-grained manifold distillation. To study the impact of different layer selections in manifold relation map matching, we evaluate 5 layer selecting schemes and compare their per- formance. In particular, we take CaiT-S24 as the teacher model and DeiT-Tiny the student model. The number of selected layers set to 6.
Table 4: Ablation study on different selected layer numbers. We denote the indices of layers lected from a I--layer model as {1, ..., k; L- 1, ..... L- 1, L), and conduct experiments with the Csill teacher (L = 24) and the DeiT-Tiny (L 12) student to compare different & setting.
Num. of selected layers
Top-1(9)
From the results reported in Table 5] conducting fine-grained manifold distillation at the head and the tail of student models are both crucial. Hence, we speculate that the "Shallow/Deep" layer se- lecting scheme is the best because of its outstand-
(A==)
76.3
(k=2)
76.4
(k=4)
76.5
12 (k=6)
76.5

---

Table Distillation results with different layer selecting schemes. We use a CaîT-S24 teacher and DeiT-Tiny student to perform fine-grained manifold distillation. A total of 6 layers are selected from the "Shallow", "Medium", or "Deep" part of model. "Uniform" refers to selecting layers across the whole model uniformly. Note that the number of select layers is different from other experiments.
Scheme
Teacher layers
Student layers
Top-1(%)
Shallow
{1. 2, 6,
{1,2,3, 4, 5,
75,8
Deep
{19. 20, 21, 22, 23, 24,
7,,,9,10,11,12
75.5
Shallow/Deep
{1, 2,, 23,
(1,2,3 10,
76.3
Shallow/Mediun/DDep
[1. 12, 13, 13,,,,2.. 24)
(1,2,6.7,11,22 12]
76.2
Uniform
{4,8, 12, 16, 20, 24)
(2,4,6,8,10, 12]
76.2
Table 6: Ablation study results of decoupled manifold loss terms. The teacher is CaiT--S24 and the student is DeiT-Tiny. indicates that we use the full precision training instead of mixed precision training for the result.
Table 7:
Comparison of different hyper- parameter settings. The teacher is CaiT-S24 and the student is DeiT-Tiny.
Hyper parameter
Top-1(%)
3
K
Loss term
Top-1(%)
2.0
0.1
0.2
192
76.1
Lintra
Linter
Crandom
8.0
0.1
0.2
192
76.1
X
75.8
4.0
0.05
0.2
192
76.2
X
X
76.0
4.0
0.2
0.2
192
76.2
X
V
75.8
X
X
76.2
4.0
0.1
0.1
192
76.2
X
V
V
76.1
4.0
0.1
0.4
192
76.2
X
V
76.4
4.0
0.1
0.2
96
76.0
V
76.0
4.0
0.1
0.2
384
76.3
76.5
4.0
0.1
0.2
192
76.5
ing performance and ease of use. Following this selecting strategy, further ablate the influence of the number of selected layers. We denote the indices of layers selected from a L-layer model as {1, 212,... k, L and the DeiT-Tiny (I. = 12) student to compare different & setting. The corresponding are shown in Table 4 We find that our method is robust with various layer numbers, selecting 4 layers (k = 2) obtains 76.4 top-1 accuracy, and selecting 8 (k 4) and 12 (k 6) layers achieve 76.5 top-1 accuracy, respectively.
1, ...... -- T.). and conduct experiments with the CaiT-small teacher (I. 24)
Ablation of decoupled manifold loss terms. The decoupled fine- ame-ggrined manifold distillation loss consists of three terms. We study their effectiveness and report the results in Table 6 The results show that each component in the decoupled fine-graing rreegrained manifold distillation loss contributes to improving the final performance. In particular, only transferring inter- relation maps leads to an unstable training process (mixed precision training will corrupt the student model, leading to a NAN loss), speculate that these relations are highly related to the randomly sampled image batches. When equipped with intra-image and random-image maps, the inter-image relation maps can further improve the accuracy by +0.1%.
Parameters in decoupled manifold distillation loss. There are 4 typer-parameters in decoupled manifold distillation loss: the weight of inter-image loss the weight of intra-image loss 3, the weight of randomly sampled loss 7 and the sampling number K. We compare different settings of these parameters and report the results in Table 7 Our default setting outperforms those either increasing or reducing one of the parameters. However, due to our coarse parameter searching scheme, believe that the performance can be further improved by setting these hyper-parameters, more carefully.
4.4 Transfer learning
To measure the generalization ability of the proposed method, we conduct transfer learning ex- periments. We fine-tune DeiT-Tiny student trained with a CaiT--24 teacher on CIFAR-10 and CIFAR-100 datasets for 1000 epochs. We adopt a batch size of 768 and an Adam W optimizer with a learning rate of 5 10-6 66 Other settings follow DeiT [16].
8

---

Table 8: Transfer learning results on CIFAR-10/100. Each student is fine-tuned for 1000 epochs.
Dataset
Teacher
Student
Distillation
Top-1(%)
DeiT Tiny
98.19
CIFAR-I0
CaiT-S24
DeiT-Tiny
Hard Manifold
98.23
CaiT--24
DeiT-Tiny
98.48
DeiT- Tiny
86.61
CIFAR-
CaiT-S24
DeiT-Tiny DeiT: Tiny
Hard Manifold
87.34
CaiT-S24
88.05
Table 9: Object detection results on COCO 2017. Backbones pre-trained on ImageNet-Ik, The teacher is trained for 36 epochs and each student is trained for 12 epochs.
Model
#params
FL.OPs
Ap
APhox
APler
(Teacher) Swin-S Mask R-CNN
69M
365G
48.5
70.2
53.5
(Student) Swin-T T Mask R-CNN (Manifold distilled) Swin-T + Mask R-CNN
48M
272G
43.7
66.6
47.7
48M
272G
44.7
67.1
48.6
Table
reports the transfer learning results. Students trained with the fine-grained manifold dis- tillation method generalize better than others (+0.25% on CIFAR-10 and +0.71 % on CIFAR-100), demonstrating favorable generalization ability of the proposed method.
4.5
Downstream task
To further evaluate the effectiveness of our proposed method, we adopt the fine- grained manifold distillation to train object detection models on COCO 2017 dataset and semantic segmentation models on ADE20K dataset.
We adopt Mask R-CNN 45 as the detection framework. The teacher backbone Swin-Small and the student backbone Swin-Tiny. All used backbones are pretrined ImageNet-1k, kk. The teacher is trained for 36 epochs and each student is trained for 12 epochs. When training students with fine-grained manifold distillation, we adopt the distillation loss on outputs of the last two backbone stages and outputs of the feature pyramid network neck [46).
Implementation details COCO.
Results on COCO. presenss the detection results. Our manifold distilled student outperforms the student trained without distillation (+1.0% box AP), demonstrating that the proposed method benefits the object detection downstream task.
Implementation details on ADE20K. We adopt UPerNet [47 as the segmenta- tion framework. The student is trained for 160K iterations following (5). Other settings follow experiments COCO.
Table 10: Semantic segmentation results on ADE20K. The student is trained for 160K iterations.
Model
#params
FLOPss mloU
(Teacher) Swin-S + UPerNet (Student) Swin-T+ UPerNet (FitNet distilled) Swin-T UPerNet
81M
1038G
47.64
60M
945G
44.51
Results on ADE20K. Table 10 presents the segmentation results. The proposed method outperforms the ditect training
60M
945G
44.85
(Manifold distilled) Swin-T UPerNet
60M
945G
45.66
method and FitNet [48]. an intermediate feature distilling approach, indicating that the manifold distillation approach can also help train semantic segmentation model.
Conclusion
This paper proposes a fine-grained manifold distillation method for vision transformers. We match patch and batch level features of student and teacher in manifold space, and decouple the matching loss into three terms to reduce the computational complexity. Moreover, we adopt a patch merging scheme to further simplify the computation. Different from previous works, the student is distilled with soft labels and fixed depth. Experiments are conducted on ImageNet- Ik. CIFAR- 10/100, MS COCO, and ADE20K. Corresponding results demonstrate that manifold KD consistently outperforms existing methods. The large search space of apper-parameters is one of the most serious drawbacks of the proposed method. In the future, we will study to further simplify the proposed method and help it be more parameter insensitive.
Acknowledgment. This work is supported by National Key Research and Development Program of China under No. 2021 YFC3300200. Also, this work is partially supported by the National Nature Science Foundation of China under Grant 62071013 and 6167 1027, and National Key R&D Program

---

of China under Grant 2018AAA0100300, We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend Al Processor used for this research.
References
[1] Jacob Devlin, Ming-We Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: pre-training deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019.
[2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
[3] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, et al. Attention is all you need, In Advances in Neural Information Processing Systems, 2017.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner, et al. An image worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations. 2021.
[5] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, et Swin transformer: Hierarchical vision transformer using shifted windows. In International Conference on Computer Vision, 2021.
[6] Li Yuan, Yumpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, al. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In International Conference on Computer Vision, 2021.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, et al. Twins: Revisiting the design of spatial attention vision transformers. Advances in Neural Information Processing Systems, 2021.
[8] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. In Advances in Neural Information Processing Systems, 2021.
[9] Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, 2019.
[10] Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, et al. Patch slimming for efficient vision transformers. arXiv preprint arXiv:2106.02855 2021.
[11] Yixing Xu, Yunhe Wang, Hanting Chen, Kai Han, Chunjing Xu, Dacheng Tao, et al. Positiie-umlabbledd compression the cloud. In Advances Neural Information Processing Systems, 2019.
[12] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, et al. Quantization and training of neural networks for efficient aninger-rithmeeic-only inference. In Computer Vision and Pattern Recognition, 2018.
[13] Yixing Xu, Kai Han, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Learning frequency domain approximation for binary neural networks. Advances in Neural Information Processing Systems, 2021.
[14] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. arXiv preprint Xrvi 0ee.: 503.02531, 2015.
[15] Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Distilling object detectors via decoupled features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[16] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, 2021.
[17] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compres- sion. In Conference on Empirical Methods in Natural Language Processing, 2019.
[18] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, et al. Tinyhert: Distilling BERT for natural language understanding. In Conference Empirical Methods in Natural Language Processing, 2020.
[19] Hanting Chen, Yunhe Wang, Chang Xu, Chao Xu, and Dacheng Tao. Learning student networks via feature embedding, IEEE Trans. Neural Networks Learn. Syst., 32(1):25 35, 2021.
10

---

[20] Baoyun Peng, Xiao Jin, Dongsheng Li, Shunfeng Zhou, Yichao Wu, Jiaheng Liu, et al. Correlation congruence for knowledge distillation. In International Conference on Computer Vision, 2019.
[21] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, et al. Attention you need. In Advances in Neural Information Processing Systems, 2017.
[22] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, 2020.
[23] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, al. Generative pretraining from pixels. International Conference on Machine Learning, 2020.
[24] Daquan Zhou, Bingyi Kang, Xiaojic Jin, Linjie Yang, Xiaochen Lian, Qihin Hou, et al. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2 103.. /1886 2021.
[25] Hugo Touvron, Matthicu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In International Conference Computer Vision, 2021.
[26] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, al. Pyramid vision transformer: A versatile backbone dense prediction without convolutions. International Conference on Computer Vision, 2021.
[27] Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2022.
[28] Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: survey. International Journal of Computer Vision, 18966))179-1188 2021.
[29] Junho Yim. Donggyu Joo, Ji-Hoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Computer Vision and Pattern Recognition, 2017.
[30] Seung Hyun Lee, Dae Ha Kim, and Byung Cheol Song. Self-supervised knowledge distillation using singular value decomposition. In Earopean Conference on Computer Vision, 2018.
[31] Zhiwei Hao, Yong Luo, Zhi Wang, Han Hu, and Jiamping An. Cdfkd-mfs: Collahorative data-free knowledge distillation via multi-level feature sharing. arXiv preprint arXiv:2205.1 1845, 2022.
[32] Jianyuan Guo, Kai Han, Han Wu, Chao Zhang, Xinghao Chen, Chunjing Xu, Chang Xu, and Yunhe Wang. Positive- unlabeled data purification in the wild for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[33] Guorui Zhou, Ying Fan, Rumpeng Cui, Weijie Bian, Xiaoqiang Zhu, and Kun Gai. Rocket launching: universal and efficient framework for training well-performing light net. In AAAI Conference on Artificial Intelligence, 2018
[34] Sam T Roweis and Lawrence K Saul. Nonlincar dimensionality reduction by locally lincar embedding. science, 29005500):2333-2236 2000.
[35] Joshua Tenenbaum. Vin de Silva, and John Langford. A global geometric framework for nonlinear dimensionality reduction. science, 290(5000)23199 2323,, 2000.
[36] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In International Conference on Learning Representations, 2015.
[37] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In European Conference Computer Vision, 2016.
[38] Ilija Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Computer Vision and Pattern Recognition, 2020.
[39] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009.
[40] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
11

---

[41] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays. Pietro Perona, Deva Ramanan, et al. Microsoft COCO: common objects in context. In European Conference Computer Vision, Lecture Notes Computer Science, 2014.
[42] Bolei Zhou, Hang 7hao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralha Scene parsing through ADE20K dataset. In Computer Vision and Puttern Recognition, 2017.
[43] Adam Paszke, Sam Gross, Francisco Massa, Adam James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances neural information processing systems, 2019.
[44] Mindspore. http://www.mmindspor.ccu 2020.
[45] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. International Conference on Computer Vision, 2017.
[46] Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Computer Vision and Puttern Recognition. 2017.
[47] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unitied perceptual parsing for scene understanding. In European Conference Computer Vision 2018
[48] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In International Conference Learning Representations, 2015.
Checklist
1. For all authors...
(a) Do the main claims made the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL.)? [No]
(b) Did you specify all the training details (e.g., g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experiments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs,
internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're us-
ing/curating? [No]
(c) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [No]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you descrihe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "yhKmzpoc1pY", "reviewer": "Reviewer_ipbi", "summary": "The authors apply knowledge distillation on vision transformer to learn a more efficient one, so that it could be  deployed on edge devices like cell phones and smart watches. \n\nConsidering the knowledge distillation has been fully investigated in neural networks,  the authors utilize the patch-level information and propose a fine-grained manifold distillation method. \n\nTo achieve it, the authors use a kind of manifold learning-based KD method, which supports mismatched embedding dimensions. And to utilize the patch-level information in VIT , the authors propose a fine-grained manifold distillation method. The aim is to teach the student layer to output features having the same patch-level manifold structure as the teacher layer.\n\nAnd in order to simplify the computation, the authors decouple a manifold relation map into three parts, which are intra-image relation map,  inter-image relation map and  randomly sampled relation map.\n\nTogether with the path merging strategy, the computation complexity can be further reduced.", "questions": "see weakness", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "First of all, the motivation of this paper is clear. The authors want to do knowledge distillation to learn a smaller and more efficient vision transformer. And as it is applied in the vision transformer architecture, the authors consider to preserve the relationships among patches.\nSo, the afterwards proposed schemes are just reasonable. There are mainly two strategies, both of which are proposed to reduce the computation complexity. One is to use three kinds of maps to replace the manifold relation map. The other is to merge patches.\n\nSo the weakness of this paper comes, that it doesn't surprise me a lot. It is just looked like a normal solution. In distillation, the manifold map is like a kind of structure information in the pixel-level tasks.\n\nTo be more specific, I wonder:\n1. If we only use the patch merging strategy, it could reduce the computation complexity a lot. So what is the necessarily to uncouple the map?\n2. And when uncoupling the map, what is the exact meaning of the random sampled relation map. I think that the first two kinds may work.\n3. And in equation 10, there introduces three weights for balancing each item. When it is applied in each block, I think choosing the weights for each block is a complex task.\n4. In line 120-121,  the authors say the teacher-student layers are manually selected, is it reliable enough?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZIWxACZ1yGs", "reviewer": "Reviewer_eVwU", "summary": "This paper presents a fine-grained patch-level distillation method for vision transformers. Specifically, the proposed method matches the student and teacher models in the patch-level manifold space. The proposed method further decouples the relation map into three terms to solve the high computational complexity brought by the patch-level distillation. Experimental results on various vision tasks and benchmarks demonstrate the effectiveness of the proposed method.", "questions": "1. Some notations are confusing. For example, in Section 3.3, the authors use the same notation $F$ for the feature maps before and after reshaping/merging operations. Please make the notations more distinguishable.\n\n2. In Table 2, for different students, the authors only compare the proposed method with either the “Hard” or “KD”. It would be better for the authors to compare the proposed method with both “Hard” and “KD”, which would strengthen the paper.\n\n3. The authors manually select teacher-student layers as distillation pairs. However, it is unclear how the number of selected layers would affect the performance. Please provide more discussions and experimental results.\n\n4. The authors introduce a pair of hyper-parameters $(H^{\\prime}, W^{\\prime})$ for patch merging. However, the effect of different values of $(H^{\\prime}, W^{\\prime})$ is unclear. It would be better for the authors to provide more discussions and experiments, which would strengthen the paper.\n\n5. In Table 5, the authors state that training with only inter-image patch-level distillation loss fails because of a nan loss value. In this case, how to obtain the Top-1 accuracy? Why does the model fail to converge? It would be better for the authors to provide more explanations.\n\n6. Minor issues\n\n(1) In Line 3 of Page 1, “... their ...” should be “... them ...”.\n\n(2) In Line 42 of Page 2, “... result ...” should be “... results ...”.\n\n(3) In Line 84 of Page 3, “train ...” should be “trained ...”.\n\n(4) In Line 123, the notations for features of the student and teacher layer should be $F_S$ and $F_T$, respectively.\n\n(5) In Line 143, the dimension of $F_r$ should be $K \\times D$.\n\n(6) In Line 172 of Page 6, “... that and a ...” should be “... that a ...”.\n\n(7) In Line 198 of Page 6, “... model.” should be “... models.”.\n\n(8) In Line 208 of Page 7, “... grad ...” should be “... grid ...”.\n\n(9) In Line 228 of Page 8, “... demonstrates ...” should be “... demonstrate ...”.\n\n(10) In Line 232 of Page 8, “... result ...” should be “... results ...”.\n\n(11) In Page 8, the reference position of Tables 5 and 6 seems reversed.", "limitations": "The authors do not provide the potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors propose a manifold learning-based knowledge distillation method for vision transformers to match patch-level manifold information between teacher and student models.\n\n2. The authors decouple the relation map into three terms, reducing the computational complexity in computing the patch-level matching loss.\n\n3. Extensive experiments demonstrate the effectiveness of the proposed method.\n\nWeaknesses:\n1. The contribution of the paper is limited. The proposed soft distillation has been proposed in [1]. Relying on the proposed patch-level distillation loss, the authors simply combine [1] to learn a compact vision transformer. Moreover, the proposed fixed depth seems to be an empirical trick to adjust the regularization strength.\n\n2. The proposed method belongs to the relation-based knowledge distillation [2]. What is the difference between the proposed method and existing relation-based knowledge distillation methods [3][4]? It would be better for the authors to provide more discussions.\n\nReference:\n[1] Distilling the knowledge in a neural network. arXiv 2015.\n[2] Knowledge Distillation: A Survey. IJCV 2021.\n[3] A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. CVPR 2017.\n[4] Self-supervised knowledge distillation using singular value decomposition. ECCV 2018.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MfL3CtHRz5t", "reviewer": "Reviewer_JpXV", "summary": "The authors found the existing vanilla distillation in ViT can not work with different embedding dimensions between the teacher and student model. Moreover, the authors argue previous ViT distillation process is conducted at a coarse level.\nTo resolve the above issues, they propose a fine-grained patch-level manifold distillation method. \nBy decomposing the distillation process into inter and intra parts, the authors found the results have better accuracy with negligible computational overhead. ", "questions": "1. Why does the Soft Distillation (3rd row) in Table 3 improve only 0.3% compared with the 2nd row?\n2. The proposed manifold-based KD still belongs to a subset of KD, but why the authors separate the proposed MD from the original KD, in the caption of Table 2?\n3. I appreciate the authors admit the failure of distillation with L_inter, but why do not you re-run the experiment and report a new number? \n4. Why missing semantic segmentation experiments as an extension of the downstream tasks for the proposed new KD?", "limitations": "The authors should add discussions about societal impact. \nFor other limitations, please refer the question part.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. originality:\nManifold learning-based KD, which does not require matching dimensions among teacher and student networks, was first implemented in ViTs.\nTo reduce the computational complexity, the relation maps are decomposed into three parts for efficiency.\n2. quality:\nThe paper is well written, with detailed experiments and ablation studies with other methods and the proposed variants.\n3. clarity:\nThe paper consists of an illustration of the decoupled manifold relation map, text explanations of the proposed methods.\n4. significance:\nThe paper solves the problem that distillation of different hidden dimensions in ViTs, reduces computational cost than its baseline, and improves the distilled accuracy of previous ViT+distillation methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "1A_wLFZ8axy", "reviewer": "Reviewer_jrGf", "summary": "This paper works on improving manifold learning-based knowledge distillation when using lightweight vision transformer models. The authors proposed a patch-level fine-grained manifold distillation method by matching the intermediate features between student and teacher in a manifold space and further decoupled the matching loss to reduce the computational complexity.\n\nThe proposed approach is evaluated on ImageNet-1K classification task and COCO2017 object detection task and shows better performance than other baselines. Ablation studies are done on the four design components.", "questions": "Is there a trade-off between the scheme for reducing computational complexity and the performance on downstream tasks? More explanation and reasoning should be done here.", "limitations": "More quantitative evaluation and examples should be provided in the Complexity analysis section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The evaluation and ablation study is relatively comprehensive, covering different downstream tasks and proposed design components.\n\nWeakness: The efficientness of the proposed method is not well supported due to the lack of discussion/details/evaluation on common public benchmarks. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept