Improved Feature Distillation via Projector Ensemble
Authors: Yudong Chen
Sen Wang
Jiajun Liu
Xuwei Xu
Frank de Hoog
Zi Huang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Knowledge distillation
feature distillation
ensemble learning

Abstract: In knowledge distillation, previous feature distillation methods mainly focus on the design of loss functions and the selection of the distilled layers, while the effect of the feature projector between the student and the teacher remains under-explored. In this paper, we first discuss a plausible mechanism of the projector with empirical evidence and then propose a new feature distillation method based on a projector ensemble for further performance improvement. We observe that the student network benefits from a projector even if the feature dimensions of the student and the teacher are the same. Training a student backbone without a projector can be considered as a multi-task learning process, namely achieving discriminative feature extraction for classification and feature matching between the student and the teacher for distillation at the same time. We hypothesize and empirically verify that without a projector, the student network tends to overfit the teacher's feature distributions despite having different architecture and weights initialization. This leads to degradation on the quality of the student's deep features that are eventually used in classification. Adding a projector, on the other hand, disentangles the two learning tasks and helps the student network to focus better on the main feature extraction task while still being able to utilize teacher features as a guidance through the projector. Motivated by the positive effect of the projector in feature distillation, we propose an ensemble of projectors to further improve the quality of student features. Experimental results on different datasets with a series of teacher-student pairs illustrate the effectiveness of the proposed method. Code is available at https://github.com/chenyd7/PEFD.

Improved Feature Distillation via Projector Ensemble
Yudong Chen'
Sen Wang Jiajun Liu Xuwei Xu'
Frank de Hoog² Zi Huang'
'The University of Queensland CCSRO Data6l [yudong chen,sen.
sen. wang, xuwei xxx edu { jjiajun. 1iu frank. dehooo) huang@itee uq uq.uuu.aaa edu.au
Abstract
In knowledge distillation, previous feature distillation methods mainly focus on the design of loss functions and the selection of the distilled layers, while the effect of the feature projector between the student and the teacher remains under- explored. In this paper, we first discuss a plausible mechanism of the projector with empirical evidence and then propose a new feature distillation method based a projector ensemble for further performance improvement. We observe that the student network benefits from projector even if the feature dimensions of the student and the teacher are the same. Training a student backbone without a projector can be considered as a multi-task learning process, namely achieving discriminative feature extraction for classification and feature matching between the student and the teacher for distillation empirically verify that without projector, the student network tends to overfit the teacher's s feature distributions despite having different architecture and weights initialization. This leads degradation the quality of the student's deep features that are eventually used in classification. Adding a projector, on the other hand, disentangles the learning tasks and helps the student network to focus better the main feature extraction task while still being able to utilize teacher features as guidance through the projector. Motivated by the positive effect of the projector in feature distillation, we propose an ensemble of projectors to further improve the quality of student features. Experimental results different datasets with a series teacher-student pairs illustrate the effectiveness of the proposed method. Code is available /tttps://gi t/gitt/ub. com www.cc/emcccceeeeeee chenyd7/PFFD
same time. We hypothesize and
Introduction
The last decade has witnessed the rapid development of Convolutional Neural Networks (CNNs)
19 29 9 20) The resulting increases in performance however, have come with substantial increases in network size and this largely limits the applications of CNNs on edge devices [13]. To alleviate this problem, knowledge distillation has been proposed for network compression. The key idea of distillation is to the knowledge obtained by the large network (teacher) guide the optimization of the lightweight network (student) 12 31].
Existing methods be roughly categorized into logit-based, Seature-based and similarity-bige based distillation 77). Recent research shows that feature-based methods generally distill a better student network compared the other two groups (30 4. We conjecture that the process of mimicking the teacher's features provides a clearer optimization direction for the training of the student network. Despite the promising performance of feature distillation, it is still challenging narrow the gap between the student and teacher's feature spaces. To improve the feature learning ability of the
"Corresponding Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Lyp
LD
g(s)
Ln
f((s)
LeE
L
P1
L LE
P
P2
Ps
Classifier
Classifier
Classifier
S
Teacher
Student
Teacher
Student
Teacher
Student
(a) w/o Projector
(b) w/ Single Projector
(c) w/ Multiple Projectors
Figure I: Illustration of (a) feature distillation without a projector when the feature dimensions of the student and the teacher are the same, (b) the general feature-based distillation with a single projector [ (3 and (c) the proposed method with multiple projectors, where Lce and LF are the cross-entropy loss and the feature distillation loss, respectively.
student, various feature distillation methods have been developed by designing more powerful 33 4 and determining more effective links between the layers of the
objective functions [30] 336
student and the teacher 1
We have found that the feature projection process from the student to the teacher's feature space plays key part in feature distillation and be redesigned to improve the performance. Since the feature dimensions of student networks are not always the same with that of teacher networks, projector is often required to map features into common space for matching. As shown in Table imposing a projector on the student network can improve the distillation performance even if the feature dimensions the student and the teacher are the same. We hypothesize that adding projector for distillation helps to mitigate the overfitting problem when minimizing the feature discrepancy between the student and the teacher. As shown Figure [(a) distillation without projector can be regarded as multi-task learning process, including feature learning for classification and feature matching for distillation. In this case, the student network may overfit the teacher's feature distributions and the generated features are less distinguishable for classification. Our empirical results in Section support this hypothesis some extent. Besides, inspired by the effectiveness of adding a projector for feature distillation, propose ensemble of projectors for further improvement. Our intuition that projectors with different initialization generate diverse transformed features. Therefore, helpful to improve the generalization of the student by using multiple projectors according to the theory behind ensemble learning [37 32). Figure shows the comparisons of existing distillation methods and our method.
Our contributions are three-fold:
We investigate the phenomenon that the student benefits from projector during feature distillation even when the student and the teacher have identical feature dimensionalities.
Technically, propose ensemble of feature projectors to improve the performance of feature distillation. The proposed method is extremely simple and easy to implement.
Experimentally, we conduct comprehensive comparisons between different methods on benchmark datasets with wide variety of teacher-student pairs. It is shown that the proposed method consistently outperforms vate-of-the-as feature distillation methods.
Related Work
Since this paper mainly focuses the design of the projector, we divided the existing methods into two categories in term of the usage the projector as follows:
Projector-free methods. As the most representative distillation method, Knowledge Distillation (KD) 11 proposes utilize the logits generated by the pre-trained teacher to be the additional targets of the student. The intuition of KD is that the generated logits able to provide more

---

useful information than the general binary labels for optimization. Motivated by the success of KD, various logit- methods have been proposed for further improvement. For example, Deep Mutual Learning (DML.
35 proposes to replace the pre-trained teacher with an ensemble of students so that the distillation mechanism does not need to train a large network in advance. Teacher Assistant Knowledge Distillation (TAKD) 21 observes that a better teacher may distill worse student due to the large performance gap between them. Therefore, a teacher assistant network is introduced to alleviate this problem. Another technical route of projector-free methods is the similarity-base distillation. Unlike the logit-based methods that aim to exploit the category information hidden the predictions of the teacher, similarity- based methods try explore the latent relationships between samples in feature space. For example, Memilarity-Preserviie similarity matrices of the student and the teacher by computing the inner products between features and then minimises the discrepancy between the obtained similarity matrices. Similarly, Correlation Congruence (CC) (23 forms the similarity matrices with kernel function. Although the logit-based and similarity-bane based methods do not require an extra projector during training, they are generally less effective than the reature-based methods shown in the recent research 4 33
(SP) [1] distillation first constructs the
Projector-depende methods. Feature distillation methods aim to make student and teacher fea- tures as similar as possible. Therefore, projector is essential to map features into common space. The first feature distillation method FitNets [24] minimizes the L2 distance between student and teacher feature maps produced by the intermediate layer of networks. Furthermore, Contrastive Representation Distillation (CRD) 30 Softmax Regression Representation Learning (SRRL) [3 and Comprchensive Interventional Distillation (CID) 4 show that the last feature representations of networks are more suitable for distillation. One potential reason is that the last feature representations closer to the classifier and will directly affect the classification performance B3) The aforemen- tioned feature distillation methods mainly focus on the design of loss functions such as introducing contrastive learning (30) and imposing causal intervention [4] A simple 1xl convolutional kernel or a linear projection is adopted to transform features in these methods. We note that the effect of projectors largely ignored. Previous works such as Factor Transfer (FT) [66] and Overhaul of Feature Distillation (OFD) [I]] try to improve the architecture of projectors by introducing the auto- encoder and modifying the activation function. However, their performance is not competitive when compared to the state-of-the-- art methods [33] 4. Instead, this paper proposes simple distillation framework by combining the ideas of distilling the last features and projector ensemble.
3 Improved Feature Distillation
We first define the notations used in the following sections. In line with observations in re- cent research 30 we apply the feature distillation loss to the layer before the classifier. S = {$1, $2, ..... St Sb. E Rddx denotes the last student features, where d and b are the fea- ture dimension and the batch size, respectively. The corresponding teacher features are represented by T' = {t1, f,,... ..., f.,,,, } E Rmxbb, where m is the feature dimension. To match the dimensions of Š and T, a projector g(-) is required to transform the student or teacher features. We experimentally find that imposing the projector on the teacher is less effective since the original and more informative feature distribution from the teacher would be disrupted. Therefore, in the proposed distillation framework, projector will be added to the student as g(st) o(Ws) s;) during training and be removed after training, where (-)) is the ReLU function and W E Rm xd is a weighting matrix.
$b}
3.1 Feature Distillation as Multi-task Learning
In recent work, SRRL and CID combine the feature-based loss with the logit-based loss to improve the performance. Since distillation methods are sensitive to hyper-parameters and changes of teacher-student combinations, the additional objectives will increase the training cost for coefficients adjustment. To alleviate this problem, we simply use the following Direction Alignment (DA) loss g for feature distillation:
LDA
Σ
g(s:)
t, 1
1
1 b
(g(s:).tt)
(1)
26 Ig((:))ll
||t.11
where ||-112 indicates the 1.2-norm and ( represents the inner product of two vectors. By convention 12 30 33), the distillation loss is coupled with the cross-entropy loss to train student. As mentioned
3

---

w/o projector
w/ projector
wio projector w projector
0.5
Detwen-Cllus Class Cosine Similarity
0.4
Direction
0.1
Sced1
Seed2
Seed3
Sced1
Seed2
Seed3
Figure 2: The left figure displays the direction alignment loss between teacher and student features with and without projector. The right figure displays the average between-class cosine similarities in students' feature spaces. These results are obtained on CIFAR-100 with a teacher-student pair ResNee33x4-ResNettee The feature dimensions of ResNet32x4 and ResNet8x4 are the same, which means m d.
in the Introduction, adding a projector helps to improve the distillation performance even the feature dimensions of the student and the teacher are the same. We hypothesize that the training process of the student network without projector can be viewed as a multi-task learning (i.c. distillation and classification tasks) in the same feature space. As such, student features tend to overfit teacher features and become less discriminative for classification. We use two measurements to empirically verify this hypothesis. The first one is measure the discrepancy between student and teacher features as follows:
(8,,
MDA = - ==1 1s11211411
Σ
We display the MDA results of students with and without projector in Figure Apparently, since student features will directly interact with teacher features, the MDA results of the student without a projector significantly lower than that of the student with projector in different seeds. However, we found that the distilled student features without a projector less discriminative by investigating the between-class cosine similarity in the student's feature space. The between-class cosine similarity is computed as follows:
(8
MBc b
(3)
c ==
1=
where 8j is the j-th sample belonging class different from that of and e; is the total number of 8y corresponding to 8;- The results of MBC are in Figure is shown that the student network with a projector generate more discriminative features compared the student network without a projector. The results of MDA and MBC suggest that the student tends to overfit the teacher's feature space without a projector. Since the tasks classification and distillation are performed in the same feature space, the generated features are less distinguishable for classification. By adding projector to disentangle the two tasks, we can improve the classification performance of the student.
3.2 Improved Feature Distillation with Projector Ensemble
The above analysis suggests that the projector can boost the distillation performance of the student. Motivated by this, we propose an ensemble of projectors for further improvement. There two motivations of using multiple projectors. Firstly, projectors with different initialization provide different transformed features, which is beneficial the generalizability of the student [77 32] Secondly, since we use the ReLU function to enable the projector to perform nonlinear feature extraction, the projected student features may contain zeros. the other hand, teacher features are less likely to be zeros due to the commonly used average pooling operation in CNNs. That is to say, the feature distribution gap between the teacher and the student is large with single projector. Therefore, using ensemble learning is natural way to achieve good trade-off between training

---

Algorithm Improved Feature Distillation via Projector Ensemble.
The pre-trained teacher, the structure of the student, training data X and labels. Parameter: Total iterations N, a and the number of projectors q. Initialization: Initialize different projectors and the student. Training:
Input:
1: for = do
2: Sample mini-batch data from X.
3: Generate S. T and the student's s prediction by forward propagation.
4: Update projectors and the student network by minimizing objective
5: end for
Output: The distilled student.
enror and generalizability. By introducing multiple projectors, the Modified Direction Alignment (MDA) loss as follows:
LMDA 11- -1-1---- - [
(f(s)).t))
(4)
where ((s:) Ei 9h (8:), q is the number of projectors and (=(-) indicates the k-th projector. By combining the distillation loss 4 and the classification loss together, we obtain the following objective function to train the student:
Liotal = LCE + aLADDA
(5)
where LCF is the cross-entropy loss and is hyper-parameter. The details of our method are shown Algorithm T
4
Experiments
We conduct comprehensive experiments to evaluate the performance of different methods and the effectiveness of the proposed projector ensemble-based feature distillation, on image classification task. Implementation details are as follows:
Baselines. We select representative distillation methods in various categories for comparisons,
including logit-based method KD [12), similarity-basek methods CC (23). SP 31 and Relational
Knowledge Distillation (RKD) (22), feature-based methods FitNets 124. FT 16), CRD [30] SRRL
33 and CID 4. The logit- and similarity-bar methods are projector free and the feature- based methods require additional projectors. FitNets and SRRL use convolutional kernels to transform the student features. FT adopts an auto-encoder to extract the latent feature representations of the student and the teacher. CRD maps student and teacher features into dimensinnional space while CID maps student features into the teacher space with linear projections. For simplicity, the proposed method constructs the projector by combining linear projection and the ReLU function.
Datasets. Two benchmark datasets are used for evaluation in our experiments. ImageNet (25 contains approximately 1.28 million training images and 50,000 validation images from 1,000 classes. The validation images are used for testing. Each image resized to 224x224. CIFAR-100 [18] dataset includes 50,000 training images and 10,000 testing images from 100 classes. Each image resized to 32x32. On ImageNet and CIFAR- 100, we adopt the commonly used random crop and horizontal flip techniques for data augmentation.
Teacher-student pairs. To validate the generalizability of different distillation methods, we select group of popular network architectures to form different teacher-student pairs. The teacher networks
include ResNet34 [9), DenseNet201 14, WRN-40-2 [34), VGG13 (77. ResNet32x4
[9] and
ResNet50 [9] The student networks comprise of ResNet18 (9. MobileNet [13), WRN-16-2 [34],
VGG8 (27]. ResNet8x4 9 and MobileNetV2 [26]. By combining different teacher and student
networks, we perform distillation between similar architectures (e.g., ResNee34-ResNetlll and
different architectures (e.g., DenseNee201-ResNeelle
Training. Following the settings of previous methods" the batch size, epochs, learning rate de- cay rate and weight decay rate are 256/64, 100/240. 0.1/0.1, and 0.0001/0.0005, respectively on
5

---

Table 1: Top-1 classification accuracy (%) on CIFAR-100 using horizontal ensembles with different teacher-student pairs. 1-Proj, 2-Proj, 3-Proj and 4-Proj indicate the number of single-layer projectors in the ensemble.
Pair
Student
w/o Proj
1-Proj
2-Proj
3-Proj
4-Proj
Teacher
VGG13-VGG8
70.74
73.76
73.84
74.21
74.35
74.18
74.64
ResNet3224-Resseetee
72.93
73.66
75.14
75.66
76.08
75.93
79.42
Table 2: Top-1 classification accuracy (%) on CIFAR-100 using deep projectors with different
teacher-student pairs. 2-MLP, 3-MLPP and 4-MLP indicate the depth of the projectors with different number layers.
Pair
Student
w/o Proj
1-Proj
2-MLP
3-MLP
4-MLP
Teacher
VGG13-VGGS
70.74
73.76
73.84
73.31
73.02
72.73
74.64
ResNee33xx4RReNet88x
72.93
73.66
75.14
75.12
74.56
74.30
79.42
ImageNer/CIFAR-100 The initial learning rate is 0.1 on ImageNet, and 0.01 for MobileNetV2, 0.05 for the other students on CIFAR-100, Besides, the learning rate drops at every 30 epochs on ImageNet and drops at 150, 180, 210 epochs on CIFAR-100. The optimizer is Stochastic Gradient Descent (SGD) with momentum 0.9. All the experiments are performed on an NVIDIA V100 GPU.
Hyper-parammeers. By following the conventions in CRD (30], we use the same settings for the prrer-parameters of KD, CC, SP, RKD, FitNets, FT and CRD. For SRRL and CID, the settings of wapeey-paaameee parameters are provided by the corresponding authors. For the proposed method, we set a = 25 and g = 3 by tuning with teacher-student pair BesNet33-RReseeee comparison, the typer--parmeetes of different methods are fixed in all experiments.
on ImageNet. For fair
4.1 Ablation Studies
This section studies the effectiveness of the proposed projector ensemble method, and how different ensemble strategies affect the performance. In this experiment, two different network architectures, ie. VGG-style and ResNet-style networks are used for illustration in Tables I
and
Horizontal ensemble of projectors. Table I shows the top-1 classification accuracy of the proposed projector ensemble with different number of projectors. verifies that imposing a projector improves the distillation performance when the feature dimensions of the student and the teacher are the same. potential reason is that the projector helps disentangle the two learning tasks (i.e. distillation and classification) and improves the quality of student features. In addition, by integrating multiple projectors, the proposed method further increases the classification accuracy by a clear margin with various numbers of projectors.
Deep cascade of projectors. Another com- mon way to modify the architecture is to in-
Table 3: Top-1 accuracy (%) on CIFAR- 100 using deep projectors. 2x2-MLP and 2x3-MLP indicate two-layer MLPs with wider hidden layers.
crease the depth of the projector. Table 2
demonstrates the changes of distillation per- formance by gradually stacking non-linear projections. In this table, 2-MLP, 3-MLP and 4-MLP are multilayer perceptrons and each layer outputs m-ddimensional features fol- lowed by ReLU activation. For instance, the output of 2-MLP is gs;)) (W2a(W1s,
Teacher Student
VGG13
ResNet32x4
VGG8
ResNet8x4
2-MLP
73.31
75,12
2x2-MLP
73.31
75.00
2x3-MLP
73.37
75.13
where W1 E m and W E R are weighting matrices. It is shown that simply increasing the depth of the projector does not improve the performance of the student and tends degrade the effectiveness of the projector. We hypothesize that with the increase of depth, the teacher's 's features can be overfitted by the projector. Besides, we also evaluate the effect of deep projectors with wider hidden layers as shown in Table 3 In this experiment, the hidden dimensions of 2x2-MLP and 2x3-MLPP are 2 m and 3 x m, respectively. Experimental results demonstrate that the improvement marginal by increasing the width of deep projectors.

---

Table 4:
Diversity of projectors in the proposed distillation framework on CIFAR-100 with teacher-
student pair S
Epoch
10
40
80
120
160
200
240
L2 distance
440.93
394.57
372.82
360.69
298.86
213.29
206.32
Table 5: Top-1 classification accuracy and standard deviation (%) on CIFAR-100 with different
teacher-student pairs.
Teacher WRN-40-2
VGG13
ResNet32x4
ResNet50
ResNet50
Student
WRN-16-2
VGG8
ResNet8x4
VGG8
MobileNetV2
Teacher
75.61
74.64
79.42
79.34
79.34
Student
73.22:+0.13
70.74=0.31
72.93=0.28
70.74=0.31
65.03=0.09
KD
74.92+-0.28
02.9
73.33+-0.25
73.81+0.13
67.35+00.22
CC
0.25
7..71+0.24
72.97+0.117
7..25+
6..43+0.15
SP
73.83+00112 72.68
72.94+00.3
73.34+0034
68.08+0.38
RKD
73.35+0009
7..48=0005
71.90#00.11
71.50+0007
64.43+0.42
FitNets
73.58
71.02
73.50=0.22
70,69
63.161:.47
FT
73.25+0.20 70.08
72.86:1.12
10.29
60.99
CRD
75.48 73.94
10.51
74.30-10.14
69.11
SRRL
75.59=0.17 73.44=0.07
7..33=0.04
74.23+0.08
68.41=0.54
Ours
76.02=0.100
74.35=0011
76.08=00.3
74.28
69.81=0442
4.2 Diversity of Projectors
This section investigates the diversity of projectors in the proposed ensemble- distillation framework. this experiment, we set = 2 for demonstration. The L.2 distances between two projectors at different epochs are given in Table 4 We gradually decreases with the increase training epochs and appears to converge after 200 epochs. In Supplementary Material, discuss how promote the diversity of projectors from the perspectives of using different initialization methods and adding a regularization term.
that the diversity of projectors
4.3 Results on CIFAR-100
Table reports the experimental results CIFAR- 100 with five teacher-student pairs. We our method for three times with dif- ferent seeds and obtain the average accuracy. Since CID requires different for different pairs to achieve good perfor- mance, we omit it for comparisons on CIFAR- 100. Among the projector-free distillation methods, the logit-based method KD shows
Table 6: Top-1 accuracy (%) of our method with different activation functions on CIFAR-100.
Teacher Student
VGG13
ResNet32x4
VGG8
ResNet8x4
w/ ReLU (Ours)
74.35+:0.12
76.08+0..3
w/ GELU
74.39:10.18
76.32.10.27
w/o activation
73.46:10.42
75.04
better performance compared to the similarity-based methods CC, SP and RKD. Furthermore, KD outperforms the projector-baad methods FitNets and FT in most cases. Since FitNets is designed to distill the intermediate features, its performance is unstable for teacher-student pairs using different architectures. FT uses an auto-encoder as the projector to extract latent representations of the teacher, which may disturb the discriminative information to some extent and consequently degrade the performance. The recently proposed feature distillation methods CRD and SRRL show competitive performance compared to the previous methods by distilling the last layer of features. By harnessing the power of both distilling the last features and projector ensemble, the proposed method consistently achieves the highest accuracy CIFAR-100.
We further investigate the effect of the activation function. Table 6 shows that the addition of the ReLU activation function has significant positive impact on the performance of the proposed method. The reason that the lack of an activation function and the non-linearity introduced by limits the diversity of the projectors, as group of linear projections can be mathematically reduced to single linear projection through sum-pooling, which will degrade the distillation performance. Recently, Gaussian Error Linear Units (GELU) [10] has received attention because of its effectiveness

---

Table 7: Classification accuracy (%) ImageNet with different teacher-student pairs (a) ResNet34- ResNet18, (b) ResNec500-MobleNNe, (c) DenseNec201-ResNNell and (d) CenseNer201-Moobleee
Pair
Accuracy
Student
KD
SP
CRD
SRRL
CID
Ours
Teacher
Top-1
69.75
70.83
70.94
70.85
71.71
71.86
71.94
73.31
(a)
Top-5
89.07
90.15
89.83
90.12
90.58
90.63
90.68
91.41
T
69.06
70.65
70.14
71.03
72.58
72.25
73.16
76.13
(b)
Top-5
88.84
90.26
89.64
90.16
91.05
90.98
91.24
92.86
Top-1
69.75
70.38
70.75
70.87
71.76
71.99
72.29
76.89
(c)
Top-5
89.07
90.12
90.01
89.86
90.80
90.64
90.99
93.37
Top-1
69.06
69.98
70.34
70.82
72.28
71.90
73.24
76.89
(d)
Top-5
88.84
89.93
89.63
90.09
90.90
90.97
91.47
93.37
ResNet34-ReNNe118
ResNet50-MobileNet
74
Top-1 Accuracy (%)
70
Top-1 Accuracy (%6
66
KD
66
SP
62
62
CRD
58
SRRL
58
SRRL
54
CID
Ours
Ours
50
30
90
100
Epoch
Epoch
Figure 3: Top-1 accuracy of different methods on ImageNet with different number epochs and different teacher-student pairs.
Transformer 5 6
We replace ReLU with GELU in the proposed method to test the performance. It shown that the performance of our method can be further improved by using GELU as activation.
4.4 Results on ImageNet
The performance of the students distilled by different methods are listed in Table Compared to the settings previous methods (30 33 4] we introduce more teacher-student pairs for evaluation in this experiment so that the generalizability of different methods can be better evaluated. As presented in the table, feature distillation methods (CRD, SRRI., CID and our method) outperform both the logit-based method (KD) and the manilarity-basee method (SP) in most cases.
One major difference between CRD and the other feature distillation methods is the way of feature transformation. CRD transforms teacher and student features simultaneously while the other methods only transform student features. By solely mapping student features into the teacher's space, the origi- nal teacher feature distribution can be preserved without losing discriminative information. Therefore, SRRL. CID and our method obtain better performance than CRD. Besides, our method consistently outperforms the state-of-the-an methods SRRL and CID with different teacher-student pairs. With pair MotoeeeeetO--Moolllleii pared to the second best method in terms of top-1 and top-5 accuracy, respectively. MobileNet (4.2M parameters) distilled by our method can obtain similar performance and reduce about 80% of the parameters compared to the ResNet34 (21.8M, parameters). Figure 3 reports the top-1 accuracy of different methods with different training epochs. shown that the proposed method converges faster than the other distillation methods.
the proposed method obtains 0.96% and 0.50% improvements com-
In 21, the authors observe that better teacher may fail to distill better student. Such phenomenon also exists in Table For example, compared to the pair BesNeet5-MMobileNe most of the methods distill worse student by using better network DenseNer201 as the teacher. One plausible expla- nation for this phenomenon is that the knowledge of a better teacher is more complex and is more difficult to learn. To alleviate this problem, TAKD (1 introduces some smaller assistant networks to facilitate training. Densely Guided Knowledge Distillation (DGKD) (28) further extends TAKD

---

Table 9: Training times (in second) of one epoch and peak GPU memory usages (MB) of different
methods on ImageNet with teacher-student pair DenseNer201-ResNerils
Method
KD
SP
CRD
SRRL
CID
Ours
Time
2,969
2,989
3,158
3,026
3,587
2,995
Memory
11,509
11,509
15,687
11,991
12,021
11,523
with dense connections between different assistants. However, the training costs of these methods are greatly increased by using the assistant networks. As shown in Table 7 the proposed method has the potential to alleviate this problem without introducing the additional networks.
We compare the training costs of different methods in Table Since KD and SP are projector-free methods, their training costs are lower than that of the feature distillation methods. The training cost of our method is slightly higher than KD and SP because optimization of the student. On the other hand, the proposed method only uses a naive direction alignment loss to distill the knowledge. Therefore, the computation complexity and memory usages are lower compared to the other fcature-based methods.
multiple projectors to improve the
Two recently proposed methods, namely Attention-bast based Feature Distillation (AFD)
Table 8: Comparisons of the proposed method and distillation methods using multiple layers of features.
[15] and Knowledge Review (KR) (2) are also introduced for comparisons as reported in Ta-
Teacher
ResNet34 ResNet18
ResNet50 MobileNet
Student
8 Unlike methods that utilize the last layer of features for distillation (CRD, SRRI., CID and ours), AFD and KR propose to extract information from multiple layers of features. Table shows that the proposed method per- forms better than AFD and KR with different teacher-student pairs terms of the compar- isons of top-1 I and top-5 accuracy, which in- dicates that using the last layer of features is sufficient to obtain good distillation performance on ImageNet.
Acc
Top-1
Top-5
Top-1
Top-5
Teacher
73.31
91.41
76.13
92.86
Student
69.75
89.07
69.06
88.84
AFD
71.38
KR
71.61
90.51
72.56
91.00
Ours
71.94
90.68
73.16
91.24
5 Conclusion
This paper studies the effect of the projector in feature distillation and proposes projector ensemble- based architecture to improve the feature projection process from the student to the teacher. We first investigate the phenomenon that the addition of a projector improves the distillation performance even when the feature dimensions of the student and the teacher are the same. From the perspective of multi-task learning, we speculate that the student network without a projector tends to overfit the teacher's 's feature space and pays less attention to the task of discriminative feature learning for classification. By imposing a projector on the student network to mitigate the overfitting problem, the student's 's performance can be increased. Based on the positive effect of the projector in feature distillation, we propose an ensemble of projectors for further improvement. Empirical results on ImageNet and CIFAR-100 show that our method consistently achieves competitive performance with different teacher-student combinations, compared to other sfute-of-thee art methods.
I Limitations and future work. In addition to the image classification task, the proposed method can be further applied other downstream tasks (e.g., object detection and semantic segmentation), which can be explored in future work. Besides, the proposed method focuses on using the direction alignment loss for distillation. How effectively and efficiently integrate logits and similarity information into the proposed framework is a potential research direction.
6 Acknowledgments
This work is partially supported by Australian Research Council DF200101610 and CSTRO's Re- search Plus Science Leader Project R-91559.

---

References
[1] Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang. Yan Feng, and Chun Chen. Cross-layer distillation with semantic calibration. AAAI, pages 7028 7028-7036, 2021.
[2] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR, pages 5008-5011 2021.
[3] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, pages 15750-15758, 2021.
[4] Xiang Deng and Zhongfei Zhang. Comprehensive knowledge distillation with causal interven- tion. In NeurIPS, 2021.
[5] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprini arXiv: (atn:.8880000 2018.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
[7] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey, IJCV, 1789-1178,-1889, 2021.
[8] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In NeurlPS, pages 21271-21284 2020.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[10] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 1rXiv:1606.00841. 2016.
[11] Byeongho Hco, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In ICCV, pages 1921-1930, 2019.
[12] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 103.00550 2015.
[13] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1744. 04861, 2017.
[14] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, pages 4700-4708, 2017.
[15] Mingi Ji, Byeongho Heo, and Sungrae Park. Show, attend and distill: Knowledge distillation via attention-based feature matching. In AAAI, pages 7945-7952, 2021.
[16] Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. In NeurlPS, 2018.
[17] Nikos Komodakis and Sergey Zagoruyko. Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. ICLR, 2017.
[18] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009.
[19] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, pages 1097-1105, 2012.
10

---

[20] Zhuang Liu, Hanzi Mao, Chao Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. arXiv preprint arXiv:2201.03555, 2022.
[21] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In AAAI, pages 5191-5198, 2020.
[22] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, pages 3967-3976, 2019.
[23] Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In ICCV, pages 5007-5016, 2019.
[24] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: 1412.5550, 2014.
[25] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. IJV,, 115(3):211 2015.
[26] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pages 451004520, 2018.
[27] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.15555 2014.
[28] Wonchul Son, Jaemin Na, Junyong Choi, and Wonjun Hwang. Densely guided knowledge distillation using multiple teacher assistants. In ICCV, pages 9395-9404, 2021.
[29] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pages 1-9, 2015.
[30] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020.
[31] Frederick Tung and Greg Mori. semilarity-preseevviie knowledge distillation. In ICCV, pages 1374-1374, 2019.
[32] Xiaofang Wang, Dan Kondratyuk, Eric Christiansen, Kris M Kitani, Yair Alon, and Elad Eban. Wisdom committees: An overlooked approach to faster and more accurate models. arXiv preprint arXiv:2012.01998,, 2020.
[33] Jing Yang, Brais Martinez, Adrian Bulat, and Georgios Tyimiropoulos. Knowledge distillation via softmax regression representation learning. In ICLR, 2021.
[34] Sergey Zagoruyko and Nikos Komodakis. 1rXiv:1605.07144 046, 2016.
Wide residual networks. arXiv preprint
[35] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR, pages 1320-43288 2018
[36] Youcai Zhang, Zhonghao Lan, Yuchen Dai, Fangao Zeng. Yan Bai, Jie Chang, and Yichen Wei. Prime-aware adaptive distillation. In ECCV, pages 658 658-674, 2020.
[37] Zhi-Hun Zhou, Jianxin Wu, and Wei Tang. Ensembling neural networks: many could be better than all. Artificial intelligence, 137(1-2):239 2002.

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section S
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "tYV7GxVJQMH", "reviewer": "Reviewer_2exr", "summary": "The paper presents a feature matching-based distillation method that makes use of a set of feature projectors to better align the features of the student and teacher network. \nThe authors show that such an ensemble of projectors can improve the distillation performance further even if the student and teacher have the same feature dimension.\nSome analysis is conducted to show the reason of introducing a projector can benefit the student network learning.\nExperiments were performed on benchmark datasets using various teacher-student network structure pairs to compare with different methods.", "questions": "1. Line 251, which table is this referring to? Is it table 1?\n2. Is there any limitation in the training of an ensemble of projectors? Is the training time impacted by the number of projectors and/or the depth of the projectors?", "limitations": "The authors have a section for discussing limitations and future works. However, it did not directly address the limitation of the multiple projectors, e.g. training time, memory usage, etc.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well written with comprehensive experiments.\nWeakness:\n- The idea is simple but the authors should explore more options for an ensemble of projectors, e.g. ensemble of various structure projectors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "r6cCftIJiXw", "reviewer": "Reviewer_Y25R", "summary": "This is an experiment-driven paper. The authors do lots of experiments and comparisons to demonstrate the effectiveness of feature projectors in feature distillation. ", "questions": "See weaknesses. ", "limitations": "See weaknesses. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe authors do lots of experiments and comparisons to show the superiority of feature projectors in feature distillation. \n\nWeaknesses:\n\n1. This is an experiment-driven paper. The authors do lots of experiments and give empirical analyses. However, I still think the contributions are not enough for NeurIPS. \n2. The authors should re-organize the Method section. It is not obvious from one equation to the next. And the notations are somewhat confused. \n    - The notations $s^p$ and $W^p$ is confused. I suggest to use $t$ as the $t$-th iterations and use $W_p$ to represent the parameters of projector. It is more clear. \n    - The Eq. (2), Eq. (3) and Eq. (4) should be more clear. It is not easy to understand, especially for the junior researchers. For example, Eq. (3) can be denoted as $\\frac{\\partial L_{DA}}{\\partial s_i^p}=\\frac{\\partial L_{DA}}{\\partial g(s_i^p)}\\frac{\\partial g(s_i^p)}{\\partial s_i^p}$. The last term of Eq. (4) can be denoted as $\\frac{\\partial L_{DA}}{\\partial g(s_i^p)}\\frac{\\partial g(s_i^p)}{\\partial W^p}$. \n3. The hypothesis (Line 125-126) is a little coarse. I think the main contribution in this paper is the analysis about the effectiveness of the projector. Therefore, the authors should give deeper analyses. \n4. In experiments, table is more suitable and clear to demonstrate the performance than figure. I recommend the authors use tables instead of Fig. 3 and Fig. 4.  \n5. Considering the feature distillation, some experiments about CUB-200-2011, Cars-196 and SOP should be included. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "T5OMXV4g7lO", "reviewer": "Reviewer_52Wo", "summary": "The authors discuss the phenomenon that using a projector on a student’s feature can improve the performance of distillation when the feature dimensions are the same.\nThey focus on the projector between teachers and students, which is unnoticed in the past.\nThen the authors propose a simple ensemble method to further improve the performance.\nThey conduct comprehensive experiments in this paper and the authors claim the superiority of their proposal.\nA complete story, from the phenomenon to the essence, which is simple and effective.\n", "questions": "\n“the student network benefits from a projector even if the feature dimensions of the student and teacher are the same” , Is this conclusion first discovered and proposed by the author of this paper? The word used in the abstract is “observe”.\n\nInitialization of different projectors is not clear, do different initialization methods have a big impact on the experimental results?\n", "limitations": "As mentioned above in the Weakness.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1.\tThe method is simple and easy to implement.\n2.\tExperiments are detailed and code is provided.\n3.\tThis paper is well written.\nWeakness\n1.\tThe novelty of the proposed method is limited. Are you sure that there are no more researchers who have used this method to improve the performance of their model?\n2.\tthe number of projectors in the ensemble seems to be sensitive, how to determine this hyperparameter appropriately in practice?\n3.\tThe ablation study does not provide the effect of different numbers of projectors on distillation when the feature dimensions are different.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "C4xb4MDiPZ3", "reviewer": "Reviewer_EwZs", "summary": "This work proposed a new feature distillation method via projector ensemble. In particular, different from the traditional single-projector method, it introduces multiple projectors and computes the average output to perform KD. Further, two different designs including a single-layer projector and MLP projector are discussed and compared. Experiments on popular datasets are conducted to verify the effectiveness of the method.", "questions": "Please refers to the weaknesses mentioned above.", "limitations": "As mentioned in the paper, the proposed method is only verified on classification. Its effectiveness on other tasks can be further explored. ", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths are listed below:\n(1)\tThe proposed method achieves significant performance improvement. \n(2)\tThis proposed method is simple and easy to be reproduced.\n\nThe weaknesses are listed below:\n(1)\tThe novelty is limited. The single-layer projector is widely used in KD. Further, a simple extension from single-layer to multi-layer is rather simple. It is easy to come up with this idea. \n(2)\tLack of references in all of the tables. It is recommended to add references of comparing methods in tables.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1qHzafPrddc", "reviewer": "Reviewer_n8Zt", "summary": "This paper proposes a feature distillation method based on an ensemble of multiple projectors.  The model uses multiple projectors to project the features of a student model, and the Direction Alignment loss are calculated between the multiple projected features and the teacher features. Some analysis are given to explain why projector is helpful. Experimental results show that the proposed method outperforms previous competitors.", "questions": "1. I suggest the authors give some visualization results or experimental analysis of the projector behaviors during and after training. If different projectors show significant differences, it would be better to give some theoretical analysis about what brings them variety. \n2. The theoretical analysis in section 3.1 needs to be clearer. Also, the authors need to give explanations of why more projectors are better under their feature gradients perspective.", "limitations": "The authors have addressed the limitations and potential negative societal impacts.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper focuses on the projectors in feature distillation models and proposes a simple but effective ensemble method. This angle is interesting and the proposed model can be widely applicated.\n\n2. The writing is clear and easy to understand. \n\n3. Experimental results show that the proposed model obtains promising results.\n\nWeaknesses:\n\n1. The ensemble strategy is too simple. The authors simply add all the projected features together.  In fact, ensemble strategy plays a critical role in ensemble learning. In many situations, simply averaging the outputs of weak learners cannot bring improvements.\n\n2. This paper lacks discussions about the projector behaviors, especially their diversity. The authors only give a simple description of \"projectors with different initialization would provide different transformed features, which is beneficial to the generalizability of the student\".  However, different initialization cannot guarantee enough variety between projectors. Moreover, the different projected features are simply added together to calculate losses. There is no special architecture and training designs to promote the projector diversity. In experiments, there is also no experimental analysis of the projector behaviors.\n\n3. The theoretical analysis is not clear enough. In section 3.1, the authors try to analyze why projector is helpful by comparing the feature gradients with and without projector. The explanation is that the non-linear transformation updated from previous data helps to better capture the global feature distribution. This is not convincing enough to me. Furthermore, this explanation cannot explain why multiple projectors are better.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept