Masked Distillation with Receptive Tokens
Authors: Tao Huang
Yuan Zhang
Shan You
Fei Wang
Chen Qian
Jian Cao
Chang Xu
Venue: ICLR 2023 poster (2023)

Abstract: Distilling from the feature maps can be fairly effective for dense prediction tasks since both the feature discriminability and localization information can be well transferred. However, not every pixel contributes equally to the performance, and a good student should learn from what really matters to the teacher. In this paper, we introduce a learnable embedding dubbed receptive token to locate the pixels of interests (PoIs) in the feature map, with a distillation mask generated via pixel-wise attention. Then the masked distillation will be performed via the pixel-wise reconstruction. In this way, a distillation mask refers to a pattern of pixel dependencies. We thus adopt multiple receptive tokens to investigate more sophisticated and informative pixel dependencies within feature maps to enhance the distillation. To obtain a group of masks, the receptive tokens are learned via the regular task loss but with teacher fixed, and we also leverage a Dice loss to enrich the diversity of obtained masks. Our method dubbed MasKD is simple and practical, and needs no priors of ground-truth labels, which can apply to various dense prediction tasks.  Experiments show that our MasKD can achieve state-of-the-art performance consistently on object detection and semantic segmentation benchmarks.

Published as a conference paper at ICLR 2023
MASKED DISTILLATION WITH RECEPTIVE TOKENS
Tao Huang 12+ Yuan Zhang Shan You
Fei Wang" Chen Qian² Jian Cao Chang Xu
School of Computer Science, Faculty of Engineering. The University of Sydney
2 "SenseTime Research "Scoool of Software and Microelectronics, Peking University *University of Science and Technology of China
ABSTRACT
Distilling from the feature maps can be fairly effective for dense prediction tasks since both the feature discriminability and localization priors can be well trans- ferred. However, not every pixel contributes equally to the performance, and a good student should learn from what really matters to the teacher. In this pa- per, we introduce a learnable embedding dubbed receptive token to localize those pixels of interests (PoIs) in the feature map, with distillation mask generated via pixel-wise attention. Then the distillation will be performed on the mask via pixel-wise reconstruction. In this way, distillation mask actually indicates a pattern of pixel dependencies within feature maps of teacher. We thus adopt multiple receptive tokens to investigate more sophisticated and informative pixel dependencies to further enhance the distillation. To obtain group of masks, the receptive tokens are learned via the regular task loss but with teacher fixed, and we also leverage a Dice loss to enrich the diversity of learned masks. Our method dubbed MasKD is simple and practical, and needs no priors of tasks in applica- tion. Experiments show that our MasKD can achieve state-of-the-at performance consistently on object detection and semantic segmentation benchmarks. Code is available at htttps:: ///pttttbbb/bt www.hhu.....mmaaaaae www.toowt.//ooootooooootoooo..tt MasKD
INTRODUCTION
Recent deep learning models tend to grow deeper and wider for ultimate performance (He et al., 2016 Xie et al.] 2017; [i et al. 2019). However, with the limitations of computational and memory resources, such huge models are clumsy and inefficient deploy on edge devices. As friendly solution, knowledge distillation (KD) (Hinton et al.) 2015 Romero et al. (2044) has been proposed to transfer knowledge in the heavy model (teacher) to small model (student). Nevertheless, applying KD on dense prediction tasks such as object detection and semantic segmentation sometimes cannot achieve significant improvements as expected. For example, Fitnet (Romero et al. 2014) mimics the feature maps of teacher element-wisely but it has only minor improvement in object detection
Therefore, feature reconstruction for all pixels may not be good option for dense prediction, since not every pixel contributes equally to the performance. Many followups (Li al. 2017 Wang et al.) 2019 Sun al. 2020 Guo al. [202]) thus dedicated to show that distillation on sampled valuable regions could achieve noticeable improvements over the simple bascline methods. For example, 2017) distills the positive regions proposed by region proposal network (RPN) of the student; FGFT (Wang et al.] 2019) and TADF (Sun et al.) 2020) imitate valuable regions near the foreground boxes; Defeat (Guo et al. 2021) uses ground-truth bounding boxes to balance the loss weights of foreground and background distillations; GID (Dai et al. 2021 selects valuable regions according to the outputs of teacher and student. These methods all rely on the priors of bounding boxes; however, are all pixels inside the bounding boxes necessarily valuable for distillation?
Mimicking (Li al.
The answer might be negative. As shown in Figure I the activated regions inside each object box much smaller than the boxes. Also, different layers, even different strides of features in FPN,
"Equal contributions. 'Correspondence to: Shan You <youshan @ senset me. com>. Fitnet (Romero al. 2014) improves Faster RCNN-R50 by only 0.5%, while has no gain RetinaNet- R50 (see Table

---

Published as a conference paper at ICLR 2023
Annotations
Stage O
Stage 1
Stage 2
Stage 3
Stage 4
Figure I: Visualization of learned masks on COCO dataset. In Faster RCNN-R101 model, the earlier stages in FPN focus more on small objects, while the later ones focus on larger objects. Complete visualization results can be found in Appendix A.6 Zoom up to view better.
have different regions of interest. Moreover, objects that do not exist in ground-truth annotations would be treated as "background", but they actually contain valuable discriminative information. This inspires us that we should discard the ground truth boxes and select distillation regions on fine-grained pixel level.
In this way, we propose to learn pixel-wise mask as an indicator for the feature distillation. An intuitive idea is that we need localize what pixel the teacher's feature map is really meaningful to the task. To this end, we introduce learnable embedding dubbed receptive token to perceive each pixel via attention calculation. Then a mask is generated to indicate the pixels of interests (Pols) encoded by a receptive token. As there may be sophisticated pixel dependencies within feature maps, we thus leverage multiple receptive tokens in practice to enhance the distillation. The receptive tokens well as the corresponding masks can be trained with the regular task loss with the teacher fixed. For the group of masks, we adopt Dice loss to ensure their diversity, and devise a mask weighting module accommodate the different importance of masks. During distillation, also propose to customize the learned masks using the student's 's feature, which helps our distillation focus more on the pixels that teachers and students really care about simultaneously.
Our MasKD is simple and practical, and does not need task prior for designing masks, which is friendly for various dense prediction tasks. Extensive experiments show that, MasKD achieves state-of-the-art performance consistently on object detection and semantic segmentation tasks. For example, MasKD significantly improves 2.4 AP over the Faster RCNN-R50 student on object de- tection, while 2.79 mloU over the DeepLab V3-R 18 student on semantic segmentation.
2 RELATED WORK
Knowledge distillation on object detection. Knowledge distillation methods on object detection task have been demonstrated successful in improving the light-weight compact detection networks with the guidance of larger teachers. The distillation methods can be divided into response-based and feature-based methods according to their distillation inputs. Response-based methods (Hinton et al,, 2015] Chen et al. 2017: Li et al.) (017)) perform distillation on the predictions (e.g.. clas- sification scores and bounding box regressions) of teacher and student. contrast, feature-based methods (Romero et al.) 2014] Wang et al.) 2019; Guo et al., (2021) are more popular as they can distill both recognition and localization information in the intermediate feature maps. Unlike the classification tasks, the distillation losses in detection tasks will encounter an extreme imbalance between positive and negative instances. To alleviate this issue, some methods (Wang et al. 2019; Sun et al. 2020; Dai al.) 2021: Guo et al. 2021 Yang et al.) 2021) propose distill the fea- tures on various sophisticatedly selected sub-regions the feature map. For instance, FGFI (Wang al. 2019) selects anchors overlapping with the ground-truth object anchors distillation regions;
2

---

Published as a conference paper at ICLR 2023
Mask Weight
Logits
Teacher
RPN
Roi-Aligned
Localication
Backbone
Distillttion
Datilaatton
Tokena
Localization
loss
RPN
Boi-Aligned
Stadent
Takon Misked Feature
Logits
mede Cle loss
Backbone
JUPN Loss
Figure 2: Overview of our KD framework on Faster R-CNN. We perform our masked distillation on the feature pyramid, where set of receptive tokens are proposed to generate the masks, and mask weighting module is conducted adapt the loss weights for each mask. denotes Hadamard product, and ( denotes matrix multiplication.
GIDs (Dai et al. 2021) distills student detectors based discriminative instances selected by the predictions of teacher and student; In this paper, we propose to learn the pixels of interests in the feature map without box priors, and perform feature distillations
finer pixel level.
Knowledge distillation on semantic segmentation. Knowledge distillation methods on semantic segmentation often focus preserving the structural semantic relations between teacher and stu- dent. He et al. (He et al. 2019) optimize the feature similarity in a transferred latent space using a pre-trained autoencoder alleviate the inconsistency between the features of teacher and student. SKD (Liu et al.) 2019) performs a pairwise distillation among pixels to retain the pixel relations and adversarial distillation score maps distill holistic knowledge. IFVD (Wang al. 2020) transfers the intra-class feature variation from teacher to student for more robust relations with class- wise prototypes. CWD (Shu et al. 2021) proposes channel-wise distillation for a better mimic on the spatial scores along the channel dimension. CIRKD (Yang et al. (022)) proposes to learn better semantic relations from the teacher by adopting intra- intra-image and cross pross-image relational distillations.
DISTILLATION WITH FEATURE RECONSTRUCTION
In order distill the feature maps of teacher, typical manner is mimic the tensor pixel-wisely (Romero et al. 2014 Chen al. 2017). Formally, with the feature maps F(1) E xx and F(*) E XRC X of teacher and student networks, where C, H, W denote the number of channels, height, and width, respectively, the mimicking can fulfilled via feature reconstruction as
p - (0w)-((F(()) (F(())
Liiimic
HWC
where o is a linear projection layer to adapt F() to the same resolution as F(1), For example, the mimic loss Cmiuie is usually conducted on the outputs of feature pyramid network (Lin et (017a) (FPN) for detection tasks.
However, on dense prediction tasks, the predictions are highly determined by its corresponding spatial regions in the feature maps. Treating all regions equally would weaken student's attentions on those small but critical regions, egg. the feature map contains both smaller objects and larger ones, while the larger ones obtain larger loss values they have larger areas. Moreover, different regions often have different importance to the predictions, e.g., foreground is usually more important than background in detection. Recklessly imitating the unimportant noise features would also limit the distillation performance.
As result, typical improvement of feature distillation in recent methods (Wang al. 2019 Sun
(et al.) 2020; Guo et al., (202T) is to reconstruct the features on selected and separate regions (masks). Speecfically. suppose have set of K generated masks M E TR Kxxxx W these methods generate

---

Published as a conference paper at ICLR 2023
K' masked features and perform distillation separately, i.e.,
1 # K i= CE M.,
F M, |
Lieature
However, the masks in these methods are usually generated with the priors of bounding boxes, which are highly influenced by the annotations in ground-truth or the configurations of anchors. For exam- ple, the unlabeled boxes in the annotation data (see Figure D or unsuitable scales of anchors would assign wrong classes (e.g., foreground and background) over-valued pixels, thus weaken the performance. In this paper, we propose novel mask genera- tion method to locate the finer pixel-level interest areas, with neither ground-truth annotations nor predictions used.
some pixels, resulting in overlooked or
4 PROPOSED APPROACH: MASKD
Our MasKD comprises two stages: the mask learning and the conventional knowledge distillation stages. In the mask learning stage, we learn the masks from trained teacher, then adopt the learned masks to the knowledge distillation stage. Note that these two stages can be merged into one stages like previous KD methods, and the cost of the first mask learning stage is negligible. In this paper, we formulate them separately for better understanding.
4.1 LEARNING MASKS WITH RECEPTIVE TOKENS
We introduce learnable embedding E RTCC dubbed mask tokens to represent T pixel depen- dencies in the feature map, then the pixels of interests (Pols) can be obtained by calculating the similarities between mask tokens and the spatial points in the feature map:
M o (EF()))
(3)
where a denotes Sigmoid function, for simplicity, we flatten the teacher's ' feature map F(() into shape (C, H W) in the paper, and thus the masks (9) have shape of (T, H W).
To learn the masks with task knowledge, we propose to train them with teacher's 's task loss, where masked feature f(()) Σ M, F(() is used replace the original feature F(O), Then the loss can be treated as an observation of mask quality, and we fix the teacher's weights and minimize the loss to foree the masks focus on the substantial pixels.
However, simply minimizing Clos would lead to an undesired collapse of the mask tokens. Specifi- cally, some mask tokens will be learned to directly recover all the features (the learned mask is filled with everywhere). However, we want to partition the feature into meaningful regions. To make the masks represent different spatial pixels, we propose mask diversity loss based on Dice coefficient:
Laiv
1 T2
Σ
Paico (M M
==
with
2 N N
-== agb;
Pdice (a, b)
(5)
Σj=1 +f
where N and b E R are two vectors. Dice coefficient Pdice is widely used to measure the similarity of two images segmentation tasks. By minimizing the coefficients of each mask pair, make masks associated with different Pols. As result, the training loss of mask tokens is composed of teacher's task loss and mask diversity loss:
(6) where y is a factor for balancing the loss. Note that we simply set = 1 in all experiments as the mask tokens are easy to converge and the masked feature F(() is only used for learning semantic- aware mask tokens. In the distillation stage, we use the teacher's original feature to get predictions.
Lroken c(() + PLaivv
For example, the teacher Cascade Mask RCNN ResNeXt101 has 45.6 mAP COCO val set, and the masked one is 45.4.
4

---

Published as a conference paper at ICLR 2023
you
Radupe
ICAM
Toloma
Ang Pook
2554% 47
4.2255e05 1.0857.004
Sigoodd
IF.
Redupe
Soltaaa
J.C.AI
1.600e-00
2.440e-01
7.5599e011
(a) Computation of masked features.
(b) Visualization of learned masks and weights.
Figure 3: (a) Illustration of the computation of masked features. We learn receptive tokens to gener- ate the masks containing pixels of interests, then multiply the masks to the original features to get the masked features, along with a mask weighting module learn the mask importance. (b) Visualiza- tion of learned masks and their corresponding importance stage 4 of FPN in Faster RCNN-R101.
Mask weighting module. By learning mask tokens, we can obtain multiple masks of different Pols, yet not all masks are of the same importance. For example, we would assign larger loss weights on foreground distillations than on background, the foreground regions are usually more important in detection. In this paper, we thus propose a mask weighting module to determine the mask importance independently for each image. Concretely, we conduct simple convolution-based module (see Figure 3 (a)) to predict the weights each mask in each feature map, then weight the importance vector w onto the masked features,
(1)) Σ
wi (M + F(!).
Therefore, the masks on important pixels could obtain larger weights, while the pixels redundant the task get small weights. As illustrated in (b) we visualize the masks and the correspond- ing importance on the last stage (stride 32) of FPN. The background and meaningless empty masks have negligible weights, while the masks associated with real objects have larger weights.
4.2 DISTILLATION WITH LEARNED MASKS
Now we formulate our feature distillation in the knowledge distillation stage. With the learned masks ( straightforward way to reconstruct the teacher's feature is adopting Eq. as previous methods. However, as previously discussed in Section 4.1 our masks contain both important Pols and less valuable ones, and we thus propose mask weighting module to capture the importance of each mask, which can be used in distillation for better balance.
Instance-dependeen mask weights. With the learned mask importance in Section 4.1 we can naturally adopt these importance weights the loss weights each mask; i.e., if mask critical to the performance, it should be highlighted in teacher-student knowledge transfer. Therefore, Eq. can be reformulated
T
W
M & F(O (, o o(FOO)
Lleature
(8)
C
HxW M.
i=1
Note that the mask importance determined instance-wisely, and thus our adaptive mask loss weights be more precise and effective than pre-assigning fixed loss weights in heuristic way.
Customizing masks with student feature. feature distillation, construct all the feature maps from the teacher finely due to their capacity gap. Forcing the student to leam from the teacher would disturb the optimization for those redd-to-eccnstructt pixels. Besides, some pixels that meaningless the student could also exist, and the unnecessary reconstructions on them may also weaken the distillation performance. In MasKD, we propose to alleviate these problems by refining the masks with the student feature.
difficult for the student to re-
5

---

Published as a conference paper at ICLR 2023
The teacher's learned mask tokens can provide precise observation of the reconstruction rate of the student's s feature, i.e., if a pixel fully recovers the teacher's corresponding feature, their similarities to the mask tokens will be the same. In contrast, the similarities will be different when the features vary significantly. Therefore, we propose to customize the mask regions by multiplying the student's mask regions, and the distillation pixels should consist of the ones both important to the teacher and student. In contrast, the noise pixels in the teacher's feature map will be ignored. Our eventual feature distillation loss can be formulated as
W1 HxW M 5=1
M F(O) M & $
MacsKD
where Mr)) MM and M = a (( is generated by the teacher's learned mask tokens E and student's feature F(*), Notably, to avoid masking out critical regions by the student's s mask, we add warmup stage in the early training period, and perform mask customization after the warmup to ensure a sufficient convergence of student.
Overall loss function. Following the previous detection KD method GID (Dai et al. 2021), also adopt a distillation on the predicted bounding box regressions. Therefore, as the overall knowledge the overall loss function of student formulated as
distillation framework illustrated in Figure 2
Lstudent = Ltaek + λ MaaKKD + (Te, rs),
(10)
where r. and rs are regression predictions of teacher and student, \ and x are factors for balancing the losses. Note that we do not adopt distillation on the classification outputs, as the classification losses vary in different detectors (e.g., SoftMax CrossEntropy loss in Faster R-CNN (Ren et al.) 2 and Sigmoid Focal loss in RetinaNet (Lin et al. (017b)), we empirically find that their suitable loss weights classification KD are quite different, requiring large amount of resources in hyper- parameter tuning compared to A and λg. As a result, we remove the classification distillation for better generalizability and transferability of our method, though it could gain further improvements.
5 EXPERIMENTS
In this section, to show our superiority and generalizability, we conduct experiments two popular dense prediction tasks: object detection and semantic segmentation. Furthermore, we also experi- ment on image classification task to show that our MasKD can also achieve improvements on it.
5.1 OBJECT DETECTION
We first validate our efficacy on object detection task. We conduct experiments on MS COCO dataset (Lin et al. 2014) following previous KD works (Wang et al. 2019 Dai al.) 2021
Du
et al. 2021). and evaluate the networks with average precision (AP) on COCO val2017 set.
Network architectures. For comprehensive experiments, first follow (Wang al., 2019 Dai jet al., 2021 Du et al., (021), and adopt multiple detection frameworks for our baseline settings, including two-stage detector FaSter-RCNN (Ren
2015), one-stage detector RetinaNet (Lin ct al. 2017b), and anchor-free detector FCOS (Tian et al. 2019b). We take ResNet-101 (R101) (He
al. 2016) backbone as the teacher network, with ResNet-50 (R50) as the student. Follow (Zhang & Ma 2020 Yang et al., 2021) methods, we conduct experiments on stronger teacher detectors, including two-stage detector Cascade Mask RCNN (Cai & Vasconcelos 2018), one-stage detector RetinaNet (Lin et al. 2017b), and anchor-free detector RepPoints (Yang et al. 2019), with stronger
backbone ResNeXt101 (X101) (Xie et al. 2017).
Training strategies. We train our mask tokens for 2000 iterations using an Adam optimizer with 0.001 weight decay, and a cosine learning rate decay is adopted with an initial value of 0.01. In the distillation stage, we follow the standard 2 schedule
detection as previous works (Wang et al., 2019 Dai et al. |002 Du et al. 2021: Yang et al. 2021). For the loss weights, we simply set X and A2 to I in Eq. 10) Faster RCNN-R50 student, and the weights other architecture variants adjusted to keep a similar amount of loss values as Faster R50N-R50. Detailed training strategies can be found in Appendix A.5

---

Published as a conference paper at ICLR 2023
Table Object detection performance with baseline settings on COCO val set. T: teacher. St student. t: inheriting strategy (Yang et al. 2021) adopted. References for the methods can found in Appendix A.I
Table 2: Object detection performance with stronger teachers on COCO val set. CM RCNN: Cascade Mask RCNN. t: inheriting strategy adopted. References for the methods can found Appendix A.I
Method
AP AP AP7o APs APM API.
Method
AP APso APTs APş APM APr.
Two-stage detectors
Two-stage detectors
T: Faster RCNN-RIOI
39.8 60.1 43.3 22.5 43.6 52.8
T: CM RCNN-XIOI
45.6 64.1 49.7 26.2 49.6 60.0
S: Faster RCNN-R50
38.4 59.0 42.0 21.5
42.1 50.3
S: Faster RCNN-R50
38.4 59.0 42.0 21.5 42.1 50.3
Fitnet
38.9 59.5 42.4
21.9
42.2
51.6
LED
38.7 59.0 42.1 22.0
41.9
51.0
GID
40.2 60.7
43.8
22.7
44.0
53.2
FGFI
39.1 59.8 42.8
22.2
42.9
51.1
FRS
39.5 60.1
43.3
22.3
43.6
51.7
COFD
38.9 60.1
42.6
21.8
42.7
50.7
FGD
40.4
22.8
44.5
53.5
FKD
41.5 62.2
45.1
23.5
45.0
55.3
MasKD
40.8 60.7 44.4 23.2
44.6
53.6
FGD
42.0
23.7
46.4
55.5
MasKD
41.0 60.8 44.6 23.5
45.0
53.9
MasKD
42.4 62.9 46.8 24.2
46.7
55.9
One-stage detectors
MasKD
42.7 63.1 47.0 24.5
47.4
56.2
T: RetinaNct-RIIn
38.9 58.0 41.5 21.0 42.8 52.4
One-stage detectors
S: RetinaNet-R5t
37.4 56.7 39.6 20.0
40.7 49.7
T: RetinaNet-X
41.2 62.1 45.1 24.0 45.5
53.5
Fitnet
37.4 57.1 40.0
20.8
40.8
50.9
S: RetinaNet-R50
37.4 56.7 39.6
20.0
40.7
49.7
GID
39.1 59.0
42.3
22.8
43.1
52.3
COFD
37.8 58.3 41.1 21.6
41.2
48.3
FRS
39.3 58.8
42.0
21.5
43.3
52.6
FKD
39.6 58.8
42.1
22.7
43.3
52.5
FGD
39.6
22.9
43.7
53.6
FRS
40.1 59.5
42.5
21.9
43.7
54.3
MasKD
39.8 59.0 42.5 21.5
43.9
54.0
FGD
40.4
23.4
44.7
54.1
MasKD
39.9 59.0 42.5 23.3
43.9
54.4
MasKD
40.9 60.1 43.6 22.8
45.3
55.1
Anchor-free detectors
MasKI)
41.0 60.2 43.8 22.6
45.3
55.3
T: FCOS-R101
40.8 60.0 44.0 24.2 44.3
52.4
Anchor-free detectors
S: FCOS-R50
38.5 57.7 41.0 21.9
42.8
48.6
T: RepPoints-XII
44.2 65.5 47.8 26.2 48.4 58.5
Fitnet
39.9 58.6 43.1
23.1
43.4
52.2
S: RepPoints-R50
38.6 59.6 41.6 22.5
42.2 50.4
GID
42.0 60.4
45.5
25.6
45.8
54.2
FKD
40.6 61.7
43.8
23.4
44.6 53.0
FRS
40.9 60.3
43.6
25.7
45.2
51.2
FGD
41.3
24.5
45.2 54.0
FGD
42.1
27.0
46.0
54.6
MasKD
41.8 62.6
45.1
24.2
45.4 55.2
MasKI
42.6 61.2 46.3
26.5
46.9
54.2
MasKDt
42.5 63.4 45.8 24.9 46.1 56.8
MasKD
42.9 61.5 46.5 27.2 46.8 54.9
Distillation settings. We conduct feature distillation the predicted feature maps, and train the student with our MasKD) loss LasKD, regression KD loss, and task loss, as formulated in Eq. The number of tokens set to 6 in all experiments.
Experimental results, Our results compared with previous methods are summarized in Table and Table 2 In Table we first compare KD methods with baseline settings, where the teacher and student are the same ResNet (He et al. 2016) variants. Our MasKD can significantly surpass the other of-te-o-thee- art methods. For example, the RetinaNet with ResNet-50 backbone gets 2.5 AP improvement on COCO. We further investigate our efficacy on stronger teachers whose backbones are replaced by stronger ResNeXts. As Table [ student detectors achieve more enhancement on both AP and AR with our MasKD, especially when with Cascade Mask RCNN-X101 teacher, MasKD gains a significant improvement of 3.8 AP over the Faster RCNN-R50.
Our MasKD outperforms existing state-of-the-art methods consistently all the popular model set- tings. Note that our method more general to dense prediction tasks without any ground-truth prior manual heuristics on specific detection frameworks, showing that our mask tokens effectively locate the important Pols that benefit the distillation.
5.2 SEMANTIC SEGMENTATION
Learning masks in semantic segmentation task straightforward as it needs to predict fine- ane-ggained pixel-wise semantic classifications. We conduct experiments Cityscapes dotaset[Cordts et al.
7

---

Published as a conference paper at ICLR 2023
Table 3: Semantic segmentation results on Cityscapes dataset. trained from scratch. Other models are pretrained on ImageNet. FLOPs is measured based on an input size of 1024 x 2048.
References for the methods can be found in Appendix A.I
Params
FLOPs
mloU (%)
Params
FLOPs
mloU (%)
Method
Method
(M)
(G)
Val
Test
(M)
(G)
Val
Test
T: DeepLabV3-R101
61.1
2371.7
78.07
77.46
T: DeepLabV3-R101
61.1
2371.7
78.07
77.46
DeepLabV3-R18
74.21
73.45
DeepLabV3-MBV2
73.12
72.36
SKD
75.42
74.06
SKD
73.82
73.02
IFVD
75.59
74.26
IFVD
73.50
72.58
13.6
572.0
128.9
CWD
75.55
74.07
CWD
74.66
73.25
CIRKD
76.38
75.05
CIRKD
75.42
74.03
MasKD
77.00
75.59
MasKD
75.26
74.23
S: DeepLabV3-RI8'
65.17
65.47
S: PSPNet-R18
72.55
72.29
SKD
67.08
66.71
SKD
73.29
72.95
IFVD
65.96
65.78
IFVD
73.71
72.83
13.6
572.0
12.9
507.4
CWD
67.74
67.35
CWD
74.36
73.57
CIRKD
68.18
68.22
CIRKD
74.73
74.05
MasKD
73.95
73.74
MasKD
75.34
74.61
2016) following previous KD methods (Wang et al. 2020; Shu et al. 2021 Yang et al. 2022), and
evaluate the networks with mean Intersection-over Union (mloU) on Cityscapes val and test sets.
Network architectures. For all experiments, we use DeepLab (Chen et al. 2018) framework with ResNet-101 (R (He et al., 2016) backbone the teacher network. While for the students, we use various frameworks (Deepl V3 and PSPNet (Zhaa et al. 2017)) and backhones (ResNet-18 and MobileNetV2 (Sandler et al., 2018) to valid the effectiveness our method.
Training strategies. Following CIRKD Yang et al. 2022), we adopt standard data augmentation, which consists random flipping, random scaling in the range of [0.5, 2]. and a crop size of 512 x 1024. We train the models using SGD optimizer with momentum of 0.9, and polynomial annealing learning rate scheduler is adopted with an initial value of 0.02. We train the mask tokens for 2000 iterations in the mask learning stage, and then train the student for 40000 iterations.
Distillation settings. We conduct feature distillation on the predicted segmentation maps, and train the student with MasKD loss LMasKD and task loss. Note that use the distillation loss in (2 and do not involve mask weighting and mask customization as in detection, since all the regions in segmentation are equally important. The loss weight of LMasKD is 0.5, and the number of tokens is set to 8 for all experiments.
Experimental results. Our results compared with previous methods are summarized Table 3 By simply distilling different mask regions separately, our MasKD significantly outperforms state- of-the-art methods. especially when the student randomly initialized without pretraining on ImageNet. For example, MasKD achieves 73.74% mloU on Cityscapes test set with trained- from-scratch DeepLabV3-RIe student, while the previous state-of-the-art method CIRKD obtains 68.22%. This indicates that the semantic regions can help students learn better on semantic segmen- tation tasks. Besides, we also visualize the learned masks on Cityscapes in Appendix A.7] which shows that mask tokens generate promising segmentation results compared to the ground-truth labels.
5.3 ABLATION STUDY
Effeets of components in MasKD, We perform experiments to show the effects each proposed component in MasKD in Table 4 Feature distillation with random regions. Compared to the mimic baseline, distillation with mundomly-initializee mask tokens can also gain improvements, as it still captures weak segmentations on the feature (see Figure (a)). + mask divergence loss. As shown in (b) the masks learned without task loss can also obtain fairly good segmentations, but lack task knowledge (e.g., it mixtures the background and foreground pixels into mask). The
8

---

Published as a conference paper at ICLR 2023
result is even 0.4 worse than the random tokens, indicating that the task-related masks are important in MasKD. + task loss. Using task loss can learn better meaningful masks to the task, and thus gains further improvement of 0.6 compared to the random tokens. + adaptive mask loss weights. Adaptive mask loss weights provide a better balance on the distillation regions, thus having an additional 0.2 gain. mask customization. We adopt the student's feature customize the masks generated with the teacher's 's feature, and the results show that it can obtain further 0.2 increment. Our MasKD with all components can achieve significant 1.5 (41.4 vs. 39.9) improvement over the mimic baseline.
Table 4: Ablation of components in MasKD. We train the student Faster RCNN-R50 with teacher Cascade Mask RCNN-XIO1 using 1x schedule, and only adopts distillation on feature. The AP of mimic baseline is 39.9.
Mask learning
Feature distillation
AP
Laiv
(8) task
mask weighting mask custom.
X
X
40.4
V
X
X
X
40.0
X
V
X
X
40.6
(a)
(b)
d
41.0
Figure 4: (a) Masks with
d
41.2
tokens. (b) Masks with tokens learned with only
/
V
/
41.4
Ldiv.
Using ground-truth semantic masks on semantic seg- mentation. The semantic segmentation task provides accurate fine-grained segmentations in ground-truth an- notations, which can be regarded as the masks in our masked distillation. We conduct experiments to com- pare the ground-truth semantic masks with our learned masks. Concretely, with 19 classes and 1 ignore class in Cityscapes dataset, we generate 20 masks, where the points with the corresponding class in the mask set to 1, and others are set to O. We train the student with these generated masks and learned masks separately using (2) As shown in Table 55) using ground-truth masks also obtain very high performance, which outper- forms existing KD methods, yet our MasKD achieves even higher performance. One possible reason that our learned masks contain soft probabilities on each point and thus have more information to teach the student.
Table 5: Efffec of using ground-truth
masks in semantic segmentation. We DeepLabV334 work, and train students with R18 back- bone, then report the evaluation results on val set.
teacher net-
Mask
DeepLabV3
PSPNet
gt. learned
76.71
75.26
77.00
75.34
CONCLUSION
Feature distillation matters for dense prediction tasks as the feature contains both recognition and localization information. In this paper, unlike previous object detection methods that usually per- form masked distillation on bounding boxes, we present a new mask generation method by locating the pixels of interests using learnable receptive tokens. Our method enjoys finer pixel-level fea- ture reconstruction and better generalizability without bounding box priors. Besides, our additional adaptations of receptive tokens in distillation loss improve the performance by making the student focus more on those important pixels. Extensive experiments on object detection and semantic seg- mentation tasks validate efficacy.
Acknowledgements. This work was supported in part by the Australian Research Council under Project DP210101859 and the University of Sydney Research Accelerator (SOAR) Prize.

---

Published as a conference paper at ICLR 2023
REFERENCES
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of IEEE conference on computer vision and pattern recognition, pp. 6154-6162,
2018.
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. Advances in neural information processing systems, 30, 2017.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: arXiv:190.007/3:
2019.
Liang-Chiee Chig-- Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Xing Dai, Zeren Jiang, Zhao Wu, Yiping Bao, Zhicheng Wang, Si Liu, and Erjin Zhou. General in- stance distillation for object detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7842-7851, 2021.
Shangchen Du, Shan You, Xiaojie Li, Jianlong Wu, Fei Wang. Chen Qian, and Changshui Zhang. Agree to disagree: Adaptive ensemble knowledge distillation in gradient space. advances in neural information processing systems, 33: 18:12255-12355 2020.
Z. Du, R. Zhang, M. Chang, X. Zhang, S. Liu, T. Chen, and Y. Chen. Distilling object detectors with feature richness. In 35th Conference Neural Information Processing Systems, 2021.
Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Dis- tilling object detectors via decoupled features. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Tong He, Chunhua Shen, Zhi Tian, Dong Gong, Changming Sun, and Youliang Yan. Knowledge adaptation for efficient semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 578-587 2019.
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1921-1930, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503 150.:00333,0,,,, 2015.
Tao Huang, Lang Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Lightvit: Towards light convolution-fie vision transformers. arXiv preprint arXiv:2207.055577 2022a.
Tao Huang, Zekang Li. Hua Lu, Yong Shan, Shusheng Yang. Yang Feng, Fei Wang, Shan You, and Chang Xu. Relational surrogate loss learning. International Conference on Learning Representations, 2022b.
Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from stronger teacher. arXiv preprint arXiv: 10536, 2022c.
10

---

Published as a conference paper at ICLR 2023
Quanquan Li, Shengying Jin, and Junjie Yan. Mimicking very efficient network for object detection. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 1356-63644 2017.
Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pp. 510-519, 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 2117-2125, 2017a.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017b.
Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2604-2613, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chin Ching-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510 2018.
Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen. Channel-wise knowledge distillation for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5311-5320, 2021.
Xiu Su, Tao Huang, Yanxi Li, Shan You, Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. Prioritized architecture sampling with monto-carlo tree search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10968-10977, 2021.
Ruoyu Sun, Fuhui Tang, Xiaopeng Zhang, Hongkai Xiong, and Qi Tian. Distilling object detectors with task adaptive regularization. arXiv preprint arXiv:2006. 10100006.3100 13108, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Inter- national Conference on Learning Representations, 2019a
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEF/CVF international conference on computer vision, pp- 9627-9636, 2019b.
Tao Wang, Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine-grained feature imitation. In Proceedings of the IEEF/CVE Conference on Computer Vision and Patlern Recognition, pp.
2019.
Yukang Wang, Wei Zhou, Tao Jiang, Xiang Bai, and Yongchao Xu. Intra-class feature variation distillation for semantic segmentation. In European Conference on Computer Vision, pp. 346- 362. Springer, 2020.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
11

---

Published as a conference paper at ICLR 2023
Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang. Cross-image relational knowledge distillation for semantic segmentation. arXiv preprint arXiv:2204.06986, 2022.
Ze Yang. Shaohui Liu, Han Hu, Liwei Wang, and Stephen Lin. Reppoints: Point set representation for object detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9657-9666, 2019.
Zhendong Yang. Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan. Focal and global knowledge distillation for detectors. arXiv preprint arXiv:2111.11837, 2021.
Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1285-1294, 2017.
Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, and Changshui Zhang, Greedynas: Towards fast one-shot nas with greedy supernet. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1999-2008, 2020.
Linfeng Zhang and Kaisheng Ma. Improve object detection with feature- knowledge distilla- tion: Towards accurate and efficient detectors. In International Conference on Learning Repre- sentations, 2020.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 2881-2890, 2017.
A APPENDIX
A.1 REFERENCES OF COMPARED KD METHODS
Object detection. Fitnet Romero et al. 2014), GID (Dai et al. 2021), FRS (Du al. 2021
FGD Yang et al. 2021). LED (Chen et al. 2017). FGFI (Wang al. 2019). COFD (Heo et al.
2019 FKD (Zhagg & Ma 2020).
Semantic segmentation. SKD al. 2019). IFVD (Wang et al.
2020). CWD Shu et al. 202
CIRKD Yang et al. 2022).
A.2 ARCHITECTURE MASK WEIGHTING MODULE
As shown in Figure 3 (a), the mask weighting module (yellow box) first takes distillation feature F(1) input, then conducts sequential of x 3 convolution and average pooling to get the feature of the whole image, then a 1 x convolution is adopted to predict the importance scores of mask tokens.
A.3 DISTILLATION RESULTS ON IMAGE CLASSIFICATION
In previous experiments, we show that MasKD has obvious effects on dense prediction tasks (object detection and semantic segmentation). As our method could be applied arbitrary tasks, we further conduct experiments to validate efficacy image classification task.
Following CRD (Tian et al. 2019a) and DIST (Huang et al., 2022c), we train ResNet-18 with ResNet-34 teacher on ImageNet, and adopt distillations with our MasKD and the baseline mimic, respectively. We perform feature distillations on the outputs of the last four stages of the networks. As the results shown in Table 6 mimicking the backbone features could obtain similar performance compared to the KD (Hinton et al. (2015) baseline, while our MasKD can improve mimic by 0.55%, showing that our masked distillation can also benefit the feature distillation classification task.
12

---

Published as a conference paper at ICLR 2023
Table 6: Classification performance on ImageNet. Teacher: ResNet-34. Student: ResNet-18.
KD
CRD
DIST
Teacher
Student
Mimic
MasKD
Hinton et
2015
Tian et al. (019a)
Huang et 2022c
73.31
69.76
70.66
71.17
72.07
70.71
71.26
A.4 MORE ABLATION STUDIES
Ablation on the number of tokens. We conduct experiments on COCO dataset investigate the effects of different numbers of mask tokens in MasKD. As shown in Table 7 with only 2 tokens, MasKD improves mimic by 1.4% AP, while more tokens could achieve further improvements. In this paper, choose moderate number of 6 for better performance--fficiene trade-off.
Table 7: Ablation on the number of tokens in MasKD. We train the student Faster RCNN-R50 with teacher Mask RCNN-X10/ using x schedule.
0 (mimic)
2
10
14
39.9
41.3
41.4
41.4
41.5
A.5 DETAILED TRAINING SETTINGS ON OBJECT DETECTION
We conduct experiments on object detection using COCO dataset (Lin al., 2014). and evaluate our performance on COCO val2017 set. In Table I and Table 2 we report our main results on various detection frameworks such as Faster RCNN (Ren et al. 2015), RetinaNer (Lin et al.] 20/7b),
FCOS (Tian et al. 2079b), and RepPoints (Yang el al.) 2079). All the models are trained with the official strategies 2x schedule in MMDetection (Chen et al. 2019).
Loss weights: We set MasKD loss weight Å₁ 1 and regression loss weight X2 1 Faster RCNN students. For other detection frameworks, we simply adjust the loss weight \ of LMASKD to keep a similar amount of loss value Faster RCNN. Concretely, the loss weights X on RetinaNet, FCOS, and RepPoints are 5, 10, and 10, respectively.
A.6 VISUALIZATION OF LEARNED MASKS ON COCO
We visualize the learned complete masks of Faster RCNN-RIO/ teacher in Figure 5 We can see that (1) The earlier stages in FPN prefers to recognize small objects, while the stage 4 is used to detect large objects.
(2) The effective region for representing an object is usually smaller than the object, which indicates that we may not need to reconstruct all the pixels in the bounding box in distillation, and many of them are useless background information. (3) The mask tokens are associated with different objects, while which shows that the background features are also helpful for detection. (4) Some objects that are not marked in the ground- truth annotations features are also informative and useful to the student, this could be a superiority our MasKD compared to previous methods with ground-truth priors.
mask token is learned to cover all the remaining background pixels,
also be included in our masks, and these foreground
A.7 VISUALIZATION LEARNED MASKS CITYSCAPES
We visualize the learned masks of DeepLabV3-RR01 teacher on Cityscapes dataset in Figure 6 We can see that, the learned masks on semantic segmentation task is much clear than those on object detection, and the edges of objects can be precisely masked. This is because the semantic segmentation task provides a direct supervision on the segmentations, thus the features are more
discriminative on various semantic types.
13

---

Published as a conference paper at ICLR 2023
Annotations
Mask
Misk
Maok?
Maok
Mask
Stage
Stage
Stage 2
Stage 3
Stage 6
Stage 0
Stage
Stage 2
Stage
Stage 4
Stage D
Stago
Stage
Stage
Stago
Stage 0
Stage 1
Stage 2
Stage
Stago
Figure 5: Complete visualization of learned masks on COCO dataset. Zoom up to view better.
14

---

Published as a conference paper at ICLR 2023
Annotations
Mask
Mask 2
Mask
Mask
Mask
Mask 7
Mask 8
Figure 6: Complete visualization of learned masks Cityscapes dataset. Zoom up to view
better.
15


Reviews:
[{"review_id": "pdeC8C5swi", "reviewer": "Reviewer_EXB7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about model distillation for dense prediction tasks, specifically for object detection and semantic segmentation. Prior works on this topic identify per-region distillation as an important ingredient to a successful knowledge transfer. This paper proposes a novel way to identify what regions should be distilled and with what importance (weight). These regions are modeled as masks, can be trained based on the respective task loss, and are not biased by the form of ground truth used (bounding boxes or segmentation maps). Given a teacher network, embedding vectors (one per region, the number of regions being a hyper-parameter) are learned that create a per-pixel similarity map with the feature maps, and then a (soft) masking. These embeddings (along with a corresponding weighting) are learned such that (a) the sum of all masked features can minimize the task loss, and (b) the masks are diverse. In a second phase, these masks are used for knowledge distillation into a student model. Additionally, the student also builds a mask (from the learned embeddings and the feature maps) and the distillation then becomes dependent on the agreement between teacher and student masks. Experiments on both object detection and semantic segmentation demonstrate consistent improvement over existing methods on multiple base models and backbones.", "summary_of_the_review": "In summary, I think the experimental setting needs to improve to justify and solidify the improvement over prior works. I lean towards rejection of the submission.", "strength_and_weaknesses": "Strengths:\n- The proposed method for generating masks is task-agnostic in a sense that no priors about the output space (bounding boxes or segmentation maps) are used.\n- It is good to see multiple detection and segmentation architectures and backbones evaluated.\n- The method is simple and appears to be consistently effective across model designs\n\nWeaknesses:\n- I found the motivation for identifying (un)important regions/pixels in a task-agnostic manner unsatisfactory. Most of the motivation builds on foreground vs. background objects. But this is only valid for object detection, not so much for semantic segmentation. The better motivation seems to be a \"balanced loss/impact\" for different regions when doing distillation.\n- I'm do not fully understand why it is \"optimal\" that only the pixels where both teacher and student masks agree should be distilled. This seems to be a guess (or a random justification) for something that works practically (i.e., the experiments show improvements).\n- Overall, I found the experimental setting to be insufficient.\n  - I'm missing more recent detectors like CenterNet [A] or DETR-based models like DINO [B-D].\n  - Choosing R101 and R50 as teacher and student, respectively, seems odd to me. The performance difference is not big between these two backbones (Table 1). Isn't the more practical situation for model distillation to have a very strong model distilling information into a much smaller one? I see such experiments for segmentation (with MobileNet backbones), but shouldn't this be the default for most experiments?\n  - Evaluations are only done on two benchmarks (COCO and CityScapes), although the method is advertised as general and task-agnostic. Given the rather small improvements over prior works across all experiments, I think additional evaluations on other datasets are needed to confirm consistent improvements of the proposed method over existing ones.\n- Significance of the improvement over prior works\n  - The improvement over FGD is consistent but minor in all results of Tables 1 and 2. This relates to my comment above about the experimental setting. With such consistent but only minor improvements, I find it necessary to provide more experimental evidence in the form of (a) additional datasets, and (b) reporting mean errors with standard deviation over multiple runs with different random seeds.\n  - I see similarly small margins in Table 4, where the proposed method outperforms randomly initialized tokens by only 0.6% AP. Again, are these improvements statistically significant?\n  - Why is the improvement much bigger for models trained from scratch (w/o ImageNet pre-training)? Is there any intuition for that? Also, are the teacher models also trained from scratch? If no, is this a practical situation where the teacher model can use ImageNet-pretrained weights, but the student cannot?\n- Table 1: Is there an explanation why the student model outperforms the teacher? The motivation and intuition of model distillation is to mimic the teacher's behavior. How does the model become better than the student?\n\nReferences:\n- [A] Objects as Points. Zhou et al. arXiv 1904.07850\n- [B] DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection. Zhang et al. arXiv 2203.03605\n- [C] Dn-detr: Accelerate detr training by introducing query denoising. Feng et al. CVPR 2022\n- [D] DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR. Liu et al. ICLR 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MQ3JsJajEoH", "reviewer": "Reviewer_q4W5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a knowledge distillation method for dense prediction tasks such as object detection and semantic segmentation. The method proposes to learn important regions (masks) adaptively for each image with task loss and mask diversity loss. Then the learned mask tokens, as well as mask weighting module, are adopted to distill teacher features to student. Results on COCO and Cityscapes datasets show the effectiveness of the proposed method.\n", "summary_of_the_review": "I don't find major issues with this work. Please see my comments in strengths and weakness.", "strength_and_weaknesses": "Strengths:\n1. The idea is novel and interesting. The mask tokens learned by minimizing the task loss can generate finer and task-related distillation regions, which are demonstrated through the visualizations and experiments in the paper.\n\n2. The proposed method is generic to various tasks, as it does not require ground-truth labels to select distillation regions as previous methods.\n\n3. The experiments on detection and segmentation tasks show that, MasKD achieves significant improvements compared to previous state-of-the-art methods. Sufficient ablation studies are provided to show the efficacy of the method.\n\nWeaknesses:\n1. The performance on ImageNet dataset is not state-of-the-art.\n2. How long does it take to learn the mask tokens for the teacher?\n3. Why the authors learn the mask tokens for only 2000 iterations? How about training it for more iterations?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "I6-j7n-h5w", "reviewer": "Reviewer_mLef", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a new distillation method that is more fine-grained than the previous bounding box distillation methods. It achieves performance improvement in both object detection and semantic segmentation.\n", "summary_of_the_review": "Though I still have some concerns, I think this paper is novel and well-written. Therefore, I would lean toward acceptance. ", "strength_and_weaknesses": "Pros:\n\n1. Figure 2 clearly illustrates the framework of the proposed method.\n\n2. This paper is overall well-written and easy to follow.\n\n3. This paper is novel. I have never seen similar works before.\n\nCons:\n\n1. For equation (7), how the weights of each mask are computed? What is the loss? Why this would not collapse to trivial solutions like emphasizing the masks that are easy to learn?\n\n2. In Tables 1 and 2, the improvement over FGD seems marginal (around 0.2-0.4), which may come from randomness.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster