Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation
Authors: Jiawei Fan
Chao Li
Xiaolong Liu
Meina Song
Anbang Yao
Venue: NeurIPS 2023 poster (2023)
Keywords: Knowledge distillation
semantic segmentation
contrastive learning
TL;DR: This paper proposes augmentation-free dense contrastive distillation (Af-DCD), a new knowledge distillation method towards efficient semantic segmentation.

Abstract: In recent years, knowledge distillation methods based on contrastive learning have achieved promising results on image classification and object detection tasks. However, in this line of research, we note that less attention is paid to semantic segmentation. Existing methods heavily rely on data augmentation and memory buffer, which entail high computational resource demands when applying them to handle semantic segmentation that requires to preserve high-resolution feature maps for making dense pixel-wise predictions. In order to address this problem, we present Augmentation-free Dense Contrastive Knowledge Distillation (Af-DCD), a new contrastive distillation learning paradigm to train compact and accurate deep neural networks for semantic segmentation applications. Af-DCD leverages a masked feature mimicking strategy, and formulates a novel contrastive learning loss via taking advantage of tactful feature partitions across both channel and spatial dimensions, allowing to effectively transfer dense and structured local knowledge learnt by the teacher model to a target student model while maintaining training efficiency. Extensive experiments on five mainstream benchmarks with various teacher-student network pairs demonstrate the effectiveness of our approach. For instance, DeepLabV3-Res18|DeepLabV3-MBV2 model trained by Af-DCD reaches 77.03\%|76.38\% mIOU on Cityscapes dataset when choosing DeepLabV3-Res101 as the teacher, setting new performance records. Besides that, Af-DCD achieves an absolute mIOU improvement of 3.26\%|3.04\%|2.75\%|2.30\%|1.42\% compared with individually trained counterpart on Cityscapes|Pascal VOC|Camvid|ADE20K|COCO-Stuff-164K. Code is available at https://github.com/OSVAI/Af-DCD.

Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation
Jiawei Fan" Intel Labs China jiawei. fan@intel.com
Chao Li
Intel Labs China chao3, www.iiineee.....
Xiaolong Liu HoloMatic Technology Co. Ltd. Liuxiaoloonghholommaiccccoo com
Meina Song BUPT
Anbang Yao
Intel Labs China anbang yao@intel.com com
mnsongSbupt.c www.geebppeeeuu..... edu. cn
Abstract
In recent years, knowledge distillation methods based contrastive learning have achieved promising results on image classification and object detection tasks. However, in this line of research, we note that less attention is paid semantic segmentation. Existing methods heavily rely on data augmentation and memory buffer, which entail high computational resource demands when applying them to handle semantic segmentation that requires to preserve high- resolution feature maps for making dense pixel-wise predictions. order address this problem, we present Augmentation-fiee Dense Contrastive Knowledge Distillation (Af-DCD), a new contrastive distillation learning paradigm train compact and accurate deep neural networks for semantic segmentation applica- tions. Af-DCD leverages a masked feature mimicking strategy, and formulates a novel contrastive learning loss via taking advantage of tactful feature partitions across both channel and spatial dimensions, allowing to effectively transfer dense and structured local knowledge learnt by the teacher model to target student model while maintaining training efficiency. Extensive experiments on five main- stream benchmarks with various teacher-studen network pairs demonstrate the effectiveness of our approach. For instance, the Deepl Lab V- 1eReeee11@ee..aall. 8iDeepLab MBV2 model trained by Af-DCD reaches 77. 03%/76.338% mIOU Cityscapes dataset when choosing DeepLabV3-Reste mance records. Besides that, Af-DCD achieves an absolute mIOU improvement of 226%/33.04%/2..54/43 (10%%11442% .42% compared with individually trained counterpart Cityscapes/Pascal DOCICamiidIAADE2OKCCCOO... Sou---ttf-------K Code is available at hhttps://githut http:///ggtttbccoooo////////--/-
as the teacher, setting new perfor-
Introduction
In computer vision, semantic segmentation is a mainstream dense prediction task aiming to assign the corresponding class to every pixel of input images. Although fundamental models in this task have achieved remarkable progress 3 the heavy computational burden and latency of these models severely prohibit their deployment bottleneck, Knowledge Distillation (KD) 4 aims to transfer the knowledge from large models to light-weight ones. Considering KD in supervised semantic segmentation, teachers usually have
resource-constrained devices. As popular solution this
"This work was done when the first two authors were interns at Intel Labs China, supervised by Anbang Yao who proposed the original idea and led the project. Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Herd
Mosk
Teacher Network (Patannd vut frod
Lya
Generator
lad
Hend
Leask
F*
Figure 1: The overall framework of Af-DCD, which contains two major parts: (i) masked feature reconstruction; (ii) augmentation-Fie contrastive distillation loss. Detail illustrations on Af-DCD are shown in Figure 2
stronger ability of capturing detailed local information of images where the dense and structured knowledge contributes significantly to segment complicated areas, such as boundary and occlusion. Therefore, distillation methods tailored to this task should keep the integrity of dense and structured knowledge, rather than compressing it excessively such as simply using global representations or salient areas. As result, directly applying traditional KD methods in semantic segmentation, like vanilla KD or feature imitation (5), cannot achieve promising results in most cases.
Thanks to the development of contrastive learning, some distillation methods, like SSKD (6), CRD 77 and G-DetKD [8] employed this paradigm into their designs, which promoted the performance significantly, as they exploited teacher's intrinsic knowledge in the consistencies among different augmentations or various same-category instances. semantic segmentation, CIRKD [ designed an implicit contrastive method which aimed to guarantee pixel-pixel and pixel-region consisten- cies between teacher and student by involving additional contrastive samples. On one side, these existing methods have achieved remarkable progress in image-level or object-level contrastive mim- icking. On the other side, however, the efficacy of the aforementioned methods heavily relies on the augmentation and memory buffer, which entail heavy computational and memory cost. Besides, previous contrastive distillation methods have not explicitly modeled relations among pixel-wise or more fine-graine tine-graiedd representations within each local patch to transfer teacher's 's dense and structured knowledge to student, which is critical in semantic segmentation. Thus, when designing a contrastive distillation method for this task, two vital technical problems should be tackled: (i) High resource demands: No matter leveraging augmented samples or storing feature maps in memory buffer, more computational or storage cost is required. For one, the forward of augmented samples incurs extra computational cost. For another, the output feature maps for semantic segmentation are in high resolution. If we want make dense and structured contrasting, these feature maps should be originally preserved, which occupy a large amount of memories; (ii) Structured knowledge transfer: Although CIRKD defined pixel-wise alignment between student and teacher, it employed contrastive pixel-wise representations from other images, rather than local areas within the same image. It means that no explicit contrasting was defined between student-specific pixel-wise representation each teacher's 's pixel-wise representation within local areas, which was agnostic to the structure in teacher's s feature. In brief. a contrastive distillation method specially tailored semantic segmentation, which also entails no extra high resource demands (data augmentation and memory buffer), is essential.
Driven by achieving this target, first look into the aforementioned two problems and surprisingly discover both of them are incurred from the simple inheritance in the basic definitions of traditional contrastive learning [10. TTI. Therefore, our Augmentation--fee Dense Contrastive Knowledge Distillation ( (Af-DCD) for supervised semantic segmentation tasks can come out, by re-defining these basic concepts: (i) Contrasting samples are not coarse- grained representations of images or objects. Instead, in our AF-DCD, we move further on pixel-level representations and divide each of them into several disjoint partitions (fine-graie (fie-graiied representations), which are treated as our contrasting samples; (ii) Positive and negative pairs are not conditioned on categories or objects. Instead, based on the definition of contrasting samples, we define our pairs in specific teacher-studed student feature pair (F". FF), where fine- ine-grained representations having the same absolute positions in feature maps are used to formulate positive pairs, while representations having different absolute positions but within neighbourhoods are used as negative pairs. Based on these definitions, our Af-DCD can naturally tackle those two problems discussed above. To the first problem, unlike previous methods which had to introduce data augmentation and memory buffer to construct sufficient positive and negative pairs,
coarse-grained

---

our Af-DCD can directly construct these pairs in each local patch without any extra computational and memory cost. To the second problem, we approach it by introducing two new contrastive designs. Spatial Contrasting and Channel Contrasting, which enable student to capture teacher's local dense and structured knowledge by focusing on contextual and positional channel-group information, respectively. As a natural progression, we introduce hybrid design called Omni-Contrasting which combines Spatial Contrasting and Channel Contrasting in neat manner. This design facilitates the simultaneous transfer of both types of information from the teacher to the student, enabling effective augmentation-fiee contrastive knowledge distillation. Despite leveraging rather dense contrasting, Af-DCD also performs efficiently. This is due to the patch separation technique use, which significantly reduces the computational complexity. Additionally, the distance measuring and contrasting caleulation can be carried out in parallel, further enhancing the overall efficiency of our method.
Experimental results demonstrate that: (i) Af-DCI exhibits superior performance compared to state-of-the-art methods, on various benchmarks with different teacher-student network pairs; (ii) Af-DCD exhibits even more significant improvements larger datasets, such ADE20K, indicating can enhance student's generalization capability. Moreover, our analytical experiments illustrate that by effectively learning teacher's self-similarity distribution within neighbourhoods and thus reducing the fine-gre grained feature distances to the teacher, Af-DCD benefits addressing difficult scenarios in semantic segmentation.
Related Works
Knowledge Distillation. Knowledge distillation can be generally divided into probability- approach and feature-based approach. Specifically, the former one forces student to mimic teacher's logits as soft-labels 12). while the later one leverages teacher's 's hidden feature maps or its variants [3 as distillation supervisions. Some recent feature-basse approaches [13][7] designed their distillation methods based on masked image modeling mechanism [18] 19 and achieved promising performance in various tasks, the feature reconstruction can enhance the interdependencies among pixels, which is beneficial to feature distillation.
Knowledge Distillation in Semantic Segmentation. As semantic segmentation is a dense prediction task, the methods specially designed for this task aimed to capture teacher's s structured local infor- mation. In order to achieve this target, SKD [20] directly measured the similarities of pixel-wise representations between teacher and student, while He et al. [2]] leveraged non-local operation with autoencoder to encode local information. CWD [22] evaluated channel-wise pixel distribution contributing to learn teacher's 's spatial information in each individual channel. Some methods further explored intrinsic knowledge among different samples. For instance, IFVD 23 measured distances from prototypes of different classes and forced student to mimic teacher's intra-class relations, and CIRKD g forced student keep pixel-wise and region-wise consistencies to teacher among various samples in memory buffer.
Contrastive Knowledge Distillation. Inspired by the development of contrastive learning in self- supervised 10 24] and supervised tasks [II, some recent works leveraged this paradigm and designed new contrastive distillation methods, which made contrasting between teacher's and student's features by employing different views generated from data augmentation (6), or using various samples' features 25 7 well gradients (26) stored in memory buffer. Specifically in dense prediction tasks, DeDetKD 8 constructed ROI feature pairs and executed soft semantic-gun -uideedgg matching which promoted the performance in object detection, and CIRKD 9 designed an implicit contrastive method which leveraged both pixel and region representations to learn structured information in spatial dimension.
Method
3.1 Overall Framework
Figure Iillustrates our distillation process semantic segmentation from a pre-trained teacher T to a
student S which needs to be trained on a specific dataset. The output features of T and S are denoted
as F E R xxXxC* and 1 x respectively, where H and W are height and width of
3

---

F
R
FilM
Faus
Figure 2: Detailed illustrations on three different types of Af-DCD, which are Spatial Contrasting (top left), Channel Contrasting (top right) and Omni- ConitCoonn (bottom). For brevity, the contrasting process is illustrated merely using specific contrastive sample in student feature maps, denoted as Fi Pi.j, p.A. in three Af-DCD designs, respectively. The red arrows denote constructing positive pairs, while the blue arrows denote constructing negative pairs. The gray blocks denote other patches which are not considered in calculating the loss in this patch.
feature maps, C and C are the number of channels of teacher and student, respectively. Our method uses random mask M E R #x to overlap F in spatial dimension and then inputs the masked feature into generator, which can be formulated as F fgemerator [M @ f*). Finally, we can get F #xxxx''' which has the same dimension as teacher's feature. In order to fit general definition, teacher's feature is also processed by a transform F fr (F*). In our design, fi is an identity transformation, thus F = Ft. After these operations, the reconstructed student feature F has been projected into the same feature space as teacher's feature Ft. The feature imitation loss Lşa and our Af-DCD loss LAf-Det are calculated based on feature pair (Ft. F*). Besides, following the settings in 22] we also employ vanilla logits- logits-basek based KD loss 4 In general, the overall loss of our method can be defined as:
I Ttak X Iikd + >2 + AgLAA-DDD.
(1)
where X; is the balancing weight.
3.2 Augmentation-free Dense Contrastive Knowledge Distillation
After illustrating the overall framework, we further explain our core design in detail, which contains two major parts, Lid and LAI-DOD- As we mentioned above, both of them are based the same feature pair (F', F*). Therefore, in order to clarify our design, we first answer two key questions: (i) What are the targets of Lfa and Laj L DCD separately? (ii) How these two losses cooperate with each other? To the first question, Lsa directly help student to imitate teacher's global and salient features, while LAf-DCD designed to mimic teacher's 's dense structured knowledge within local areas, which can further force F to approach Ft in the micro and fine-grained views. To the second question, in terms of mimicking dense and structured knowledge, these two losses are inextricably bound up with each other. For one, I.fa provides A/--CD an absolute imitation trend, which cannot be implemented in contrastive loss. For another, LAf-DCD offers Lfa essential constrains on encouraging this process towards mimicking teacher's 's structured knowledge in each local patch, making student further approach teacher.
Masked Reconstruction based Feature Imitation. The feature imitation loss L is a basic part of our design, which forces student feature F" imitate teacher feature Fr directly. Previous methods commonly leverage simple upsampling to align teacher's 's and student's feature dimensions. Recently, some methods leveraged masked image reconstruction [5 [7], which is stronger baseline, as can promote the performance by strengthening the interdependencies among pixels. This process can be formulated as:
F Igenerator (M o F*),
(2)

---

where M E IR "X is the mask matrix generated randomly with the mask ratio Ç E [0, 1). Then feature distillation can be formulated as:
W Lyd = DC U. Fig..)
C
H
(3)
Although masked reconstruction achieves promising performance, one potential weakness may affect its gain in semantic segmentation: Reconstructed features tend be similar in local areas However, in semantic segmentation, local differences indicate structured knowledge. in which student's reconstructed feature maps should keep these differences. Here comes oul why we choose masked reconstruction as our basic feature distillation method: (i) choosing a stronger baseline to further test the effectiveness of our Af-DCD; (ii) examining whether our Af-DCD can promote masked reconstruction for generating more structured representations, enhancing feature imitation process.
3
Motivation and Key Ideas of Af-DCD. In semantic segmentation, notice two important facts. On one side, different pixels within local areas may contain different semantic information, as they may belong different categories different parts of an object. We call this contextual information. On the other side, differences among channel groups of a pixel representation implicitly indicate the semantic meaning of this pixel, as each output channel maps. We call this as positional channel-groug information. In short, the contextual and positional grrupp information within each local area in spatial and channel dimensions is of significance in semantic segmentation. If both types of information from teacher's 's feature maps can be densely utilized as the distillation guidance facilitate the training of the student model, we conjecture that student's ss performance and generalization capability can be greatly boosted. However, traditional feature imitation loss La difficult to capture these two types of information, as they are not directly observable and measurable comparing to salient representation. Aiming to tackle this problem, we define contrastive loss across student's and teacher's pixel-level or more fine ain-ggrinedd representations to explicitly model such kind of knowledge transfer. Specifically, as shown in Figure 2 Af-DCD incorporates three key ideas:
specific projection of all input feature
(1) Spatial Contrasting, With the first phenomenon, we leverage pixel-wise representations in F and F and define pixel-wise dense contrasting based on spatial positions, aiming to transfer teacher's spatial contextual information student;
(2) Channel Contrasting, With the second phenomenon, we then move further on pixel-wise dense contrasting and split every pixel-specific representation into disjoint groups and define group-wise dense contrasting based on channel positions, aiming to force student to learn teacher's positional channel-group information;
(3) Omni-Contrasting Progressively, we unify the above two contrasting methods in an hybrid design, which takes advantage of tactful feature partitions across both channel and spatial dimensions in local areas by splitting the feature map into disjoint patches of the same size.
Formulation of AF-DCD. The formulation of Af-DCD follows the notations in Section 3.1. The projected feature maps of teacher and student are denoted as F E #x''' and F E xxxx" respectively. In the perspective of contrastive learning, we first clarify some important concepts in our Af-DCD:
(1) Samples and Views. Different from existing contrastive distillation methods in classification and detection, the samples in Af-DCD are pixel-level or more fine-grained representations in feature maps of the same image, which are naturally dense. Furthermore, we define that representations of teacher and student are two views which should be aligned by our contrastive loss;
(2) Positive Pairs and Negative Pairs. Under the definitions above, positive pair can be defined as: teacher-student sample pair (F;, FJ) which has the same index i j. where i and j are general location indices. Similarly, a negative pair can be defined as: a teacher- student sample pair (P7 Ff) whose indices are different j.
'Experimental results illustrate that the distances between feature partitions within local areas are usually small (see in Figure
5

---

p.i.j. F F...j.e)) ). while negative pairs are (F.....ke Ft. pw for any u, v. w # i, k, where the can Based on the above definitions, the loss for specific sample F. E F can be formulated as: exp(-d(FF) F()/T) 2
LAI-DCD(Fi,F) -log
cxp(-
where denotes the temperature score, N is the total number of contrastive pairs and Lişj is an indicator function which is iff. * j. Different from traditional contrastive loss that adopts cosine distance measuring similarities, where d(F ET) = F - 1- Our ablative experiments illustrate that improved performance would be attained when select the same type of the distance function for L and LAf-DOD.
use Euclidean distance measure sample pair similarities,
Spatial Contrasting. shown Figure 2 (top left), aiming to learn teacher's spatial contextual information, Spatial Contrasting constructs dense contrasting operations between teacher's and student's pixel-level representations, where for each sample Fi.. the positive pair is (Fi Piy))
while the other { pair and HW 1 negative pairs. Then Spatial Contrasting can be formulated as:
are all negative pairs. In such condition, we obtain postive
exp(-d(FF) F//)/T)
1 A/-DCD F*) -log
(5)
Σ
Σ Lwtii
FU,))/T)
Channel Contrasting, As shown in Figure (top right), Channel Contrasting first splits each pixel representation into M non-overlapping channel groups of the same length. Specifically, F?, and Fr, are split into (f and order to transfer teacher's ' channel-group information at each pixel position, the dense contrasting happens on each pixel-Ievel representation pair between [F ( w==I... and { F } For each fine-gra -rainedied representation Fi.j the positive pair is (F +.j.k F....) and negative pairs are {(F A.j.A Ft. (. bwykk where the number of positive and negative pairs is 1 and M - 1, respectively. where w denotes the index. Channel contrasting can Then we substitute these pairs into Formula 4 be formulated as:
where ue is the index of channel group. Then, in
ICC Af DCD (F ii Fi.)) -log exp(-d(E) d(F i.j.k F F/T)
Σ 1utk exp( (pp(-d(Fig.)) FTT
Omni- Contrasting. Ommi-CCntrastiin tial Contrasting, shown in Figure (bottom). Different from the design of Spatial Contrasting, Omni- Moni-Contrastinn does not contrast positions of feature maps. Instead, in order to exploit local information, Omni- ContiiConrasttngg groups pixels into series of local patches and leverages spatial and channel contrasting within each local patch. This local contrasting can force student to learn teacher's spatial contextual information and positional channel-group information simultaneously, which are beneficial for accurately segmenting the boundary and correctly classify those pixels. After that, F and F are split into {F.) p=....... and {Fp)p=1...N. where F F² E < and N HW Then, like the operation in Channel Contrasting, the channels will be divided into M non-overlapping groups. Teacher's and student's 's representations at position (i, k) in local patch p are denoted as
an neat combination of Channel Contrasting and Spa-
HW
P.i.j.k 1x1x1 37 For Fe p.ij..> the positive pair Ft p.i.j.k and Fa p.i... respectively, where Ft. p.i.j.k: Fe
number of positive and negative pairs is 1 and HWM respectively. The be defined as:
exp( exp(-d(F) A Ft F/T
1CC AF-DDD Paiki Ft)) -log
A
W
M 1 La exp(
(sp(- exp(-d(Peagst P.igk Ft. )/T)
w=1
Finally, the overall Omni-Con Coni-Contrasting loss can be defined as:
N
A
OC I. LA-DDD
ΣΣΣΣ
09 Si-DCD(F Ft).
(8)
HWM

---

Table 1: Performance comparison of Af-DCD and recent state-of-ther th- distillation methods on Cityscapes, evaluated with different teacher-studen segmentation network pairs. & denotes distillation without Ltd- We follow the way in CIRKD to calculate FI.OPs. Best results are bolded.
(a) The same framework with different backbones
(b) Different frameworks with the same backbone
mIOU (%)
Method
(M)
Mothod
Params (M)
FLOPs (G)
Val mIOU (%)
Val
Test
61.IM
2371.76
78.07
77.46
PSPNet-Ros101
68.07M
1868.5G
78.34
Deeplabb3-Resth
74.21
Deept
73,20
SKD
74.06
75.59
74.26
75.93
CWD
13.6M
572.0G
75.55
74.07
MGD+ 15
13.6M
76.02
CIRKD
76.38
75.05
76.31
MasKD 17
77.00
75.59
AF-DCD
77.03
75.12
DeeplabVV-MBBV
73.12
72.36
PSPNet-Ras18
69.85
SKD 20
73.82
73.02
SKD
20
72,70
IFVD 231
73.50
72.58
CWD
73.53
CWD 22
3.2M
128.90
74.66
73.25
MGD 15
12.9M
73.63
CIRKD
75.42
74.03
MGD 13
74.10
MasKD 1+
75.26
74.23
ACDDCD
73.92
Af-DCD
76.38
75.06
ACDDC
74.22
4 Experiments
4.1 Datasets and Experimental Setups.
Datasets. Five popular semantic segmentation datasets, including Cityscapes 27 Pascal VOC 28 Camvid (29). ADE20K (30) and COCO-Stuff- 164K [31] are used in our experiments. We conduct our ablation studies on both Cityscapes and ADE20K, which help us ensure the best setting in most experiments and analyse the effectiveness of our design. Details for these five datasets are described in supplemental material.
Experimental Settings. Following general settings 9 20 15 in semantic segmentation distilla- tion, we adopt V3 [32 and PSPNet 3 for segmentation framework, ResNet-18 [33] and Mobilenetv2 34 for student backbones, Resnet- 101 for teacher backbone and group various teacher- student pairs. order to make fair comparison with different state-of-ther art methods, we implement our method on both MMSegmentation codebase (55 and CIRKD codebase (9. In training and evaluation, we use mean Intersection-over- In training phase, all models are optimized by SGD with the momentum of 0.9, the initial learning rate of 0.02, and the batch size of 16. The input size 512 x 1024, 400 x 400, 512 x 1024, 512 x 1024, for experiments on Pascal VOC, Cam ADE20K and COCO-Stuff- 164K, respectively. The input size for experiments on Cityscapes are different the two codebase, 512 x 1024 in CIRKD codebase and 512 512 in MMSegmentation codebase [13. In evaluation phase, follow general settings in [22). which evaluate the performance with the original image size. Our masked reconstruction generator consists of two 3 x 3 convolutional layers with ReLU, following [15]. Other defauls hyper-parameter settings and implementation details are described in supplemental material.
(mloU) to measure the performance of all methods.
4.2 Main Results
In this part, we intend to compare the distillation performance of our method Af-DCD with recent state-of-the-art methods for semantic segmentation. Aiming for comprehensive comparison, conduct experiments on the aforementioned five public datasets following general settings.
Results on Cityscapes. In Table I we conduct experiments on the most popular Cityscapes dataset to validate the generalization ability of our method to different teacher-student network pairs. The experimental results show that Af-DCD outperforms state-of-the the maximal margin of 1.12%. In average, Af-DCD brings 3.44% gain to the baseline student models, with the maximal gain of 4.37%. From the results shown in Table la we can see that our method can well handle teacher-student network pairs in which students (e.g., Deepl V and DeepLab V3-MLBB have the same segmentation framework but with different backbones. The
methods in most cases, with

---

Table Performance comparison of Af-DCD and recent state-of-the the-ttt art distillation methods on the other four datasets. We follow the way in CIRKD 9
to calculate FLOPs. Best results are bolded.
Pascal VOC
(b) Camvid
Method
Params (M) FLOPs (G)
Val mlOU (8)
Method
Params (M)
FLOPs (G)
Test m1OU (%)
DeepLabb3-Rsss101
1294.6G
DeepLabV3
61.
280.2G
84
Doepl JhV33RRss8
Deeplabb3
66.92
SKD
67.46
IPVD
67.28
13.6M
305.0G
CWD 2M
67.71
CIRKD 9
74.50
CIRKD V
68.21
AC-DCD
76.25
AF-DCD
69.27
PSPNor-Res18
73.33
S: PSPNet-Raa Ros118
66.73
SKD 20
74.07
SKD R0
67.65 83
IVD 25
73.54
IPVD 28
12.9M
45.6G
67.61
CWD 22
73.99
CWD 2M
67.92
CIRKD 0
74.78
CIRKI
Af-DCD
76.14
(c) ADE20K
(d) LOCO--ttuf-1644
Method
Params (M)
FLOPs (G)
Val mIOU (%)
Method
Params (M)
FLOPs (G)
Val m1OU (%)
DeepLabV3-
1294.6G
42.70
DeepLabVV3 Restol
61.IM
1294.6G
38.71
DeppLabh3-RRs11
DeepLabV3-Rei1
32.60
CIRKD y
13.6M 305.0G
CIRKD
1366
305.0G
33.11
AF-DCD
36.21
AF-DCD
34,02
results of Table Tb further show that method can also generalize well teacher-student network pairs in which students (e.g., Deepl Lab V-- 18 and PSPNet-Resl 8) have different segmentation frameworks but with the same backbone.
Results on Four Other Datasets. In Table we evaluate the performance of Af-DCD on four other datasets including PASCAL VOC, Camvid, ADE20K and generalization our design to handle different semantic segmentation tasks. According to the results shown in Table 2a 2d Af-DCD outperforms state-of-the- methods by significant margins these four datasets, with the maximal and average margin of 1.75% and 1. 1.22%. respectively. Furthermore, can observe that method consistently shows significant absolute mIOU gains (from 1.42% 3.04%) to different student models on small-size (Camvid), medium-size (Cityscapes and PASCAL VOC) and large-size (ADE20K and COCO-Stuff-1ed datasets.
to examine the
4.3 Ablation Studies
Ablation Study on Different Loss Terms. In our formulation, the overall loss contains three distillation terms, including Led, Lyd and Laf LAA-DCD. Accordingly, we conduct experiments that independently test the gain from each term explore the nature of Af-DCD. From the results shown in Table we can get following observations: (i) The accuracy gain of Baseline + L DCDD is slight on relatively small dataset Cityscapes (0.28% mIOU gain), but it is notably pronounced on much larger dataset ADE20K (1.50% mIOU gain); (ii) Compared Baseline + Lyd. Baseline + Lf DCD gets student model with better accuracy ADE20K dataset, while maintaining almost the same training efficiency; (iii) Baseline + Lfa + LOC - gets student models with 76.44% mIOU and 36.01% % mIOU on Cityscapes dataset and ADE:20K dataset, respectively, which are obviously better than both Baseline + Lyd and Baseline + L L showing that two loss terms L and LOC LAD-DCD are complementary, (vi) Lid can further bring minor extra gains, 0.08% and 0.2%, to Baseline + Lja Af DCD on Cityscapes and ADE20K dataset, respectively.
OC
Ablation Study on Different Designs. In this study, we conduct a set of experiments on our three basic contrasting designs, namely Channel Contrasting (CC), Spatial Contrasting (SC) and Omni-Contrasting (OC), in order to exploit the gains from different contrasting dimensions and further verify the necessity of our OC. In Table 3a and 3b
observe two phenomena:

---

Table 3: Ablation studies on loss terms and their different combinations. The experiments for Cityscapes and ADE20K are conducted on the first teacher-student network pair in Table 1b and 2c respectively. The training time measured on NVIDIA RTX A5000 GPUs with 40000 iterations.
(a) Ablation on Cityscapes
(b) Ablation on ADE20K
Mechod
(8)
Firam (h)
(1EU
Tirets (h)
Baselène
33.91
75,88
+7,g
34.02
-PAY-
73.48
+0.25
4.06
+1.77
35.41
+1.50
4.35
+Iye+ TA
VAI-D50
76.44
+3.34
4.25
+1ye
PAI-DOD
36.01
+2.10
4.48
35:22
+lpp Lug 6OP
76.52
+2y4 Les 1.00
36.21
73.20)
Bowline
33.91
IVA
+dge L22-DCD
76.25
13.0
4.13
+
35.72
+1.81
4.41
+lyee AL-000
76.26
+1.e D02
35:23
+Lge L1L Eaf
76.35
+3.13
4.29
+4y4
35.51
+1.90
4.54
+Tyc+ 1.7
76.44
+3.34
4.25
+1.8
120
Ay-ppp
36.01
+2,10
4.48
16.
16.
145
MAAEE
105
15.
2
060
442
110
050
0.27
200
262
202
2.3
6.5
(a) Ablation on T
(b) Ablation on X (c) Ablation on M (d) Ablation on N (c) Ablation on
Figure 3: Ablation studies major hyper-parameters. In (d), 4 denotes N 4 with x 4 max
pooling, which can further enhance training efficiency. Best viewed with zoom in.
(i) CC has similar gain to baseline model as SC on Cityscapes, while has much stronger per- formance on ADE20K, exceeding 0.5% to SC; (ii) The combination of CC and SC performs better than CC and SC, but worse than OC on both datasets (in distillation accuracy and train- ing speed). The above two observations show the superiority of OC, which comes from the neat combination of CC and SC. Specifically, OC groups pixels into number of disjoint local patches and tactfully leverages CC and SC within each local patch instead of the holistic feature maps to better exploit dense and structured local information for contrastive feature mimicking.
Table 4: Ablation study the choice of the function d in the Omni-Contrasting loss. The experimental setups the same to Table (default setting without Lua)). Best results bolded.
A blation Study on Choice of the Function d in the Omni-Contrastimg Loss. We compare Formula of our method with 3 types of the function d including L2- normed distance (our choice), cosine similarity (com- mon choice in contrastive loss) and L1 -normed distance. From the results summarized in Table 4 we can see that method always shows significant mIOU gains to the baseline with all types of the function d. Com- paratively, our method with Ln-normed distance is the best, which supports our intuition that improved perfor- mance would be attained by choosing the same type of the function d for the feature distillation loss (Formula
Fonction in Formula
Duinse
mIOU
AmlOU
73.20
n/s
Ll J.-nemed distance
CityScape
75.97
+2.77
Cosine similazity
76.10
+2.00
1/2 normed distance
76.44
+3,24
Baseline
n/s
Crsine simileity
Ll nermeme distunce
ADE20K
35.82
35.95
+2.04
+1.91
12 normed distance
36.01
+2.10
3) and the Omni-Contres Contrasting loss (Formula 7
Ablation Study on Training Efficiency. Our Af-DCD naturally augmentation-free and breferr-uffee--ree.. We evaluate the efficiency of our design in terms training time observe
Table 5: Comparison of training re- sources used by CIRKD and AF-DCD, under the settings of the first teacher- student network pair in Table la Best results are bolded.
and GPU memory occupation. In Table BaI that Af-DCD introduces minor extra training cost, from 4.02 hours to 4.25 hours, which only increases 5.7% training time to feature distillation. From the results shown in Table we see Af-DCD uses less memory and training time, but achieves better performance, compared to CIRKD.
(96)
GPU memory (G) Terase
CIRKD
76.38
10.09
4.34
AE-DCD
77.03
7.94
4.23
Ablation Studies on Major Hyper-paramett we also perform experiments to study them. As shown in Figure 3 the effectiveness our method is relative stable when values of these parameparameters are changing. Notably, we find using max
Recall that our method has five typer-parametent

---

(a) Feature distance between and S
(b) Feature distance within local areas
Object Boundary Object Occlation
Small
Image Boundary
Diflicult Calegory
(c) Heat maps of areas where model using Af-DCD segments correctly but using FD segments wrong
Figure 4: Verification experiments. We leverage the models in Table 3b and count IM fine-grained representations from 2000 samples. In (a), randomly select 10K fine- ane-ggrined representations and measure the distances between student and teacher. In (b), we measure the distances among fine-grained representations within 4 x 4 local areas of specific sample, and randomly select 10K to calculate the probability density. In (c), we choose several difficult cases and highlight areas that LAf-DCD D help fa segment correctly.
pooling with proper scale has no obvious effect on model performance but it has better efficiency than larger N. Other detailed analysis referred to supplementary materials.
4.4 Discussion
After demonstrating the superior performance of Af-DCD in Section 4.2 and analysing the gain of Af-DCD in Section 4.3] we perform some verification experiments, aiming to discuss how LAf-DCD boosts the distillation performance of Lttas + La (denoted as FD for abbreviation).
As shown in Figure the self-similarity distribution of fine-grained representations projected by FD has large differences from that of teacher's feature representations. In order to alleviate the aforementioned problem, Laf-no DDD contrasts feature partitions within each local patch between DCD student and teacher. The results in Figure 4a and Figure are summarized below:
LAS-DCD dramatically decreases the distances of student teacher in views of fine-gained representations and self-similarities, which proves our basic assumption that Laf- LAf-DCD DCD can further make student approach teacher in the micro and fine-grainee views;
(2) LAI-DCD increases both the mean and variance of the self- self-similarity distribution of student. Such observation verifies that LAf-DCD can force student to learn dense and structured knowledge implicitly contained among teacher's feature partitions.
The above two improvements are proved to be effective in addressing segmenting various difficult scenarios in semantic segmentation, such as object boundary, small object, object occlusion, difficult category and rare view. Heat maps shown in Figure 4c illustrate that LAf-DCD FD correctly classify these difficult pixels, which further verifies the superiority of Af-DCD.
effectively help
More examples and detailed analysis are referred to supplementary material.
Conclusion
In this paper, we present Af-DCD, an augmentation-free dense contrasive distillation method tailored to semantic segmentation. The dense contrasting in Af-DCD is omni umni-dimensional design. Thus, can effectively transfer teacher's contextual and positional channel-group information student. Experimental results show that Af-DCD is effective on different teacher-studen network pairs and datasets while maintaining training efficiency, as it born with augmentation- -free and memory-buffer free. We hope our work can inspire researchers to explore more powerful and efficient contrastive distillation methods in the future.
10

---

References
[1] Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2017.
[2] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1006.05887, 2017.
[3] Hengshuang Zhao, Jianping Shi. Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. CVPR, 2017.
[4] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 10X3::1503.02531, 2015.
[5] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
[6] Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self-supervision. In ECCV, 2020.
[7] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020.
[8] Lewei Yao, Renjie Pi, Hang Xu, Wei Zhang, Zhenguo Li, and Tong Zhang. G-detkd: towards general distillation framework for object detectors via contrastive and semantic-guided feature imitation. In ICCV, 2021.
[9] Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang. Cross- image relational knowledge distillation for semantic segmentation. In CVPR, 2022.
[10] Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In JCML, 2020.
[11] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurlPS, 2020.
[12] Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. arXiv preprint arXiv:2102.0065
2021.
[13] Xiaolong Liu, Lujun Li, Chao Li, and Anbang Yao. Norm: Knowledge distillation via n-to-one representation matching. In ICLR, 2023.
[14] Nikos Komodakis and Sergey Zagoruyko. Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.
[15] Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked generative distillation. In ECCV, 2022.
[16] Zhendong Yang, Zhe Li, Ailing Zeng, Zexian Li, Chun Yuan, and Yu Li. Vitkd: Practical guidelines for vit feature knowledge distillation. arXiv preprint 2rXiv:2209.02432. 2022.
[17] Tao Huang, Yuan Zhang, Shan You, Fei Wang, Chen Qian, Jian Cao, and Chang Xu. Masked distillation with receptive tokens. In ICLR, 2023.
[18] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[19] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, 2022.
[20] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In CVPR, 2019.
11

---

[21] Tong He, Chunhua Shen, Zhi Tian, Dong Gong, Changming Sun, and Youliang Yan. Knowledge adaptation for efficient semantic segmentation. In CVPR, 2019.
[22] Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen. Channel-wise knowledge distillation for dense prediction. In ICCV, 2021.
[23] Yukang Wang, Wei Zhou, Tao Jiang, Xiang Bai, and Yongchao Xu. Intra-class feature variation distillation for semantic segmentation. In ECCV, 2020.
[24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick, Momentum contrast for unsupervised visual representation learning. CVPR, 2020.
[25] Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation. arXiv preprint mrXiv221010473.1 2021.
[26] Jinguo Zhu, Shixiang Tang, Dapeng Chen, Shijie Yu, Yakun Liu, Mingzhe Rong. Aijun Yang, and Xiaohua Wang. Complementary relation contrastive distillation. In CVPR, 2021.
[27] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[28]
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010.
[29] Gabriel Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recognition using structure from motion point clouds. In ECCV, 2008.
[30] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019.
[31] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: context. In CVPR, 2018
Thing and stuff classes in
[32] Liang-Chien Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[34] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chier Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
[35] MMSegmentation Contributors. MMSegmentation Openmmlab semantic segmentation toolbox and benchmark. httpp://giitee http:ppnwwmmmm/////mmmmraaaaaaaa 2020.
12


Reviews:
[{"review_id": "96bckb7lvX", "reviewer": "Reviewer_YQfC", "summary": "This paper proposes a novel knowledge distillation methods for semantic segmentation, called Augmentation-Free Dense Contrastive Knowledge Distillation(Af-DCD). Af-DCD is a new attempt on the usage of contrastive learning in the task of knowledge distillation for semantic segmentation, which alleviate the problem of high computational resource brought by data augmentation and memory buffer. Af-DCD utilizes feature partitions across both channel and spatial dimensions, allowing to effectively transfer dense and structured local knowledge learnt by the teacher model to a target student model while maintaining training efficiency. Experimental results on mainstream benchmarks including demonstrate  the effectiveness of the proposed Af-DCD.", "questions": "1. Is Omni-Contrasting necessary? The combination of Channel Contrasting and Spatial Contrasting in Table 4(a) is needed to demonstrate the superiority of Omni-Contrasting.\n2. Please add some ablation studies on the choice of the function d in formula 7, which can not only help to screen the appropriate function but also increase interpretability.\n", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.The experiments are sufficient. A lot of experiments and visual analysis have proved the effectiveness and superior performance of the proposed Af-DCD.\n2. The design of Af-DCD is clever and makes use of the structural information of teachers from the aspects of both channel and space.\n3. The overall experiment is solid and the code is available, which is nice.\n", "weaknesses": "1. The organization of reference is poor. Reference is not added to specific methods in the table. And there is no reference for MaskKD in the whole paper, which actually refers to [17], which is confusing. The instruction for CKD in section2 actually is the instruction for CWD.\n2. Some ablation studies are missing. For example, the lack of combination of Channel Contrasting and Spatial Contrasting in Table 4(a), the choice of function d in formula 7.\n3. No distillation experiment of transformer-based structure has been carried out, and it is explained in the appendix that the transformer-based structure gains little improvement from Af-DCD, which limits the generality of Af-DCD.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive review, and the recognition of our work. In what follows, we provide detailed responses to address your concerns one by one:\n\n**1. The first weakness about the organization of reference**.\n\n**Our responses**: **(1)** We agree with you, and will add the references to specific methods in Table 1, Table 2 and other related Tables; **(2)** You are correct, MasKD in Table 1 refers to [17] published in ICLR 2023. MasKD uses a set of learnable embeddings to localize the pixels of interest and generate the distillation masks; **(3)** Yes, in Section 2, [22] should refer to CWD but not SKD which should refer to [20]. We are sorry for these typos in reference; **(4)** Following your careful comments, we will fix confusing/inaccurate references and make a careful job on proofreading to improve the presentation of our final paper.\n\n**2. The second weakness and the first question about the lack of ablation study** “The combination of Channel Contrasting and Spatial Contrasting in Table 4(a)”.\n\n**Our responses**: **(1)** Following your insightful comments, we perform ablative experiments on Cityscapes and ADE20K datasets using the same experimental setups to Table 4. Detailed results are shown in the below two Tables. It can be seen that the combination of Channel Contrasting (CC) and Spatial Contrasting (SC) performs better than CC and SC, but worse than Omni-Contrasting (OC), on both datasets; **(2)** Note that **the concepts of Augmentation-free CC and SC are two basic contributions of our work**. Under this context, the direct combination of them can be viewed as our Vanilla OC. Comparatively, the proposed OC is smarter and neater, which groups pixels into a number of disjoint local patches and tactfully leverages CC and SC within each local patch instead of the holistic feature maps to better exploit dense and structured local information for contrastive feature mimicking, showing the superiority both in distillation accuracy and training speed; **(3)** Now, it is clear that **Omni-Contrasting is indeed necessary**.\n\n|Method (on Cityscapes)|mIOU (%)|$\\Delta$mIOU(%)|$T_{train}(h)$\n|:--|:--:|:--:|:--:|\nBaseline|73.20|n/a|n/a\n$+L_{fd}+L_{Af-DCD}^{CC}$|76.23|+3.03|4.13\n$+L_{fd}+L_{Af-DCD}^{SC}$|76.26|+3.06|4.18\n$+L_{fd}+L_{Af-DCD}^{CC}+L_{Af-DCD}^{SC}$|76.33|+3.13|4.29\n$+L_{fd}+L_{Af-DCD}^{OC}$|76.44|+3.24|4.25\n\n|Method (on ADE20K)|mIOU (%)|$\\Delta$mIOU(%)|$T_{train}(h)$\n|:--|:--:|:--:|:--:|\nBaseline|33.91|n/a|n/a\n$+L_{fd}+L_{Af-DCD}^{CC}$|35.72|+1.81|4.41\n$+L_{fd}+L_{Af-DCD}^{SC}$|35.22|+1.31|4.45\n$+L_{fd}+L_{Af-DCD}^{CC}+L_{Af-DCD}^{SC}$|35.81|+1.90|4.54\n$+L_{fd}+L_{Af-DCD}^{OC}$|36.01|+2.10|4.48\n\n**3. The second weakness and the second question about the lack of ablation study** “the choice of the function $d$ in formula 7”.\n\n**Our responses**: Following your insightful comments, we perform ablative experiments on Cityscapes and ADE20K datasets using the same experimental setups to Table 4. Specifically, we compare formula 7 of our method with 3 types of the function $d$ including $L2$-normed distance (our choice), cosine similarity (common choice in contrastive learning research) and $L1$-normed distance. Detailed results are summarized in the below two Tables. It can be seen that **(a)** Our method always shows significant mIOU gains to the baseline with all 3 types of the function $d$; **(b)** Comparatively, our method with $L2$-normed distance is the best, which supports our intuition that improved performance would be attained by choosing the same type of the function $d$ for the feature distillation loss (formula 3) and the omni-contrasting loss (formula 7) conditioned on the same source features.\n\nFunction $d$ in Formula 7 (on CityScapes)|mIOU(%)|$\\Delta$mIOU(%)|\n|:--|:--:|:--:|\nBaseline|73.20|n/a\n$L1$-normed distance|75.97|+2.77\nCosine similarity|76.10|+2.90\n$L2$-normed distance|76.44|+3.24\n\nFunction $d$ in Formula 7 (on ADE20K)|mIOU(%)|$\\Delta$mIOU(%)|\n|:--|:--:|:--:|\nBaseline|33.91|n/a\n$L1$-normed distance|35.82|+1.91\nCosine similarity|35.95|+2.04\n$L2$-normed distance|36.01|+2.10\n\n**4. The third weakness about the limited generality of our method to transformer-based structures**.\n\n**Our responses**: **(1)** The current design of Af-DCD cannot easily generalize to transformer-based structures, as we discussed in the last Section “Limitations of Af-DCD” of the Appendix. **The main reason** is: Af-DCD exploits dense pixel-wise information within each of local patches via the feature partition across both channel and spatial dimensions for formulating contrastive feature mimicking conditioned on the single image input, but transformer-based structures built upon self-attention modules primarily encode global patch-to-patch feature dependencies, which appear to be in conflict with each other; **(2)** Actually, we had a distillation experiment to explore this. In the experiment, we applied Af-DCD to SegFormer (MiT-B4 encoder as teacher and MiT-B0 encoder as student) [1], a seminal transformer-based structure for semantic segmentation. The below table shows the results, where Af-DCD only brings 0.31% gain to the baseline; **(3)** A potential direction to address the above issue is how to preserve local information and make a good alignment of Af-DCD to transformed-based structures. Please allow us to leave it as future research.\n\nMethod (on Cityscapes)|mIOU(%)|$\\Delta$mIOU(%)\n|:--|:--:|:--:|\nTeacher: SegFormer-MiT-B4|81.23|n/a\nStudent (baseline): SegFormer-MiT-B0|75.58|n/a\nAf-DCD|75.89|+0.31\n\n[1] Enze Xie, et al. “SegFormer: Simple and efficient design for semantic segmentation with transformers”, NeurIPS 2021.\n\n**Finally**, during the rebuttal phase, we also conducted more experiments to improve ablation studies and added discussions to improve the clarifications of our method. You are referred to our top-level responses titled **“Author Rebuttal by Authors”**, and our responses to the other reviewers for details.\n\nLooking forward to your feedback."}, {"review_id": "3DzEHtnIUc", "reviewer": "Reviewer_cEUs", "summary": "This paper points out that existing knowledge distillation methods have been heavily relying on tdata augmentation and memory buffer, which require high computational resources and this is further amplified when it comes to segmentation task that requires relatively higher resolutions of feature maps for processing. To alleviate this complexity, the method called Af-DCD is proposed, which aims to tackle segmentation task by leveraging knowledge distillation based on a novel contrastive learning. More specifically, this method first leverages masked feature mimicking strategy and proposes a novel contrastive learning loss. Experimental results confirm that the proposed method is effective.", "questions": "See weaknesses above.", "limitations": "Limitations are propoerly addressed in Section E.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper is easy to read and understand.\n\n2. The proposed method achieves the best performance against competitors.\n\n3. Numerous discussions and ablations are presented to validate the choices.", "weaknesses": "1. In Table 1 and 2, it seems like ours refers to cumulatively adding all the different methods (SKD, IFVD, CWD and etcs..). The presentation needs improvements. \n\n2. It would be better if the authors cite each methods in Table 1 and 2 (SKD, IFVD..) so that the readers do not have to look up what those abbreviations refer to. \n\n3. In section 4.2, it is only 'stated' that the proposed method performs the best. I don't find any analysis, explanations or reasonings. Moreover, Although FLOPs and Params are also included, there is no texts covering them. \n\n4. In line with section 4.2, section 4.3 and 4.4 also lack explanations or reasoning. At least it is not sufficient. These sections are simply stating what the table or figure shows without sufficient analysis or attempts to deliver insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive review, and the recognition of the novelty, the effectiveness, and the ablation studies of our work. In what follows, we provide detailed responses to address your concerns one by one:\n\n**1. The first weakness** “In Table 1...ours refers to cumulatively adding all the different methods...needs improvements”.\n\n**Our responses**: **(1)** In Table 1 and Table 2, **“Ours” actually refers to applying our proposed method Af-DCD independently, but not** cumulatively adding all the different methods (SKD, IFVD, CWD and etc.); **(2)** The symbol “**+**” in each row of Table 1 and Table 2 denotes: applying a specific method independently to the target teacher-student network pair for semantic segmentation on the given dataset; **(3)** We are sorry for this confusion due to the potential misunderstanding of the symbol “**+**”. Following your careful comments, we will remove the symbol “**+**”, clarify the meaning of “Ours”, and improve the presentation of our final paper. \n\n**2. The second weakness** “It would be better if... cite each methods in Table 1...what those abbreviations refer to”.\n\n**Our responses**: Thanks a lot for your great suggestion. Accordingly, we will add references to all methods (SKD, IFVD, CWD, CRIKD, MasKD, MGD, etc.) compared in Table 1, Table 2 and other related Tables.\n\n**3. The third weakness** “In section 4.2...I don't find any analysis... Although FLOPs and Params...no texts covering them”.\n\n**Our responses**: We really appreciate your insightful comments. **(1)** In Section “**4.2 Main Results**”, we intend to compare the distillation performance of our method Af-DCD with recent state-of-the-art methods for semantic segmentation. Aiming for a comprehensive comparison, we conduct a lot of experiments on public datasets following general settings in semantic segmentation distillation: **(i)** We first conduct experiments on the most popular Cityscapes dataset to **validate the generalization ability of our method to different types of teacher-student network pair**. From the results shown in Table 1(a), we can see that our method can well handle teacher-student network pairs in which students (e.g., DeepLabV3-Res18 and DeepLabV3-MBV2) have the same segmentation framework but with different backbones. The results of Table 1(b) further show that our method can also generalize well to teacher-student network pairs in which students (e.g., DeepLabV3-Res18 and PSPNet-Res18) have different segmentation frameworks but with the same backbone; **(ii)** Next, we conduct experiments on four other datasets including PASCAL VOC, Camvid, ADE20K and COCO-Stuff-164K to **validate the generalization ability of our method to various semantic segmentation tasks**. From the results shown in Table 2(a)-(d), we can see that our method consistently shows significant absolute mIOU gains (1.42%~3.04%) to different student models on small-size (Camvid), medium-size (Cityscapes and PASCAL VOC) and large-size (ADE20K and COCO-Stuff-164K) datasets; **(2)** The superior performance of our method to existing methods demonstrates the effectiveness of the proposed omni-contrasting distillation learning scheme which transfers dense and structured local knowledge across both channel and spatial dimensions learnt by the pre-trained teacher model to the target student model; **(3)** The basic goal of our work is to leverage a pre-trained high-capacity (large and accurate) teacher model to improve the training of a low-capacity (smaller and less accurate) student model, enabling efficient deployment of semantic segmentation models. Following CIRKD [9], FLOPs and Params are included in Table 1 and Table 2 to compare the computational cost of the teacher and student networks (e.g., at most 19.09$\\times$ Params compression and 18.40$\\times$ FLOPs compression).\n\n**4. The last weakness** “In line with section 4.2, section 4.3 … without sufficient analysis...to deliver insights”.\n\n**Our responses**: Although we provide some necessary explanations and analysis in Section “**4.3 Ablative Studies**” and “**4.4 Discussion**”, we agree with you that they are still not sufficient. Restricted by limited page length, we put some detailed analysis and explanations in the supplementary material (see Section B-D), as stated in Line 291-292 of Section 4.3 and Line 307 of Section 4.4. Here, we add more explanations to deliver main insights: **(1)** Note that our main contributions are the Augmentation-free Contrastive Losses $L_{Af-DCD}^{CC}$ across channel dimension, $L_{Af-DCD}^{SC}$ across spatial dimension and $L_{Af-DCD}^{OC}$ across both channel and spatial dimensions (a neat combination of $L_{Af-DCD}^{CC}$ and $L_{Af-DCD}^{SC}$, i.e., our core contribution), which could improve the basic feature distillation loss $L_{fd}$ in our formulation while maintaining training efficiency. Ablative results in Table 4(a)-(b) progressively validate their effectiveness by different loss combinations and datasets, and ablative results in Table 3 validate the training efficiency. Ablations in Figure 3 further study the choices of major hyper-parameters, verifying the robustness of our method; **(2)** Figure 4(a)-(b) provide statistical distributions of feature distance between teacher and student models, and heat map visualizations to validate the key insight of our design: $L_{Af-DCD}^{OC}$ can effectively encourage the student model to mimic dense and structured local knowledge learnt by the teacher model; **(3)** Following your insightful comments, we will include more texts to improve explanations and analysis for Section 4.2, 4.3 and 4.4 of our final paper.\n\n**Finally**, during the rebuttal phase, we also conducted more experiments to improve ablation studies and added discussions to improve the clarifications of our method. You are referred to our top-level responses titled **“Author Rebuttal by Authors”**, and our responses to the other reviewers for details.\n\nLooking forward to your feedback."}, {"review_id": "JvY2r2Aspe", "reviewer": "Reviewer_vSYf", "summary": "This paper focuses on knowledge distillation for semantic segmentation and introduces an augmentation-free dense contrastive loss function. The student and teacher feature maps are partitioned into patches, and both spatial and channel contrasting is performed within these local neighborhoods. For contrastive loss, positive/negative feature pairs are formed using tether and student features extracted from the same image without using any augmentations. Experiments were conducted on five segmentation datasets and the proposed approach is shown to perform better than various existing works.", "questions": "Why should the proposed approach be augmentation free? The proposed loss can be used with augmentations also.\n\nWhy euclidean distance in contrastive loss?\n\nCurrent experimental comparisons do not clearly demonstrate the effectiveness of L_{AF-DCD} in the present of all the other losses.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Paper was easy to follow.\nExperiments conducted on several datasets.", "weaknesses": "The title and introduction section emphasize \"augmentation-free\". However, the motivation/need to be \"augmentation-free\" is not clear to me. In the introduction, the authors claim using augmentations leads to high resource demand which I disagree. The proposed approach passes the same image through teacher and student networks. One could also use the proposed loss function as it is by passing original image to one network and an augmented version of the image to the other network. The computation cost will be almost same except the augmentation operation cost which is usually small compared to the whole forward/backprop cost. In fact, using appropriate augmentations may even be helpful as the model will learn robust features that are invariant to these augmentations. \n\nFor most of the pixels (other than those that are close to object boundaries), their neighborhood is surrounded by pixels of the same class, and treating them as negatives in contrastive loss is counter-intuitive to me. Ideally for semantic segmentation, we would want pixels of same class to have similar representations so that they can easily be classified to the same class.\n\nIn line 197, authors mentioned that they use euclidean distances instead of cosine similarity in contrastive loss without providing any explanation.\n\nThe main contribution of this paper is the contrastive loss function L_{AF-DCD}. All the other loss functions are from prior works. In order to show the effectiveness of this loss function, authors should compare results with and without the proposed loss when all the other loss functions are present, i.e., comparison between (L_kd + L_fd) and (L_kd + L_fd + L_afdkd). Such comparison is not provided in Table (4). \n\nTypo: It should be Table. 3 not 4(a) in line 287.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive review, and the recognition of the proposed approach and the experiments. In what follows, we provide detailed responses to address your concerns one by one:\n\n**1. The first concern about why should the motivation/proposed method be augmentation-free**.\n\n**Our responses**: Our motivation/method to be augmentation-free is mainly due to the contrastive learning formulation for supervised semantic segmentation task instead of unsupervised image classification task: **(1)** Most of existing contrastive learning methods adopt the self-supervised formulation (assigning a single binary label to each image pair) conditioned on heavy data augmentations (DAs) to learn a proper representation for a given backbone from a large amount of unlabeled images, while our method addresses semantic segmentation distillation in which classification needs to be pixel-wise; **(2)** For our task, passing the original image to one of teacher and student networks and its augmented version via e.g., crop-resize, rotation and flip to the other  network (i.e., the way used in existing methods) usually breaks the geometric feature alignment, i.e., features at the same locations are no longer positional same, which leads to conflict with the pixel-wise classification in semantic segmentation distillation. Therefore, our method passes the same image through teacher and student works; **(3)** Our method creates dense negative samples within each local patch of the same image via the partition across channel and spatial dimensions, and thus is augmentation-free and efficient (see Table 3 & 4); **(4)** Yes, our method can be also used with DAs. Actually, for experiments on PASCAL VOC, etc., DAs such as crop-resize and flip are used following common setups in semantic segmentation distillation, but teacher and student networks still share the same augmented image input.\n\n**2. The second concern about why treating the neighborhood pixels of a specific pixel as its negative samples**.\n\n**Our responses**: **(1)** At the first glance, it is indeed counter-intuitive to treat neighborhood pixels of a specific pixel as its negative samples since they usually tend to be the same class. However, in our formulation (formula 1), **we already have the feature imitation loss $L_{fd}$** which directly forces student to be the same as teacher at every pixel for each channel, taking the role to attain your mentioned ideal representation learning goal. **The role of our contrastive loss $L_{Af-DCD}$** is to promote the process of transferring dense and structured local knowledge (which can better classify difficult pixels for object/image boundary, small object, object occlusion, difficult category and rare view, as illustrated in Figure 4(c) of the main Manuscript and Figure 5 of the Appendix) learnt by teacher model to student model. That is, $L_{fd}$ and $L_{Af-DCD}$ conditioned on the same source feature pairs collaboratively work to get improved feature distillation, which is verified by the ablation studies in Table 4(a); **(2)** Existing feature visualization work [1] shows that layer-specific feature channels from a pre-trained CNN model usually have changing salient activations across neighboring locations and channels. In line with it, we also perform an ablation study to compare $L_{Af-DCD}$ and $L_{fd}$. **You are referred to our first set of responses to Reviewer wLTq for details**.\n\n[1] MD Zeiler and R Fergus, \"Visualizing and Understanding Convolutional Networks\", ECCV 2014.\n\n**3. The third concern about why using Euclidean distance in our contrastive loss**.\n\n**Our responses**: **(1)** The reason is simply due to our intuition that improved performance would be attained by choosing the same type of the function $d$ for the basic feature distillation loss $ L_{fd}$ (formula 3) and the contrastive loss $L_{Af-DCD}^{OC}$ (formula 7) conditioned on the same source features; **(2)** We perform ablative experiments to compare our contrastive loss with 3 types of the function $d$ on Cityscapes and ADE20K datasets. Results show that our method with $L2$-normed distance is the best, which supports the above intuition. **You are referred to our third set of responses to Reviewer YQfC for details**.\n\n**4. The fourth concern about the lack of an ablation study to compare $L_{kd}+L_{fd}$ and $L_{kd}+L_{fd}+L_{Af-DCD}^{OC}$**.\n\n**Our responses**: **(1)** Yes, our core contribution is the augmentation-free contrastive loss function $L_{Af-DCD}^{OC}$ across channel and spatial dimensions (a neat combination of our two basic contributions $L_{Af-DCD}^{CC}$ across channel dimension and $L_{Af-DCD}^{SC}$ across spatial dimension); **(2)** Following your insightful comments, we perform ablative experiments on Cityscapes and ADE20K datasets using the same experimental setups to Table 4. It can be seen that $L_{kd}+L_{fd}+L_{Af-DCD}^{OC}$ performs better than $L_{kd}+L_{fd}$ on both datasets meanwhile maintaining similar training efficiencies, demonstrating the effectiveness of $L_{Af-DCD}^{OC}$ in the presence of all the other losses.\n\n|Method (on Cityscapes)|mIOU (%)|$\\Delta$ mIOU(%)|$T_{train}(h)$\n|:--|:--:|:--:|:--:|\nBaseline|73.20|n/a|n/a\n$+L_{kd}+L_{fd}$|76.04|+2.84|4.05\n$+ L_{kd}+L_{fd}+L_{Af-DCD}^{OC}$|76.52|+3.32|4.27\n\n|Method (on ADE20K)|mIOU(%)|$\\Delta$ mIOU(%)|$T_{train}(h)$\n|:--|:--:|:--:|:--:|\nBaseline|33.91|n/a|n/a\n$+L_{fd}+L_{kd}$|35.22|+1.31|4.34\n$+ L_{kd}+L_{fd}+ L_{Af-DCD}^{OC}$|36.21|+2.30|4.51\n\n**Finally**, thank you for pointing out the typo in line 287. We will correct it and make a careful job on writing and proofreading to improve the presentation of our final paper. During the rebuttal phase, we also conducted more experiments to improve ablation studies and added discussions to improve the clarifications of our method. You are referred to our top-level responses titled **\"Author Rebuttal by Authors”**, and our responses to the other reviewers for details.\n\nLooking forward to your feedback."}, {"review_id": "J9p58haph9", "reviewer": "Reviewer_wLTq", "summary": "In this manuscript, the authors propose a effective knowledge distillation framework for semantic segmentation task. Specifically, in addition to traditional knowledge distillation on segmentation masks as well as feature distillation, to better align the dense feature, the authors introduce contrastive learning on both spatial dimension and channel dimension. The proposed Af-DCD loss significantly improves the performance of CNN-based segmentation network via knowledge distillation without data augmentation. ", "questions": "See the weakness section. ", "limitations": "The authors have listed the limitation in the supplementary material, i.e., the gain of Af-DCD on Transformer based architecture are not as large as on CNN based architecture. We believe this limitation can be seen as a future direction. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is clear, i.e., traditional feature distillation loss is somewhat difficult to contextual information and positional channel-group information.\n\n2. The extensive experiments demonstrate the effectiveness of the proposed method. The ablation study and discussion is abundant and valuable. \n\n3. The proposed method is easy to follow. ", "weaknesses": "1. Intuitively, the proposed Af-DCD loss can align the dense feature independently, therefore the authors could show the ablation study of baseline+L_{Af-DCD} only and compare the results with baseline+L_{fd}.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive review, and the recognition of our motivation, method and experiments. In what follows, we provide detailed responses to address your concerns one by one:\n\n**1. Your comments on the weakness about the lack of an ablation study** “Intuitively, the proposed Af-DCD loss can align the dense feature independently, therefore the authors could show the ablation study of baseline$+L_{Af-DCD}$ only and compare the results with baseline$+L_{fd}$”.\n\n**Our responses**: **(1)** Yes, our proposed omni-contrasting loss $L_{Af-DCD}^{OC}$ (abbreviated as $L_{Af-DCD}$ in your comments) can align the dense features independently since it is formulated as an effective contrastive distillation learning scheme for transferring dense and structured local knowledge across both channel and spatial dimensions learnt by the pre-trained teacher model to the target student model; **(2)** Following your insightful comments, we perform ablative experiments on Cityscapes and ADE20K datasets using the same experimental setups to Table 4 in the main manuscript. Detailed results are summarized in the below two Tables. It can be seen that **(a)** Baseline$+L_{Af-DCD}^{OC}$ brings significant accuracy gains to baseline models on both Cityscapes and ADE20K datasets; **(b)** Compared to baseline$+L_{fd}$, baseline$+L_{Af-DCD}^{OC}$ gets student models with better accuracy on both Cityscapes and ADE20K datasets, while maintaining almost the same training efficiency; **(c)** The accuracy gain of  baseline$+L_{Af-DCD}^{OC}$ is slight on relatively small dataset Cityscapes (0.10% mIOU gain), but it is notably pronounced on much larger dataset ADE20K (0.49% mIOU gain); **(d)** The ablative experimental results reported in Table 4 of the main manuscript have already shown that baseline$+L_{fd}+L_{Af-DCD}^{OC}$ gets student models with 76.44% mIOU and 36.01% mIOU on Cityscapes dataset and ADE20K dataset respectively, which are obviously better than both baseline$+L_{fd}$ and baseline$+L_{Af-DCD}^{OC}$, showing that two loss terms $L_{fd}$ and $L_{Af-DCD}^{OC}$  are complementary; **(5)** We have appended this ablation study to Table 4. The updated version of Table 4 is referred to Table 1 in the one-page PDF file attached in our top-level responses titled **“Author Rebuttal by Authors”**.\n\n|Method (on Cityscapes)|mIOU (%)|$\\Delta$mIOU(%)|$T_{train}(h)$\n|:--|:--:|:--:|:--:|\nBaseline|73.20|n/a|n/a\n+$L_{fd}$|75.88|+2.68|4.02\n+$L_{Af-DCD}^{OC}$|75.98|+2.78|4.06\n\n|Method (on ADE20K)|mIOU (%)|$\\Delta$mIOU(%)|$T_{train}(h)$\n|:--|:--:|:--:|:--:|\nBaseline|33.91|n/a|n/a\n+$L_{fd}$|34.92|+1.01|4.32\n+$L_{Af-DCD}^{OC}$|35.41|+1.50|4.35\n\n**2. Your comments on our discussions about the limitations of the proposed method Af-DCD** “The authors have listed the limitation in the supplementary material, i.e., the gain of Af-DCD on Transformer based architecture are not as large as on CNN based architecture. We believe this limitation can be seen as a future direction”.\n\n**Our responses**: Thank you for accepting the limitations of our method we have discussed in the last Section “Limitations of Af-DCD” of the supplementary material. Here, we add more discussions to further improve the clarification on the main limitation of our method: **(1)** The reason for why the current design of our Af-DCD cannot easily generalize to transformer-based structures is: Af-DCD exploits dense pixel-wise information within each of local patches via the feature partition across both channel and spatial dimensions for formulating contrastive feature mimicking conditioned on the single image input fed to both the pre-trained teacher model and the target student model, but transformer-based structures built upon self-attention modules primarily encode global patch-to-patch feature dependencies in an image input, which appear to be in conflict with each other; **(2)** Previously, we performed a distillation experiment to explore this. In the experiment, we applied Af-DCD to SegFormer (MiT-B4 encoder as teacher and MiT-B0 encoder as student) [1], a seminal transformer-based structure for semantic segmentation. Detailed results are summarized in the below table where Af-DCD only brings 0.31% mIOU gain to the baseline; **(3)** A potential direction to address the above issue is how to preserve local information and make a good alignment of Af-DCD to transformed-based structures. Please allow us to leave it as a future research direction.\n\nMethod (on Cityscapes)|mIOU(%)|$\\Delta$mIOU(%)\n|:--|:--:|:--:|\nTeacher: SegFormer-MiT-B4|81.23|n/a\nStudent (baseline): SegFormer-MiT-B0|75.58|n/a\nAf-DCD|75.89|+0.31\n\n[1] Enze Xie, et al. “SegFormer: Simple and efficient design for semantic segmentation with transformers”, NeurIPS 2021.\n\n**Finally**, during the rebuttal phase, we also conducted more experiments to improve ablation studies and added discussions to improve the clarifications of our method. You are referred to our top-level responses titled **“Author Rebuttal by Authors”**, and our responses to the other reviewers for details.\n\nLooking forward to your feedback."}]

Author rebuttal:
Dear Reviewers, Area Chairs, Senior Area Chairs and Program Chairs,

We sincerely thank all four reviewers for their thorough and constructive comments. We are glad that the novelty, basic experiments and performance of our work have been mostly recognized by all four reviewers.

In the past week, we carefully improved the experiments (using all computational resources we have), the clarifications and the discussions of our work to address the concerns, the questions and the requests by all four reviewers. **Summarily, we made the following improvements**: 

**(1)** To have a better understanding of the effectiveness of our Af-DCD method, we follow the constructive comments/requests by Reviewer YQfC, Reviewer vSYf and Reviewer wLTq and add several sets of ablative experiments on Cityscapes and ADE20K datasets using the same experimental setups to Table 4, including: **(a)** An ablation study to compare the combination of our basic channel contrasting loss and spatial contrasting loss with our omni-contrasting loss; **(b)** An ablation study to compare our contrasting loss with 3 types of distance function; **(c)** An ablation study to compare the results with and without our omni-contrasting loss in the presence of all non-contrasting losses; **(d)** An ablation study to compare our basic feature imitation loss and our omni-contrasting loss; **(e)** Besides the ablation studies reported in the original manuscript, these new ablation studies further demonstrate the effectiveness of our method, and show more insights by experimental observations.

**(2)** We follow the constructive comments/suggestions/requests from all four reviewers, and add more discussions and clarifications to improve the presentation, the explanations of our design insights, experiments and method’s limitations.

**Finally, in the attached one-page PDF file, all the aforementioned experimental results are summarized in different Tables**. We will include the above experiments and discussions in our final paper. We hope our detailed responses are helpful to address the concerns, the questions and the requests of all four reviewers.

Meta-review: Four experts reviewed the paper, and all recommended Borderline/Weak Accept. The reviewers liked the method and the ablation studies, and they were impressed by the overall results. The rebuttal did a good job answering the reviewers' questions and/or concerns, so no reviewer recommended rejection after the rebuttal. Hence, the decision is to recommend the paper for acceptance. The authors are encouraged to incorporate the reviewers' suggestions and the rebuttal into the revision.
Decision: Accept (poster)