Title: Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Knowledge distillation

semantic segmentation

contrastive learning
Abstract: In recent years, knowledge distillation methods based on contrastive learning have achieved promising results on image classification and object detection tasks. However, in this line of research, we note that less attention is paid to semantic segmentation. Existing methods heavily rely on data augmentation and memory buffer, which entail high computational resource demands when applying them to handle semantic segmentation that requires to preserve high-resolution feature maps for making dense pixel-wise predictions. In order to address this problem, we present Augmentation-free Dense Contrastive Knowledge Distillation (Af-DCD), a new contrastive distillation learning paradigm to train compact and accurate deep neural networks for semantic segmentation applications. Af-DCD leverages a masked feature mimicking strategy, and formulates a novel contrastive learning loss via taking advantage of tactful feature partitions across both channel and spatial dimensions, allowing to effectively transfer dense and structured local knowledge learnt by the teacher model to a target student model while maintaining training efficiency. Extensive experiments on five mainstream benchmarks with various teacher-student network pairs demonstrate the effectiveness of our approach. For instance, DeepLabV3-Res18|DeepLabV3-MBV2 model trained by Af-DCD reaches 77.03\%|76.38\% mIOU on Cityscapes dataset when choosing DeepLabV3-Res101 as the teacher, setting new performance records. Besides that, Af-DCD achieves an absolute mIOU improvement of 3.26\%|3.04\%|2.75\%|2.30\%|1.42\% compared with individually trained counterpart on Cityscapes|Pascal VOC|Camvid|ADE20K|COCO-Stuff-164K. Code is available at https://github.com/OSVAI/Af-DCD.

Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation

Abstract

In recent years, knowledge distillation methods based on contrastive learning have achieved promising results on image classification and object detection tasks. However, in this line of research, we note that less attention is paid to semantic segmentation. Existing methods heavily rely on data augmentation and memory buffer, which entail high computational resource demands when applying them to handle semantic segmentation that requires to preserve highresolution feature maps for making dense pixel-wise predictions. In order to address this problem, we present Augmentation-free Dense Contrastive Knowledge Distillation (Af-DCD), a new contrastive distillation learning paradigm to train compact and accurate deep neural networks for semantic segmentation applications. Af-DCD leverages a masked feature mimicking strategy, and formulates a novel contrastive learning loss via taking advantage of tactful feature partitions across both channel and spatial dimensions, allowing to effectively transfer dense and structured local knowledge learnt by the teacher model to a target student model while maintaining training efficiency. Extensive experiments on five mainstream benchmarks with various teacher-student network pairs demonstrate the effectiveness of our approach. For instance, the DeepLabV3-Res18|DeepLabV3MBV2 model trained by Af-DCD reaches 77.03%|76.38% mIOU on Cityscapes dataset when choosing DeepLabV3-Res101 as the teacher, setting new performance records. Besides that, Af-DCD achieves an absolute mIOU improvement of 3.26%|3.04%|2.75%|2.30%|1.42% compared with individually trained counterpart on Cityscapes|Pascal VOC|Camvid|ADE20K|COCO-Stuff-164K. Code is available at https://github.com/OSVAI/Af-DCD .

1 Introduction

In computer vision, semantic segmentation is a mainstream dense prediction task aiming to assign the corresponding class to every pixel of input images. Although fundamental models in this task have achieved remarkable progress [1-3], the heavy computational burden and latency of these models severely prohibit their deployment on resource-constrained devices. As a popular solution to this bottleneck, Knowledge Distillation (KD) [4] aims to transfer the knowledge from large models to light-weight ones. Considering KD in supervised semantic segmentation, teachers usually have

(Pre-trained and fixed)

×

𝐿𝑘𝑑

𝐿𝑓𝑑

Generator

𝐿𝐴𝑓-𝐷𝐶𝐷

Mask

Head

Head

𝐿𝑡𝑎𝑠𝑘

𝐹 𝑡

෠

𝐹 𝑠

𝐹 𝑠

Student Network

stronger ability of capturing detailed local information of images where the dense and structured knowledge contributes significantly to segment complicated areas, such as boundary and occlusion. Therefore, distillation methods tailored to this task should keep the integrity of dense and structured knowledge, rather than compressing it excessively such as simply using global representations or salient areas. As a result, directly applying traditional KD methods in semantic segmentation, like vanilla KD [4] or feature imitation [5], cannot achieve promising results in most cases.

Thanks to the development of contrastive learning, some distillation methods, like SSKD [6], CRD [7] and G-DetKD [8], employed this paradigm into their designs, which promoted the performance significantly, as they exploited teacher's intrinsic knowledge in the consistencies among different augmentations or various same-category instances. In semantic segmentation, CIRKD [9] designed an implicit contrastive method which aimed to guarantee pixel-pixel and pixel-region consistencies between teacher and student by involving additional contrastive samples. On one side, these existing methods have achieved remarkable progress in image-level or object-level contrastive mimicking. On the other side, however, the efficacy of the aforementioned methods heavily relies on the augmentation and memory buffer, which entail heavy computational and memory cost. Besides, previous contrastive distillation methods have not explicitly modeled relations among pixel-wise or more fine-grained representations within each local patch to transfer teacher's dense and structured knowledge to student, which is critical in semantic segmentation. Thus, when designing a contrastive distillation method for this task, two vital technical problems should be tackled: (i) High resource demands : No matter leveraging augmented samples or storing feature maps in memory buffer, more computational or storage cost is required. For one, the forward of augmented samples incurs extra computational cost. For another, the output feature maps for semantic segmentation are in high resolution. If we want to make dense and structured contrasting, these feature maps should be originally preserved, which occupy a large amount of memories; (ii) Structured knowledge transfer : Although CIRKD defined pixel-wise alignment between student and teacher, it employed contrastive pixel-wise representations from other images, rather than local areas within the same image. It means that no explicit contrasting was defined between student-specific pixel-wise representation to each teacher's pixel-wise representation within local areas, which was agnostic to the structure in teacher's feature. In brief, a contrastive distillation method specially tailored to semantic segmentation, which also entails no extra high resource demands (data augmentation and memory buffer), is essential .

Driven by achieving this target, we first look into the aforementioned two problems and surprisingly discover both of them are incurred from the simple inheritance in the basic definitions of traditional contrastive learning [10, 11]. Therefore, our Augmentation-free Dense Contrastive Knowledge Distillation (Af-DCD) for supervised semantic segmentation tasks can come out, by re-defining these basic concepts: (i) Contrasting samples are not coarse-grained representations of images or objects. Instead, in our Af-DCD, we move further on pixel-level representations and divide each of them into several disjoint partitions (fine-grained representations), which are treated as our contrasting samples; (ii) Positive and negative pairs are not conditioned on categories or objects. Instead, based on the definition of contrasting samples, we define our pairs in a specific teacher-student feature pair ( F t , F s ) , where fine-grained representations having the same absolute positions in feature maps are used to formulate positive pairs, while representations having different absolute positions but within neighbourhoods are used as negative pairs. Based on these definitions, our Af-DCD can naturally tackle those two problems discussed above. To the first problem, unlike previous methods which had to introduce data augmentation and memory buffer to construct sufficient positive and negative pairs,

our Af-DCD can directly construct these pairs in each local patch without any extra computational and memory cost. To the second problem, we approach it by introducing two new contrastive designs, Spatial Contrasting and Channel Contrasting , which enable student to capture teacher's local dense and structured knowledge by focusing on contextual and positional channel-group information, respectively. As a natural progression, we introduce a hybrid design called Omni-Contrasting , which combines Spatial Contrasting and Channel Contrasting in a neat manner. This design facilitates the simultaneous transfer of both types of information from the teacher to the student, enabling effective augmentation-free contrastive knowledge distillation. Despite leveraging rather dense contrasting, our Af-DCD also performs efficiently. This is due to the patch separation technique we use, which significantly reduces the computational complexity. Additionally, the distance measuring and contrasting calculation can be carried out in parallel, further enhancing the overall efficiency of our method.

Experimental results demonstrate that: (i) Af-DCD exhibits superior performance compared to state-of-the-art methods, on various benchmarks with different teacher-student network pairs; (ii) Af-DCD exhibits even more significant improvements on larger datasets, such as ADE20K, indicating it can enhance student's generalization capability. Moreover, our analytical experiments illustrate that by effectively learning teacher's self-similarity distribution within neighbourhoods and thus reducing the fine-grained feature distances to the teacher, Af-DCD benefits addressing difficult scenarios in semantic segmentation.

2 Related Works

Knowledge Distillation. Knowledge distillation can be generally divided into probability-based approach and feature-based approach. Specifically, the former one forces student to mimic teacher's logits as soft-labels [4, 12], while the later one leverages teacher's hidden feature maps or its variants [5, 13, 14] as distillation supervisions. Some recent feature-based approaches [15-17] designed their distillation methods based on masked image modeling mechanism [18, 19] and achieved promising performance in various tasks, as the feature reconstruction can enhance the interdependencies among pixels, which is beneficial to feature distillation.

Knowledge Distillation in Semantic Segmentation. As semantic segmentation is a dense prediction task, the methods specially designed for this task aimed to capture teacher's structured local information. In order to achieve this target, SKD [20] directly measured the similarities of pixel-wise representations between teacher and student, while He et al. [21] leveraged non-local operation with autoencoder to encode local information. CWD [22] evaluated channel-wise pixel distribution contributing to learn teacher's spatial information in each individual channel. Some methods further explored intrinsic knowledge among different samples. For instance, IFVD [23] measured distances from prototypes of different classes and forced student to mimic teacher's intra-class relations, and CIRKD [9] forced student to keep pixel-wise and region-wise consistencies to teacher among various samples in memory buffer.

Contrastive Knowledge Distillation. Inspired by the development of contrastive learning in selfsupervised [10, 24] and supervised tasks [11], some recent works leveraged this paradigm and designed new contrastive distillation methods, which made contrasting between teacher's and student's features by employing different views generated from data augmentation [6], or using various samples' features [25, 7] as well as gradients [26] stored in memory buffer. Specifically in dense prediction tasks, G-DetKD [8] constructed ROI feature pairs and executed soft semantic-guided matching which promoted the performance in object detection, and CIRKD [9] designed an implicit contrastive method which leveraged both pixel and region representations to learn structured information in spatial dimension.

3 Method

3.1 Overall Framework

Figure 1 illustrates our distillation process in semantic segmentation from a pre-trained teacher T to a student S which needs to be trained on a specific dataset. The output features of T and S are denoted as ˆ F t ∈ R H × W × C t and ˆ F s ∈ R H × W × C s respectively, where H and W are height and width of

i,j,k

Fse

ES

,v

Fse

i,j,w

F'S

p,i,j,k

FS

p,i,j,k

Fs

p,u,v,w

feature maps, C t and C s are the number of channels of teacher and student, respectively. Our method uses a random mask M ∈ R H × W × 1 to overlap ˆ F s in spatial dimension and then inputs the masked feature into a generator, which can be formulated as F s = f generator ( M ⊙ ˆ F s ) . Finally, we can get F s ∈ R H × W × C t , which has the same dimension as teacher's feature. In order to fit general definition, teacher's feature is also processed by a transform F t = f t ( ˆ F t ) . In our design, f t is an identity transformation, thus F t = ˆ F t . After these operations, the reconstructed student feature F s has been projected into the same feature space as teacher's feature F t . The feature imitation loss L fd and our Af-DCD loss L Af -DCD are calculated based on feature pair ( F t , F s ) . Besides, following the settings in [9, 22], we also employ vanilla logits-based KD loss [4]. In general, the overall loss of our method can be defined as:

where λ i is the balancing weight.

3.2 Augmentation-free Dense Contrastive Knowledge Distillation

After illustrating the overall framework, we further explain our core design in detail, which contains two major parts, L fd and L Af -DCD . As we mentioned above, both of them are based on the same feature pair ( F t , F s ) . Therefore, in order to clarify our design, we first answer two key questions: (i) What are the targets of L fd and L Af -DCD separately? (ii) How these two losses cooperate with each other? To the first question, L fd directly help student to imitate teacher's global and salient features, while L Af -DCD is designed to mimic teacher's dense structured knowledge within local areas, which can further force F s to approach F t in the micro and fine-grained views. To the second question, in terms of mimicking dense and structured knowledge, these two losses are inextricably bound up with each other. For one, L fd provides L Af -DCD an absolute imitation trend, which cannot be implemented in contrastive loss. For another, L Af -DCD offers L fd essential constrains on encouraging this process towards mimicking teacher's structured knowledge in each local patch, making student further approach teacher.

Masked Reconstruction based Feature Imitation. The feature imitation loss L fd is a basic part of our design, which forces student feature F s to imitate teacher feature F t directly. Previous methods commonly leverage simple upsampling to align teacher's and student's feature dimensions. Recently, some methods leveraged masked image reconstruction [15, 17], which is a stronger baseline, as it can promote the performance by strengthening the interdependencies among pixels. This process can be formulated as:

where M ∈ R H × W × 1 is the mask matrix generated randomly with the mask ratio ζ ∈ [0 , 1) . Then feature distillation can be formulated as:

Although masked reconstruction achieves promising performance, one potential weakness may affect its gain in semantic segmentation: Reconstructed features tend to be similar in local areas 3 . However, in semantic segmentation, local differences indicate structured knowledge, in which student's reconstructed feature maps should keep these differences. Here comes out why we choose masked reconstruction as our basic feature distillation method: (i) choosing a stronger baseline to further test the effectiveness of our Af-DCD; (ii) examining whether our Af-DCD can promote masked reconstruction for generating more structured representations, enhancing feature imitation process.

Motivation and Key Ideas of Af-DCD. In semantic segmentation, we notice two important facts. On one side, different pixels within local areas may contain different semantic information, as they may belong to different categories or different parts of an object. We call this as contextual information . On the other side, differences among channel groups of a pixel representation implicitly indicate the semantic meaning of this pixel, as each output channel is a specific projection of all input feature maps. We call this as positional channel-group information . In short, the contextual and positional channel-group information within each local area in spatial and channel dimensions is of significance in semantic segmentation. If both types of information from teacher's feature maps can be densely utilized as the distillation guidance to facilitate the training of the student model, we conjecture that student's performance and generalization capability can be greatly boosted. However, traditional feature imitation loss L fd is difficult to capture these two types of information, as they are not directly observable and measurable comparing to salient representation. Aiming to tackle this problem, we define contrastive loss across student's and teacher's pixel-level or more fine-grained representations to explicitly model such kind of knowledge transfer. Specifically, as shown in Figure 2, Af-DCD incorporates three key ideas:

Spatial Contrasting. With the first phenomenon, we leverage pixel-wise representations in F s and F t and define pixel-wise dense contrasting based on spatial positions, aiming to transfer teacher's spatial contextual information to student;

Channel Contrasting. With the second phenomenon, we then move further on pixel-wise dense contrasting and split every pixel-specific representation into disjoint groups and define group-wise dense contrasting based on channel positions, aiming to force student to learn teacher's positional channel-group information;

Omni-Contrasting. Progressively, we unify the above two contrasting methods in an hybrid design, which takes advantage of tactful feature partitions across both channel and spatial dimensions in local areas by splitting the feature map into disjoint patches of the same size.

Formulation of Af-DCD. The formulation of Af-DCD follows the notations in Section 3.1. The projected feature maps of teacher and student are denoted as F t ∈ R H × W × C t and F s ∈ R H × W × C t , respectively. In the perspective of contrastive learning, we first clarify some important concepts in our Af-DCD:

Samples and Views. Different from existing contrastive distillation methods in classification and detection, the samples in Af-DCD are pixel-level or more fine-grained representations in feature maps of the same image, which are naturally dense. Furthermore, we define that representations of teacher and student are two views which should be aligned by our contrastive loss;

̸

Positive Pairs and Negative Pairs. Under the definitions above, a positive pair can be defined as: a teacher-student sample pair ( F s i , F t j ) which has the same index i = j , where i and j are general location indices. Similarly, a negative pair can be defined as: a teacherstudent sample pair ( F s i , F t j ) whose indices are different i = j .

Based on the above definitions, the loss for a specific sample F s i ∈ F s can be formulated as:

̸

̸

where τ denotes the temperature score, N is the total number of contrastive pairs and 1 i = j is an indicator function which is 1 iff. i = j . Different from traditional contrastive loss that adopts cosine distance on measuring similarities, we use Euclidean distance to measure sample pair similarities, where d ( F s i , F t i ) = || F s i -F t i || 2 2 . Our ablative experiments illustrate that improved performance would be attained when we select the same type of the distance function for L fd and L Af -DCD .

̸

̸

Spatial Contrasting. As shown in Figure 2 (top left), aiming to learn teacher's spatial contextual information, Spatial Contrasting constructs dense contrasting operations between teacher's and student's pixel-level representations, where for each sample F s i,j , the positive pair is ( F s i,j , F t i,j ) , while the other { ( F s i,j , F t u,v ) } u,v = i,j are all negative pairs. In such condition, we obtain 1 postive pair and HW -1 negative pairs. Then Spatial Contrasting can be formulated as:

̸

̸

Channel Contrasting. As shown in Figure 2 (top right), Channel Contrasting first splits each pixel representation into M non-overlapping channel groups of the same length. Specifically, F s i,j and F t i,j are split into { F s i,j,w } w =1 ...M and { F t i,j,w } w =1 ...M , where w is the index of channel group. Then, in order to transfer teacher's channel-group information at each pixel position, the dense contrasting happens on each pixel-level representation pair between { F s i,j,w } w =1 ...M and { F t i,j,w } w =1 ...M . For each fine-grained representation F s i,j,k , the positive pair is ( F s i,j,k , F t i,j,k ) and negative pairs are { ( F s i,j,k , F t i,j,w ) } w = k , where the number of positive and negative pairs is 1 and M -1 , respectively. Then we substitute these pairs into Formula 4, where w denotes the index. Channel contrasting can be formulated as:

̸

̸

Omni-Contrasting. Omni-Contrasting is an neat combination of Channel Contrasting and Spatial Contrasting, shown in Figure 2 (bottom). Different from the design of Spatial Contrasting, Omni-Contrasting does not contrast all positions of feature maps. Instead, in order to exploit local information, Omni-Contrasting groups pixels into a series of local patches and leverages spatial and channel contrasting within each local patch. This local contrasting can force student to learn teacher's spatial contextual information and positional channel-group information simultaneously, which are beneficial for accurately segmenting the boundary and correctly classify those pixels. After that, F t and F s are split into { F t p } p =1 ...N and { F s p } p =1 ...N , where F t p , F s p ∈ R ˆ H × ˆ W × C t and N = HW ˆ H ˆ W . Then, like the operation in Channel Contrasting, the channels will be divided into M non-overlapping groups. Teacher's and student's representations at position ( i, j, k ) in local patch p are denoted as F t p,i,j,k and F s p,i,j,k , respectively, where F t p,i,j,k , F s p,i,j,k ∈ R 1 × 1 × C t M . For F s p,i,j,k , the positive pair is ( F s p,i,j,k , F t p,i,j,k , ) , while negative pairs are ( F s p,i,j,k , F t p,u,v,w ) for any u, v, w = i, j, k , where the number of positive and negative pairs is 1 and ˆ H ˆ WM -1 , respectively. The Omni-Contrasting can be defined as:

̸

Finally, the overall Omni-Contrasting loss can be defined as:

(a) The same framework with different backbones

(b) Different frameworks with the same backbone

4 Experiments

4.1 Datasets and Experimental Setups.

Datasets. Five popular semantic segmentation datasets, including Cityscapes [27], Pascal VOC [28], Camvid [29], ADE20K [30] and COCO-Stuff-164K [31], are used in our experiments. We conduct our ablation studies on both Cityscapes and ADE20K, which help us to ensure the best setting in most experiments and analyse the effectiveness of our design. Details for these five datasets are described in supplemental material.

Experimental Settings. Following general settings [9, 20, 15] in semantic segmentation distillation, we adopt DeeplabV3 [32] and PSPNet [3] for segmentation framework, ResNet-18 [33] and Mobilenetv2 [34] for student backbones, Resnet-101 for teacher backbone and group various teacherstudent pairs. In order to make fair comparison with different state-of-the-art methods, we implement our method on both MMSegmentation codebase [35] and CIRKD codebase [9]. In training and evaluation, we use mean Intersection-over-Union (mIoU) to measure the performance of all methods. In training phase, all models are optimized by SGD with the momentum of 0.9, the initial learning rate of 0.02, and the batch size of 16. The input size is 512 × 1024 , 400 × 400 , 512 × 1024 , 512 × 1024 , for experiments on Pascal VOC, CamVid, ADE20K and COCO-Stuff-164K, respectively. The input size for experiments on Cityscapes are different in the two codebase, 512 × 1024 in CIRKD codebase and 512 × 512 in MMSegmentation codebase [15]. In evaluation phase, we follow general settings in [22], which evaluate the performance with the original image size. Our masked reconstruction generator consists of two 3 × 3 convolutional layers with ReLU, following [15]. Other default hyper-parameter settings and implementation details are described in supplemental material.

4.2 Main Results

In this part, we intend to compare the distillation performance of our method Af-DCD with recent state-of-the-art methods for semantic segmentation. Aiming for comprehensive comparison, we conduct experiments on the aforementioned five public datasets following general settings.

Results on Cityscapes. In Table 1, we conduct experiments on the most popular Cityscapes dataset to validate the generalization ability of our method to different teacher-student network pairs. The experimental results show that Af-DCD outperforms state-of-the-art methods in most cases, with the maximal margin of 1.12%. In average, Af-DCD brings 3.44% gain to the baseline student models, with the maximal gain of 4.37%. From the results shown in Table 1a, we can see that our method can well handle teacher-student network pairs in which students (e.g., DeepLabV3-Res18 and DeepLabV3-MBV2) have the same segmentation framework but with different backbones. The

(a) Pascal VOC

(b) Camvid

(c) ADE20K

(d) COCO-Stuff-164K

results of Table 1b further show that our method can also generalize well to teacher-student network pairs in which students (e.g., DeepLabV3-Res18 and PSPNet-Res18) have different segmentation frameworks but with the same backbone.

Results on Four Other Datasets. In Table 2, we evaluate the performance of Af-DCD on four other datasets including PASCAL VOC, Camvid, ADE20K and COCO-Stuff-164K to examine the generalization of our design to handle different semantic segmentation tasks. According to the results shown in Table 2a-2d, Af-DCD outperforms state-of-the-art methods by significant margins on these four datasets, with the maximal and average margin of 1.75% and 1.22%, respectively. Furthermore, we can observe that our method consistently shows significant absolute mIOU gains (from 1.42% to 3.04%) to different student models on small-size (Camvid), medium-size (Cityscapes and PASCAL VOC) and large-size (ADE20K and COCO-Stuff-164K) datasets.

4.3 Ablation Studies

Ablation Study on Different Loss Terms. In our formulation, the overall loss contains three distillation terms, including L kd , L fd and L Af -DCD . Accordingly, we conduct experiments that independently test the gain from each term to explore the nature of Af-DCD. From the results shown in Table 3, we can get following observations: (i) The accuracy gain of Baseline + L OC Af -DCD is slight on relatively small dataset Cityscapes (0.28% mIOU gain), but it is notably pronounced on much larger dataset ADE20K (1.50% mIOU gain); (ii) Compared to Baseline + L fd , Baseline + L OC Af -DCD gets student model with better accuracy on ADE20K dataset, while maintaining almost the same training efficiency; (iii) Baseline + L fd + L OC Af -DCD gets student models with 76.44% mIOU and 36.01% mIOU on Cityscapes dataset and ADE20K dataset, respectively, which are obviously better than both Baseline + L fd and Baseline + L OC Af -DCD , showing that two loss terms L fd and L OC Af -DCD are complementary; (vi) L kd can further bring minor extra gains, 0.08% and 0.2%, to Baseline + L fd + L OC Af -DCD on Cityscapes and ADE20K dataset, respectively.

Ablation Study on Different Designs. In this study, we conduct a set of experiments on our three basic contrasting designs, namely Channel Contrasting (CC), Spatial Contrasting (SC) and Omni-Contrasting (OC), in order to exploit the gains from different contrasting dimensions and further verify the necessity of our OC. In Table 3a and 3b, we can observe two phenomena:

77.0

77.0

77.0

77.0

77.0

AfC-DCD

Af-DCD

Af-DCD

AfC-KD

Af-DCD

76.5

baseline

76.5

baseline

76.5

baseline

76.5

76.5

baseline

76.0

76.0

76.0

76.0

76.0

(%)

75.5

(%)

75.5

75.5

(%)

75.5

(%)

75.5

no

75.0

now

75.0

mlOU

75.0

75.0

mlOU

75.0

74.5

74.5

74.5

74.5

74.5

74.0

74.0

74.0

74.0

74.0

73.5

73.5

73.5

73.5

73.5

73.0

0.05

0.06

0.07

0.08

0.09

73.0

1e-3

2e-3

5e-3

1e-2

2e-2

73.0

4

8

16

32

64

73.0

1

2

4

16

4*

73.0

0

0.25

0.5

0.75

0.9

T

入3

M

N

(i) CC has similar gain to baseline model as SC on Cityscapes, while has much stronger performance on ADE20K, exceeding 0.5% to SC; (ii) The combination of CC and SC performs better than CC and SC, but worse than OC on both datasets (in distillation accuracy and training speed). The above two observations show the superiority of OC, which comes from the neat combination of CC and SC. Specifically, OC groups pixels into a number of disjoint local patches and tactfully leverages CC and SC within each local patch instead of the holistic feature maps to better exploit dense and structured local information for contrastive feature mimicking.

Ablation Study on Choice of the Function d in the Omni-Contrasting Loss. We compare Formula 7 of our method with 3 types of the function d including L 2 -normed distance (our choice), cosine similarity (commonchoice in contrastive loss) and L 1 -normed distance. From the results summarized in Table 4, we can see that our method always shows significant mIOU gains to the baseline with all 3 types of the function d . Comparatively, our method with L 2 -normed distance is the best, which supports our intuition that improved performance would be attained by choosing the same type of the function d for the feature distillation loss (Formula 3) and the Omni-Contrasting loss (Formula 7).

Ablation Study on Training Efficiency. Our Af-DCD is naturally augmentation-free and memory-buffer-free. We evaluate the efficiency of our design in terms of training time and GPU memory occupation. In Table 3a, we can observe that Af-DCD introduces minor extra training cost, from 4.02 hours to 4.25 hours, which only increases 5.7% training time to feature distillation. From the results shown in Table 5, we can see Af-DCD uses less memory and training time, but achieves better performance, compared to CIRKD.

Ablation Studies on Major Hyper-parameters. Recall that our method has five hyper-parameters, we also perform experiments to study them. As shown in Figure 3, the effectiveness of our method is relative stable when values of these hyper-parameters are changing. Notably, we find using max

(a) Feature distance between T and S

(b) Feature distance within local areas

2.5 -

2.0

FD

A-DCD

FD

Af-DCD

FD

A-DCD

Teacher

1.5

BoUe

1.0

0.5

PAf-DCD

>PFD

0.0

PAf-DCD<PFD

0.5

0.5

0.0

0.5

Eud. Distance

1.0

2.0

0.00

0.05

0.10

Eud. Distance

0.15

0.20

0.25

2.5

5.0

7.5

Eud. Distance

10.0

12.5

15.0

17.5

20.0

Object Boundary

Object Occlusion

Small Object

Image Boundary

DifficultCategory

RareView

(c) Heat maps of areas where model using Af-DCD segments correctly but using FD segments wrong

pooling with proper scale has no obvious effect on model performance but it has better efficiency than larger N . Other detailed analysis is referred to supplementary materials.

4.4 Discussion

After demonstrating the superior performance of Af-DCD in Section 4.2 and analysing the gain of Af-DCD in Section 4.3, we perform some verification experiments, aiming to discuss how L Af -DCD boosts the distillation performance of L task + L fd (denoted as FD for abbreviation).

As shown in Figure 4b, the self-similarity distribution of fine-grained representations projected by FD has large differences from that of teacher's feature representations. In order to alleviate the aforementioned problem, L Af -DCD contrasts feature partitions within each local patch between student and teacher. The results in Figure 4a and Figure 4b are summarized as below:

L Af -DCD dramatically decreases the distances of student to teacher in views of fine-gained representations and self-similarities, which proves our basic assumption that L Af -DCD can further make student approach teacher in the micro and fine-grained views;

L Af -DCD increases both the mean and variance of the self-similarity distribution of student. Such observation verifies that L Af -DCD can force student to learn dense and structured knowledge implicitly contained among teacher's feature partitions.

The above two improvements are proved to be effective in addressing segmenting various difficult scenarios in semantic segmentation, such as object boundary, small object, object occlusion, difficult category and rare view. Heat maps shown in Figure 4c illustrate that L Af -DCD can effectively help FD correctly classify these difficult pixels, which further verifies the superiority of Af-DCD.

More examples and detailed analysis are referred to supplementary material.

5 Conclusion

In this paper, we present Af-DCD, an augmentation-free dense contrasive distillation method tailored to semantic segmentation. The dense contrasting in Af-DCD is an omni-dimensional design. Thus, it can effectively transfer teacher's contextual and positional channel-group information to student. Experimental results show that Af-DCD is effective on different teacher-student network pairs and datasets while maintaining training efficiency, as it is born with augmentation-free and memory-buffer free. We hope our work can inspire researchers to explore more powerful and efficient contrastive distillation methods in the future.

References

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI , 2017.

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 , 2017.

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR , 2017.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR , 2015.

Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self-supervision. In ECCV , 2020.

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR , 2020.

Lewei Yao, Renjie Pi, Hang Xu, Wei Zhang, Zhenguo Li, and Tong Zhang. G-detkd: towards general distillation framework for object detectors via contrastive and semantic-guided feature imitation. In ICCV , 2021.

Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang. Crossimage relational knowledge distillation for semantic segmentation. In CVPR , 2022.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML , 2020.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS , 2020.

Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. arXiv preprint arXiv:2102.00650 , 2021.

Xiaolong Liu, Lujun Li, Chao Li, and Anbang Yao. Norm: Knowledge distillation via n-to-one representation matching. In ICLR , 2023.

Nikos Komodakis and Sergey Zagoruyko. Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. In ICLR , 2017.

Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked generative distillation. In ECCV , 2022.

Zhendong Yang, Zhe Li, Ailing Zeng, Zexian Li, Chun Yuan, and Yu Li. Vitkd: Practical guidelines for vit feature knowledge distillation. arXiv preprint arXiv:2209.02432 , 2022.

Tao Huang, Yuan Zhang, Shan You, Fei Wang, Chen Qian, Jian Cao, and Chang Xu. Masked distillation with receptive tokens. In ICLR , 2023.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR , 2022.

Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR , 2022.

Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In CVPR , 2019.

Tong He, Chunhua Shen, Zhi Tian, Dong Gong, Changming Sun, and Youliang Yan. Knowledge adaptation for efficient semantic segmentation. In CVPR , 2019.

Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen. Channel-wise knowledge distillation for dense prediction. In ICCV , 2021.

Yukang Wang, Wei Zhou, Tao Jiang, Xiang Bai, and Yongchao Xu. Intra-class feature variation distillation for semantic segmentation. In ECCV , 2020.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , 2020.

Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation. arXiv preprint arXiv:2101.04731 , 2021.

Jinguo Zhu, Shixiang Tang, Dapeng Chen, Shijie Yu, Yakun Liu, Mingzhe Rong, Aijun Yang, and Xiaohua Wang. Complementary relation contrastive distillation. In CVPR , 2021.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR , 2016.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV , 2010.

Gabriel J Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recognition using structure from motion point clouds. In ECCV , 2008.

Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV , 2019.

Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR , 2018.

Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV , 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR , 2018.

MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation , 2020.

Peer reviews:
comments: SUMMARY:
In this manuscript, the authors propose a effective knowledge distillation framework for semantic segmentation task. Specifically, in addition to traditional knowledge distillation on segmentation masks as well as feature distillation, to better align the dense feature, the authors introduce contrastive learning on both spatial dimension and channel dimension. The proposed Af-DCD loss significantly improves the performance of CNN-based segmentation network via knowledge distillation without data augmentation.

STRENGTHS:
1. The motivation is clear, i.e., traditional feature distillation loss is somewhat difficult to contextual information and positional channel-group information.

2. The extensive experiments demonstrate the effectiveness of the proposed method. The ablation study and discussion is abundant and valuable. 

3. The proposed method is easy to follow.

WEAKNESSES:
1. Intuitively, the proposed Af-DCD loss can align the dense feature independently, therefore the authors could show the ablation study of baseline+L_{Af-DCD} only and compare the results with baseline+L_{fd}.

QUESTIONS:
See the weakness section.

LIMITATIONS:
The authors have listed the limitation in the supplementary material, i.e., the gain of Af-DCD on Transformer based architecture are not as large as on CNN based architecture. We believe this limitation can be seen as a future direction.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on knowledge distillation for semantic segmentation and introduces an augmentation-free dense contrastive loss function. The student and teacher feature maps are partitioned into patches, and both spatial and channel contrasting is performed within these local neighborhoods. For contrastive loss, positive/negative feature pairs are formed using tether and student features extracted from the same image without using any augmentations. Experiments were conducted on five segmentation datasets and the proposed approach is shown to perform better than various existing works.\n\nSTRENGTHS:\nPaper was easy to follow.\nExperiments conducted on several datasets.\n\nWEAKNESSES:\nThe title and introduction section emphasize \"augmentation-free\". However, the motivation/need to be \"augmentation-free\" is not clear to me. In the introduction, the authors claim using augmentations leads to high resource demand which I disagree. The proposed approach passes the same image through teacher and student networks. One could also use the proposed loss function as it is by passing original image to one network and an augmented version of the image to the other network. The computation cost will be almost same except the augmentation operation cost which is usually small compared to the whole forward/backprop cost. In fact, using appropriate augmentations may even be helpful as the model will learn robust features that are invariant to these augmentations. \n\nFor most of the pixels (other than those that are close to object boundaries), their neighborhood is surrounded by pixels of the same class, and treating them as negatives in contrastive loss is counter-intuitive to me. Ideally for semantic segmentation, we would want pixels of same class to have similar representations so that they can easily be classified to the same class.\n\nIn line 197, authors mentioned that they use euclidean distances instead of cosine similarity in contrastive loss without providing any explanation.\n\nThe main contribution of this paper is the contrastive loss function L_{AF-DCD}. All the other loss functions are from prior works. In order to show the effectiveness of this loss function, authors should compare results with and without the proposed loss when all the other loss functions are present, i.e., comparison between (L_kd + L_fd) and (L_kd + L_fd + L_afdkd). Such comparison is not provided in Table (4). \n\nTypo: It should be Table. 3 not 4(a) in line 287.\n\nQUESTIONS:\nWhy should the proposed approach be augmentation free? The proposed loss can be used with augmentations also.\n\nWhy euclidean distance in contrastive loss?\n\nCurrent experimental comparisons do not clearly demonstrate the effectiveness of L_{AF-DCD} in the present of all the other losses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper points out that existing knowledge distillation methods have been heavily relying on tdata augmentation and memory buffer, which require high computational resources and this is further amplified when it comes to segmentation task that requires relatively higher resolutions of feature maps for processing. To alleviate this complexity, the method called Af-DCD is proposed, which aims to tackle segmentation task by leveraging knowledge distillation based on a novel contrastive learning. More specifically, this method first leverages masked feature mimicking strategy and proposes a novel contrastive learning loss. Experimental results confirm that the proposed method is effective.\n\nSTRENGTHS:\n1. This paper is easy to read and understand.\n\n2. The proposed method achieves the best performance against competitors.\n\n3. Numerous discussions and ablations are presented to validate the choices.\n\nWEAKNESSES:\n1. In Table 1 and 2, it seems like ours refers to cumulatively adding all the different methods (SKD, IFVD, CWD and etcs..). The presentation needs improvements. \n\n2. It would be better if the authors cite each methods in Table 1 and 2 (SKD, IFVD..) so that the readers do not have to look up what those abbreviations refer to. \n\n3. In section 4.2, it is only 'stated' that the proposed method performs the best. I don't find any analysis, explanations or reasonings. Moreover, Although FLOPs and Params are also included, there is no texts covering them. \n\n4. In line with section 4.2, section 4.3 and 4.4 also lack explanations or reasoning. At least it is not sufficient. These sections are simply stating what the table or figure shows without sufficient analysis or attempts to deliver insights.\n\nQUESTIONS:\nSee weaknesses above.\n\nLIMITATIONS:\nLimitations are propoerly addressed in Section E.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel knowledge distillation methods for semantic segmentation, called Augmentation-Free Dense Contrastive Knowledge Distillation(Af-DCD). Af-DCD is a new attempt on the usage of contrastive learning in the task of knowledge distillation for semantic segmentation, which alleviate the problem of high computational resource brought by data augmentation and memory buffer. Af-DCD utilizes feature partitions across both channel and spatial dimensions, allowing to effectively transfer dense and structured local knowledge learnt by the teacher model to a target student model while maintaining training efficiency. Experimental results on mainstream benchmarks including demonstrate  the effectiveness of the proposed Af-DCD.\n\nSTRENGTHS:\n1.The experiments are sufficient. A lot of experiments and visual analysis have proved the effectiveness and superior performance of the proposed Af-DCD.\n2. The design of Af-DCD is clever and makes use of the structural information of teachers from the aspects of both channel and space.\n3. The overall experiment is solid and the code is available, which is nice.\n\nWEAKNESSES:\n1. The organization of reference is poor. Reference is not added to specific methods in the table. And there is no reference for MaskKD in the whole paper, which actually refers to [17], which is confusing. The instruction for CKD in section2 actually is the instruction for CWD.\n2. Some ablation studies are missing. For example, the lack of combination of Channel Contrasting and Spatial Contrasting in Table 4(a), the choice of function d in formula 7.\n3. No distillation experiment of transformer-based structure has been carried out, and it is explained in the appendix that the transformer-based structure gains little improvement from Af-DCD, which limits the generality of Af-DCD.\n\nQUESTIONS:\n1. Is Omni-Contrasting necessary? The combination of Channel Contrasting and Spatial Contrasting in Table 4(a) is needed to demonstrate the superiority of Omni-Contrasting.\n2. Please add some ablation studies on the choice of the function d in formula 7, which can not only help to screen the appropriate function but also increase interpretability.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}