SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Omni-Contrasting. Omni-Contrasting is an neat combination of Channel Contrasting and Spatial Contrasting, shown in Figure 2 (bottom). Different from the design of Spatial Contrasting, Omni-Contrasting does not contrast all positions of feature maps. Instead, in order to exploit local information, Omni-Contrasting groups pixels into a series of local patches and leverages spatial and channel contrasting within each local patch. This local contrasting can force student to learn teacher's spatial contextual information and positional channel-group information simultaneously, which are beneficial for accurately segmenting the boundary and correctly classify those pixels. After that,  $F^t$  and  $F^s$  are split into  $\{F_p^t\}_{p=1\dots N}$  and  $\{F_p^s\}_{p=1\dots N}$ , where  $F_p^t, F_p^s \in \mathbb{R}^{\hat{H} \times \hat{W} \times C^t}$  and  $N = \frac{HW}{\hat{H}\hat{W}}$ . Then, like the operation in Channel Contrasting, the channels will be divided into  $M$  non-overlapping groups. Teacher's and student's representations at position  $(i,j,k)$  in local patch  $p$  are denoted as  $F_{p,i,j,k}^t$  and  $F_{p,i,j,k}^s$ , respectively, where  $F_{p,i,j,k}^t, F_{p,i,j,k}^s \in \mathbb{R}^{1 \times 1 \times \frac{C^t}{M}}$ . For  $F_{p,i,j,k}^s$ , the positive pair is  $(F_{p,i,j,k}^s, F_{p,i,j,k}^t)$ , while negative pairs are  $(F_{p,i,j,k}^s, F_{p,u,v,w}^t)$  for any  $u,v,w \neq i,j,k$ , where the number of positive and negative pairs is 1 and  $\hat{H}\hat{W}M - 1$ , respectively. The Omni-Contrasting can be defined as:
$$
l _ {A f - D C D} ^ {O C} \left(F _ {p, i, j, k} ^ {s}, F _ {p} ^ {t}\right) = - \log \frac {\exp \left(- d \left(F _ {p , i , j , k} ^ {s} , F _ {p , i , j , k} ^ {t}\right) / \tau\right)}{\sum_ {u = 1} ^ {\hat {H}} \sum_ {v = 1} ^ {\hat {W}} \sum_ {w = 1} ^ {M} \mathbb {1} _ {u , v , w \neq i , j , k} \exp \left(- d \left(F _ {p , i , j , k} ^ {s} , F _ {p , u , v , w} ^ {t}\right) / \tau\right)}. \tag {7}
$$
Finally, the overall Omni-Contrasting loss can be defined as:
$$
L _ {A f - D C D} ^ {O C} = \frac {1}{H W M} \sum_ {p = 1} ^ {N} \sum_ {i = 1} ^ {\hat {H}} \sum_ {j = 1} ^ {\hat {W}} \sum_ {k = 1} ^ {M} l _ {A f - D C D} ^ {O C} \left(F _ {p, i, j, k} ^ {s}, F _ {p} ^ {t}\right). \tag {8}
$$
Table 1: Performance comparison of Af-DCD and recent state-of-the-art distillation methods on Cityscapes, evaluated with different teacher-student segmentation network pairs.  $\dagger$  denotes distillation without  $L_{kd}$ . We follow the way in CIRKD [9] to calculate FLOPs. Best results are bolded.
(a) The same framework with different backbones
Method	Params (M)	FLOPs (G)	mIOU (%)
Val	Test
T: DeepLabV3-Res101	61.1M	2371.7G	78.07	77.46
S: DeepLabV3-Res18	74.21	73.45
SKD [20]	75.42	74.06
IFVD [23]	75.59	74.26
CWD [22]	13.6M	572.0G	75.55	74.07
CIRKD [9]	76.38	75.05
MasKD [17]	77.00	75.59
Af-DCD	77.03	75.12
S: DeepLabV3-MBV2	73.12	72.36
SKD [20]	73.82	73.02
IFVD [23]	73.50	72.58
CWD [22]	3.2M	128.9G	74.66	73.25
CIRKD [9]	75.42	74.03
MasKD [17]	75.26	74.23
Af-DCD	76.38	75.06
(b) Different frameworks with the same backbone
Method	Params (M)	FLOPs (G)	Val mIOU (%)
T: PSPNet-Res101	68.07M	1868.5G	78.34
S: DeepLabV3-Res18	73.20
SKD [20]	73.87
CWD [22]	75.93
MGD†[15]	13.6M	572.0G	76.02
MGD [15]	76.31
Af-DCD†	76.44
Af-DCD	76.52
S: PSPNet-Res18	69.85
SKD [20]	72.70
CWD [22]	73.53
MGD†[15]	12.9M	507.4G	73.63
MGD [15]	74.10
Af-DCD†	73.92
Af-DCD	74.22
4.1 Datasets and Experimental Setups.
Datasets. Five popular semantic segmentation datasets, including Cityscapes [27], Pascal VOC [28], Camvid [29], ADE20K [30] and COCO-Stuff-164K [31], are used in our experiments. We conduct our ablation studies on both Cityscapes and ADE20K, which help us to ensure the best setting in most experiments and analyse the effectiveness of our design. Details for these five datasets are described in supplemental material.
Experimental Settings. Following general settings [9, 20, 15] in semantic segmentation distillation, we adopt DeeplabV3 [32] and PSPNet [3] for segmentation framework, ResNet-18 [33] and MobileNetv2 [34] for student backbones, Resnet-101 for teacher backbone and group various teacher-student pairs. In order to make fair comparison with different state-of-the-art methods, we implement our method on both MMSegmentation codebase [35] and CIRKD codebase [9]. In training and evaluation, we use mean Intersection-over-Union (mIoU) to measure the performance of all methods. In training phase, all models are optimized by SGD with the momentum of 0.9, the initial learning rate of 0.02, and the batch size of 16. The input size is  $512 \times 1024$ ,  $400 \times 400$ ,  $512 \times 1024$ ,  $512 \times 1024$ , for experiments on Pascal VOC, CamVid, ADE20K and COCO-Stuff-164K, respectively. The input size for experiments on Cityscapes are different in the two codebase,  $512 \times 1024$  in CIRKD codebase and  $512 \times 512$  in MMSegmentation codebase [15]. In evaluation phase, we follow general settings in [22], which evaluate the performance with the original image size. Our masked reconstruction generator consists of two  $3 \times 3$  convolutional layers with ReLU, following [15]. Other default hyper-parameter settings and implementation details are described in supplemental material.
In this part, we intend to compare the distillation performance of our method Af-DCD with recent state-of-the-art methods for semantic segmentation. Aiming for comprehensive comparison, we conduct experiments on the aforementioned five public datasets following general settings.
Results on Cityscapes. In Table 1, we conduct experiments on the most popular Cityscapes dataset to validate the generalization ability of our method to different teacher-student network pairs. The experimental results show that Af-DCD outperforms state-of-the-art methods in most cases, with the maximal margin of  $1.12\%$ . In average, Af-DCD brings  $3.44\%$  gain to the baseline student models, with the maximal gain of  $4.37\%$ . From the results shown in Table 1a, we can see that our method can well handle teacher-student network pairs in which students (e.g., DeepLabV3-Res18 and DeepLabV3-MBV2) have the same segmentation framework but with different backbones. The
Table 2: Performance comparison of Af-DCD and recent state-of-the-art distillation methods on the other four datasets. We follow the way in CIRKD [9] to calculate FLOPs. Best results are bolded.
(a) Pascal VOC
Method	Params (M)	FLOPs (G)	Val mIOU (%)
T: DeepLabV3-Res101	61.1M	1294.6G	77.67
S: DeepLabV3-Res18	13.6M	305.0G	73.21
SKD [20]	73.51
IFVD [23]	73.85
CWD [22]	74.02
CIRKD [9]	74.50
Af-DCD	76.25
S: PSPNet-Res18	12.9M	260.0G	73.33
SKD [20]	74.07
IFVD [23]	73.54
CWD [22]	73.99
CIRKD [9]	74.78
Af-DCD	76.14
(b) Camvid
Method	Params (M)	FLOPs (G)	Test mIOU (%)
T: DeepLabV3-Res101	61.1M	280.2G	69.84
S: DeepLabV3-Res18	13.6M	61.0G	66.92
SKD [20]	67.46
IFVD [23]	67.28
CWD [22]	67.71
CIRKD [9]	68.21
Af-DCD	69.27
S: PSPNet-Res18	12.9M	45.6G	66.73
SKD [20]	67.83
IFVD [23]	67.61
CWD [22]	67.92
CIRKD [9]	68.65
Af-DCD	69.48
(c) ADE20K
Method	Params (M)	FLOPs (G)	Val mIOU (%)
T: DeepLabV3-Res101	61.1M	1294.6G	42.70
S: DeepLabV3-Res18	13.6M	305.0G	33.91
CIRKD [9]	35.41
Af-DCD	36.21
(d) COCO-Stuff-164K
Method	Params (M)	FLOPs (G)	Val mIOU (%)
T: DeepLabV3-Res101	61.1M	1294.6G	38.71
S: DeepLabV3-Res18	13.6M	305.0G	32.60
CIRKD [9]	33.11
Af-DCD	34.02
results of Table 1b further show that our method can also generalize well to teacher-student network pairs in which students (e.g., DeepLabV3-Res18 and PSPNet-Res18) have different segmentation frameworks but with the same backbone.
Results on Four Other Datasets. In Table 2, we evaluate the performance of Af-DCD on four other datasets including PASCAL VOC, Camvid, ADE20K and COCO-Stuff-164K to examine the generalization of our design to handle different semantic segmentation tasks. According to the results shown in Table 2a-2d, Af-DCD outperforms state-of-the-art methods by significant margins on these four datasets, with the maximal and average margin of  $1.75\%$  and  $1.22\%$ , respectively. Furthermore, we can observe that our method consistently shows significant absolute mIOU gains (from  $1.42\%$  to  $3.04\%$ ) to different student models on small-size (Camvid), medium-size (Cityscapes and PASCAL VOC) and large-size (ADE20K and COCO-Stuff-164K) datasets.
Ablation Study on Different Loss Terms. In our formulation, the overall loss contains three distillation terms, including  $L_{kd}$ ,  $L_{fd}$  and  $L_{Af - DCD}$ . Accordingly, we conduct experiments that independently test the gain from each term to explore the nature of Af-DCD. From the results shown in Table 3, we can get following observations: (i) The accuracy gain of Baseline +  $L_{Af - DCD}^{OC}$  is slight on relatively small dataset Cityscapes (0.28% mIOU gain), but it is notably pronounced on much larger dataset ADE20K (1.50% mIOU gain); (ii) Compared to Baseline +  $L_{fd}$ , Baseline +  $L_{Af - DCD}^{OC}$  gets student model with better accuracy on ADE20K dataset, while maintaining almost the same training efficiency; (iii) Baseline +  $L_{fd}$  +  $L_{Af - DCD}^{OC}$  gets student models with 76.44% mIOU and 36.01% mIOU on Cityscapes dataset and ADE20K dataset, respectively, which are obviously better than both Baseline +  $L_{fd}$  and Baseline +  $L_{Af - DCD}^{OC}$ , showing that two loss terms  $L_{fd}$  and  $L_{Af - DCD}^{OC}$  are complementary; (vi)  $L_{kd}$  can further bring minor extra gains, 0.08% and 0.2%, to Baseline +  $L_{fd}$  +  $L_{Af - DCD}^{OC}$  on Cityscapes and ADE20K dataset, respectively.
Ablation Study on Different Designs. In this study, we conduct a set of experiments on our three basic contrasting designs, namely Channel Contrasting (CC), Spatial Contrasting (SC) and Omni-Contrasting (OC), in order to exploit the gains from different contrasting dimensions and further verify the necessity of our OC. In Table 3a and 3b, we can observe two phenomena:
Table 3: Ablation studies on loss terms and their different combinations. The experiments for Cityscapes and ADE20K are conducted on the first teacher-student network pair in Table 1b and 2c, respectively. The training time is measured on 8 NVIDIA RTX A5000 GPUs with 40000 iterations.
(a) Ablation on Cityscapes
Method	mIOU (%)	ΔmIOU (%)	\( {T}_{train}\left( \mathrm{\;h}\right) \)
Baseline	73.20	n/a	n/a
\( + {L}_{fd} \)	75.88	+2.68	4.02
\( + {L}_{Af - {DCD}}^{OC} \)	73.48	+0.28	4.06
\( + {L}_{fd} + {L}_{Af - {DCD}}^{OC} \)	76.44	+3.24	4.25
\( + {L}_{fd} + {L}_{kd} \)	76.04	+2.84	4.05
\( + {L}_{fd} + {L}_{kd} + {L}_{Af - {DCD}}^{OC} \)	76.52	+3.32	4.27
Baseline	73.20	n/a	n/a
\( + {L}_{fd} + {L}_{Af - {DCD}}^{CC} \)	76.23	+3.03	4.13
\( + {L}_{fd} + {L}_{AF - {DCD}}^{SC} \)	76.26	+3.06	4.18
\( + {L}_{fd} + {L}_{Af - {DCD}}^{CC} + {L}_{Af - {DCD}}^{SC} \)	76.33	+3.13	4.29
\( + {L}_{fd} + {L}_{Af - {DCD}}^{OC} \)	76.44	+3.24	4.25
(b) Ablation on ADE20K
Method	mIOU (%)	ΔmIOU (%)	\( {T}_{train}\left( \mathrm{\;h}\right) \)
Baseline	33.91	n/a	n/a
\( + {L}_{fd} \)	34.92	+1.01	4.32
\( + {L}_{Af - {DCD}}^{OC} \)	35.41	+1.50	4.35
\( + {L}_{fd} + {L}_{Af - {DCD}}^{OC} \)	36.01	+2.10	4.48
\( + {L}_{fd} + {L}_{kd} \)	35.22	+1.31	4.34
\( + {L}_{fd} + {L}_{kd} + {L}_{Af - {DCD}}^{OC} \)	36.21	+2.30	4.51
Baseline	33.91	n/a	n/a
\( + {L}_{fd} + {L}_{Af - {DCD}}^{CC} \)	35.72	+1.81	4.41
\( + {L}_{fd} + {L}_{Af - {DCD}}^{SC} \)	35.22	+1.31	4.45
\( + {L}_{fd} + {L}_{Af - {DCD}}^{SC} + {L}_{Af - {DCD}}^{SC} \)	35.81	+1.90	4.54
\( + {L}_{fd} + {L}_{Af - {DCD}}^{OC} \)	36.01	+2.10	4.48
(a) Ablation on  $\tau$
(b) Ablation on  $\lambda_{3}$
Figure 3: Ablation studies on major hyper-parameters. In (d),  $4^{*}$  denotes  $N = 4$  with  $4 \times 4$  max pooling, which can further enhance training efficiency. Best viewed with zoom in.
(c) Ablation on  $M$
(d) Ablation on  $N$
(e) Ablation on  $\zeta$
(i) CC has similar gain to baseline model as SC on Cityscapes, while has much stronger performance on ADE20K, exceeding  $0.5\%$  to SC; (ii) The combination of CC and SC performs better than CC and SC, but worse than OC on both datasets (in distillation accuracy and training speed). The above two observations show the superiority of OC, which comes from the neat combination of CC and SC. Specifically, OC groups pixels into a number of disjoint local patches and tactfully leverages CC and SC within each local patch instead of the holistic feature maps to better exploit dense and structured local information for contrastive feature mimicking.
Table 4: Ablation study on the choice of the function  $d$  in the Omni-Contrasting loss. The experimental setups are the same to Table 1 (default setting without  $L_{kd}$ ). Best results are bolded.
Function d in Formula 7	Dataset	mIOU (%)	ΔmIOU (%)
Baseline	73.20	n/a
L1-normed distance	CityScapes	75.97	+2.77
Cosine similarity	76.10	+2.90
L2-normed distance	76.44	+3.24
Baseline	33.91	n/a
L1-normed distance	ADE20K	35.82	+1.91
Cosine similarity	35.95	+2.04
L2-normed distance	36.01	+2.10
Ablation Study on Choice of the Function  $d$  in the Omni-Contrasting Loss. We compare Formula 7 of our method with 3 types of the function  $d$  including  $L_{2}$ -normed distance (our choice), cosine similarity (common choice in contrastive loss) and  $L_{1}$ -normed distance. From the results summarized in Table 4, we can see that our method always shows significant mIOU gains to the baseline with all 3 types of the function  $d$ . Comparatively, our method with  $L_{2}$ -normed distance is the best, which supports our intuition that improved performance would be attained by choosing the same type of the function  $d$  for the feature distillation loss (Formula 3) and the Omni-Contrasting loss (Formula 7).
Ablation Study on Training Efficiency. Our Af-DCD is naturally augmentation-free and memory-buffer-free. We evaluate the efficiency of our design in terms of training time and GPU memory occupation. In Table 3a, we can observe that Af-DCD introduces minor extra training cost, from 4.02 hours to 4.25 hours, which only increases  $5.7\%$  training time to feature distillation. From the results shown in Table 5, we can see Af-DCD uses less memory and training time, but achieves better performance, compared to CIRKD.
Table 5: Comparison of training resources used by CIRKD and Af-DCD, under the settings of the first teacher-student network pair in Table 1a. Best results are bolded.
Method	mIOU (%)	GPU memory (G)	\( {T}_{train}\left( \mathrm{\;h}\right) \)
CIRKD	76.38	10.09	4.34
Af-DCD	77.03	7.94	4.23
Ablation Studies on Major Hyper-parameters. Recall that our method has five hyper-parameters, we also perform experiments to study them. As shown in Figure 3, the effectiveness of our method is relative stable when values of these hyper-parameters are changing. Notably, we find using max
(a) Feature distance between T and S
(b) Feature distance within local areas
Object Boundary
Object Occlusion
Small Object
(c) Heat maps of areas where model using Af-DCD segments correctly but using FD segments wrong
Image Boundary
Difficult Category
Rare View
Figure 4: Verification experiments. We leverage the models in Table 3b, and count 131M fine-grained representations from 2000 samples. In (a), we randomly select 10K fine-grained representations and measure the distances between student and teacher. In (b), we measure the distances among fine-grained representations within  $4 \times 4$  local areas of a specific sample, and randomly select 10K to calculate the probability density. In (c), we choose several difficult cases and highlight areas that  $L_{Af - DCD}$  can help  $L_{fd}$  segment correctly.
pooling with proper scale has no obvious effect on model performance but it has better efficiency than larger  $N$ . Other detailed analysis is referred to supplementary materials.
4.4 Discussion
After demonstrating the superior performance of Af-DCD in Section 4.2 and analysing the gain of Af-DCD in Section 4.3, we perform some verification experiments, aiming to discuss how  $L_{Af - DCD}$  boosts the distillation performance of  $L_{task} + L_{fd}$  (denoted as FD for abbreviation).
As shown in Figure 4b, the self-similarity distribution of fine-grained representations projected by FD has large differences from that of teacher's feature representations. In order to alleviate the aforementioned problem,  $L_{Af - DCD}$  contrasts feature partitions within each local patch between student and teacher. The results in Figure 4a and Figure 4b are summarized as below:
(1)  $L_{Af - DCD}$  dramatically decreases the distances of student to teacher in views of fine-gained representations and self-similarities, which proves our basic assumption that  $L_{Af - DCD}$  can further make student approach teacher in the micro and fine-grained views;
(2)  $L_{Af - DCD}$  increases both the mean and variance of the self-similarity distribution of student. Such observation verifies that  $L_{Af - DCD}$  can force student to learn dense and structured knowledge implicitly contained among teacher's feature partitions.
The above two improvements are proved to be effective in addressing segmenting various difficult scenarios in semantic segmentation, such as object boundary, small object, object occlusion, difficult category and rare view. Heat maps shown in Figure 4c illustrate that  $L_{Af - DCD}$  can effectively help FD correctly classify these difficult pixels, which further verifies the superiority of Af-DCD.
More examples and detailed analysis are referred to supplementary material.
In this paper, we present Af-DCD, an augmentation-free dense contrastive distillation method tailored to semantic segmentation. The dense contrasting in Af-DCD is an omni-dimensional design. Thus, it can effectively transfer teacher's contextual and positional channel-group information to student. Experimental results show that Af-DCD is effective on different teacher-student network pairs and datasets while maintaining training efficiency, as it is born with augmentation-free and memory-buffer free. We hope our work can inspire researchers to explore more powerful and efficient contrastive distillation methods in the future.
[21] Tong He, Chunhua Shen, Zhi Tian, Dong Gong, Changming Sun, and Youliang Yan. Knowledge adaptation for efficient semantic segmentation. In CVPR, 2019.
[22] Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen. Channel-wise knowledge distillation for dense prediction. In ICCV, 2021.
[23] Yukang Wang, Wei Zhou, Tao Jiang, Xiang Bai, and Yongchao Xu. Intra-class feature variation distillation for semantic segmentation. In ECCV, 2020.
[24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[25] Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation. arXiv preprint arXiv:2101.04731, 2021.
[26] Jinguo Zhu, Shixiang Tang, Dapeng Chen, Shijie Yu, Yakun Liu, Mingzhe Rong, Aijun Yang, and Xiaohua Wang. Complementary relation contrastive distillation. In CVPR, 2021.
[27] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[28] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010.
[29] Gabriel J Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recognition using structure from motion point clouds. In ECCV, 2008.
[30] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019.
[31] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018.
[32] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[34] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. *Mobilenetv2: Inverted residuals and linear bottlenecks*. In *CVPR*, 2018.
[35] MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation, 2020.