SlowGuess/NeurIPS_2023_OCR huggingface dataset:
3.3 Discussion of IDRNet
The procedure of deletion diagnostics looks a lot like Dropout [61], while they are totally different mechanisms. Dropout is a mechanism to prevent deep neural networks from overfitting, while our approach introduces deletion diagnostics to help model pixel relations. In addition, our work also relates closely to some recent segmentation approaches, which extract semantic-level representations to augment pixel representations, e.g., OCRNet [79] and ISNet [40]. Whereas, these methods only use feature similarities to make connections between pixels, which is distinct from our paradigm as it is intervention-driven and without masses of predetermined priors.
Our paradigm simplifies pixel relations modeling as object relations modeling. Benefiting from this simplification, deletion diagnostics enables the network to directly utilize its objective function to examine whether one class  $i$  can help recognize another class  $j$ . The key idea behind this mechanism is that loss value of class  $j$  should increase after deleting class  $i$  if class  $i$  can facilitate the prediction
Table 2: Performance improvements on various benchmark datasets with different segmentation frameworks after leveraging our intervention-driven paradigm to model pixel relations.
Method	Backbone	Stride	ADE20K (train / val)	COCO-Stuff (train / test)	Cityscapes (train / val)	LIP (train / val)
FCN [51]	ResNet-50	8×	36.96	31.76	75.16	48.63
FCN+IDRNet (ours)	ResNet-50	8×	43.61 (+6.65)	38.64 (+6.88)	79.91 (+4.75)	51.24 (+2.61)
PSPNet [83]	ResNet-50	8×	42.64	37.40	79.05	51.94
PSPNet+IDRNet (ours)	ResNet-50	8×	44.02 (+1.38)	39.13 (+1.73)	79.88 (+0.83)	53.29(+1.35)
DeeplabV3 [8]	ResNet-50	8×	43.19	37.63	79.62	52.35
DeeplabV3+IDRNet (ours)	ResNet-50	8×	44.75 (+1.56)	39.31 (+1.68)	80.69 (+1.07)	53.87 (+1.52)
UPerNet [70]	ResNet-50	8×	43.02	37.65	79.08	52.95
UPerNet+IDRNet (ours)	ResNet-50	8×	44.84 (+1.82)	39.35 (+1.70)	80.81 (+1.73)	54.00 (+1.05)
Table 3: Ablation study on the design of IDRNet. IE-Orthogonal: feature enhancement with orthogonal matrix. IE-DL: feature enhancement with dataset-level representations. BD: balanced deletion in deletion diagnostics. RD: random deletion in deletion diagnostics.
IE-Orthogonal	IE-DL	Mrmean	Mrvar	BD	RD	ADE20K (train / val)	PASCAL-Context (train / val)	COCO-Stuff (train / test)
X	X	X	X	X	X	36.96	45.48	31.76
X	X	✓	✓	✓	X	42.82	52.65	37.79
X	✓	✓	✓	✓	X	43.61	52.97	38.64
✓	X	✓	✓	✓	X	42.99	52.80	38.86
X	✓	X	✓	✓	X	42.74	52.86	38.06
X	✓	✓	X	✓	X	43.13	52.66	37.89
X	✓	✓	✓	X	✓	42.90	52.92	37.59
of class  $j$ , and the increased value can be adopted to measure the importance of class  $i$  to  $j$ . Therefore, there is no potential predetermined priors bias in the process of deletion diagnostics. As a comparison, the geometric transformations of multi-scale-driven context schemes are manually set, which tends to aggregate some ineffective information and lacks of generalization. Similarity-driven context schemes are assumed to aggregate semantically similar pixel representations, which results in ignoring other dissimilar but effective pixel representations for building co-occurrent patterns, e.g., sky pixels for airplane pixels. In a nutshell, our intervention-driven paradigm is more effective compared to the existing similarity-/multi-scale-driven context modules.
Benchmark Datasets. Our approach is validated on five popular semantic segmentation benchmark datasets, including ADE20K [87], COCO-Stuff [4], Cityscapes [17], LIP [29] and PASCAL-Context [22]. In detail, ADE20K is one of the most well-known datasets for scene parsing, which contains 150 stuff/object category labels. There are  $20\mathrm{K} / 2\mathrm{K} / 3\mathrm{K}$  images for training, validation and test set, respectively in the dataset. COCO-Stuff is a challenging dataset which provides rich annotations for 91 thing classes and 91 stuff classes. It consists of  $9\mathrm{K} / 1\mathrm{K}$  images in the training and test sets. Cityscapes has  $5\mathrm{K}$  high-resolution annotated urban scene images, with 2,975/500/1,524 images for training/validation/testing. It covers 19 challenging categories, such as traffic sign, vegetation and rider. LIP mainly focuses on single human parsing and contains  $50\mathrm{K}$  images with 19 semantic human part classes and 1 background class. Its training, validation and test sets separately involve  $30\mathrm{K} / 10\mathrm{K} / 10\mathrm{K}$  images. PASCAL-Context involves 59 foreground classes and a background class for scene parsing. The dataset is divided into 4,998 and 5,105 images for training and validation.
Implementation Details. Our algorithm is implemented in PyTorch [54] and SSSegmentation [38]. For backbone networks (i.e., encoders), they are all pretrained on ImageNet dataset [59]. Remaining layers (i.e., decoders) are initialized by the default initialization strategy in PyTorch. As for the data augmentation, some common data augmentation techniques are utilized, including random horizontal flipping, random cropping, color jitter and random scaling (from 0.5 to 2). The learning rate is scheduled by polynomial annealing policy with factor  $(1 - \frac{iter}{total\_iter})^{0.9}$ .
Specific to ADE20K, we set learning rate, crop size, batch size and training epochs as 0.01,  $512 \times 512$ , 16 and 130, respectively. Specific to COCO-Stuff, learning rate, crop size, batch size and training epochs are set as 0.001,  $512 \times 512$ , 16 and 110, respectively. As for LIP, we set learning rate, crop size, batch size and training epochs as 0.01,  $473 \times 473$ , 32 and 150, respectively. As for Cityscapes, learning rate, crop size, batch size and training epochs are set as 0.01,  $512 \times 1024$ , 8 and 220, respectively. Specific to PASCAL-Context, we set learning rate, crop size, batch size and training epochs as 0.004,  $480 \times 480$ , 16 and 260, respectively.
Figure 3: Qualitative comparison with FCN, OCRNet, ISNet and MCIBI++ on Cityscapes dataset.
Figure 4: Compare category sampling frequency of balanced deletion (BD) and random deletion (RD) on Cityscapes (left) and LIP (right) benchmark dataset.
Table 4: Performance comparison between utilizing back propagation algorithm (i.e., BP-driven  $\mathcal{M}_r$ ) and using our deletion diagnostics (i.e., DD-driven  $\mathcal{M}_r$ ) to update the relation matrix  $\mathcal{M}_r$ .
Update strategy of \( {\mathcal{M}}_{r} \)	Backbone	mIoU on ADE20K
FCN	ResNet-50	36.96
FCN + BP-driven \( {\mathcal{M}}_{r} \)	ResNet-50	40.35
FCN + DD-driven \( {\mathcal{M}}_{r} \)	ResNet-50	43.61
Evaluation Metric. Following conventions [7, 51], mean intersection-over-union (mIoU) is utilized for evaluation.
Performance Improvements. As seen in Table 2, it is observed that introducing deletion diagnostics to help pixel relationship building can bring consistent performance improvements on different segmentation approaches (i.e., FCN, PSPNet, DeeplabV3 and UPerNet) and benchmark datasets (i.e., ADE20K, COCO-Stuff, Cityscapes and LIP). By way of illustration, our method boosts FCN, PSPNet, DeeplabV3 and UPerNet by  $2.61\%$ ,  $1.35\%$ ,  $1.52\%$ ,  $1.05\%$  mIoU on LIP dataset, respectively.
Feature Enhancement. Feature enhancement is designed to make pixel representations with different classes more discriminative to help model relationship between semantic-level representations  $R_{sl}$ . As indicated in Table 3, we can observe that yielding random orthogonal matrix (IE-Orthogonal) and generating dataset-level representations (IE-DL) for feature enhancement are both promising strategies to boost segmentation performance. Specifically, IE-Orthogonal brings  $0.17\%$ ,  $0.15\%$  and  $1.07\%$  mIoU gains for ADE20K, PASCAL-Context and COCO-Stuff, respectively. IE-DL achieves  $0.79\%$ ,  $0.32\%$  and  $0.85\%$  mIoU improvements for the three datasets, respectively.
Relation Matrix. We also validate the effectiveness of the proposed relation matrix in Table 3. It is observed that  $\mathcal{M}_{r_{mean}}$  and  $\mathcal{M}_{r_{var}}$  both contribute to the performance improvements. In detail,  $\mathcal{M}_{r_{mean}}$  makes our segmentor outperform baseline models (first row) by  $6.17\%$ ,  $7.18\%$  and  $6.13\%$  on ADE20K, PASCAL-Context and COCO-Stuff, respectively.  $\mathcal{M}_{r_{var}}$  boosts baseline models by  $5.78\%$ ,  $7.38\%$  and  $6.30\%$  on the three datasets, respectively. The combination of  $\mathcal{M}_{r_{mean}}$  and  $\mathcal{M}_{r_{var}}$  finally yields  $43.61\%$ ,  $52.97\%$  and  $38.64\%$  mIoU on the three datasets, respectively.
Deletion Diagnostics. In Table 4, we compare our proposed deletion diagnostics procedure (DD-driven  $\mathcal{M}_r$ ) with back-propagation updating strategy (BP-driven  $\mathcal{M}_r$ ). We can observe that adopting DD-driven  $\mathcal{M}_r$  outperforms using BP-driven  $\mathcal{M}_r$  by  $3.26\%$  mIoU on ADE20k. The result suggests that our deletion diagnostics strategy is more effective than the back-propagation updating strategy.
Balanced Deletion. Balanced deletion is utilized to help improve the being-diagnosed probabilities of uncommon categories in training dataset and thereby, deletion diagnostics can be performed evenly. In Table 3, we can observe that introducing the balanced deletion strategy can help improve the segmentation performance by  $0.71\%$ ,  $0.05\%$  and  $1.05\%$  for ADE20K, PASCAL-Context and COCO-Stuff, respectively. Also, we count and compare the sampling frequency of each category when using balanced deletion (BD) and random deletion (RD) in Figure 4. We can observe that the sampling frequency of uncommon categories have increased after utilizing BD as we expected.
Complexity Comparison. We also present analysis on complexity of our intervention-driven context module following the settings of OCR [79]. In Table 5, it is observed that our IDRNet requires the least Parameters, FLOPS and GPU Memory, while achieving the best mIoU on the ADE20k dataset. Moreover, our IDRNet is complementary to other context modules. For example, when incorporated with PPM (termed "PPM+IDRNet" in the table), the Parameters, FLOPS, Time and GPU Memory only increase by  $0.58\mathrm{M}$ ,  $39.78\mathrm{G}$ ,  $11.19\mathrm{ms}$  and  $73.65\mathrm{M}$ , respectively, which shows that our proposed IDRNet is light and efficient during network inference. Note that few parameters
Table 5: Complexity comparison with existing context modules on a single RTX 3090 Ti GPU. The input feature map is with size  $[1 \times 2048 \times 128 \times 128]$ . Excepted for mIoU column, all numbers are the smaller the better.
Context Module	Parameters	FLOPS	Time	GPU Memory	mIoU on ADE20K (%)
OCR [79]	15.12M	242.48G	16.58ms	617.24M	42.47
ASPP [8]	42.21M	674.47G	41.98ms	976.06M	43.19
PPM [83]	23.07M	309.45G	21.45ms	960.63M	42.64
UPerNet [70]	34.75M	500.76G	36.51ms	1429.18M	43.02
ANN [89]	22.42M	369.62G	26.58ms	1445.75M	41.75
CCNet [37]	23.92M	397.38G	30.92ms	986.28M	42.48
DNL [75]	24.12M	395.25G	51.38ms	2381.04M	43.50
IDRNet (ours)	10.79M	155.89G	20.52ms	365.66M	43.61
PPM+IDRNet (ours)	23.65M	349.23G	32.64ms	1034.28M	44.02
Table 6: Compare segmentation performance in cross domain.
Method	Urban Scene Parsing (Cityscapes train)	Human Parsing (LIP train)
Dark Zurich val	Nighttime Driving test	CIHP val
FCN [51]	10.66	17.90	27.20
FCN+IDRNet (ours)	12.55 (+1.89)	21.33 (+3.43)	28.59 (+1.39)
DeeplabV3 [8]	10.03	21.91	27.02
DeeplabV3+IDRNet (ours)	13.66 (+3.63)	23.85 (+1.94)	27.93 (+0.91)
UPerNet [83]	11.06	17.67	26.84
UPerNet+IDRNet (ours)	11.42 (+0.36)	21.13 (+3.46)	27.46 (+0.62)
increase in PPM+IDR as we adopt a shared  $3 \times 3$  convolution layer to reduce the feature channels of the backbone outputs.
Qualitative Results. Figure 3 provides qualitative comparison of the proposed approach against FCN [51], OCRNet [79], ISNet [40] and MCIBI++[41] on representative examples in various benchmark datasets. It is observed that IDRNet can output better segmentation results.
4.3 Cross Domain Segmentation Performance.
Table 6 compares the cross-domain segmentation performance. It can be observed that IDRNet can also boost cross-domain segmentation performance, which demonstrates the generalization of our intervention-driven pixel relation modeling paradigm. For example, DeeplabV3+IDRNet trained on day-time domain (i.e., Cityscapes) can also bring  $3.63\%$  and  $1.94\%$  mIoU gains to Dark Zurich [60] and Nighttime Driving [18] datasets which belong to night-time domain. In addition, after training FCN+IDRNet on single-human parsing scenarios (i.e., LIP), it also boosts the segmentation performance on multi-human parsing scenarios (i.e., CIHP [28]) by  $1.39\%$  mIoU.
4.4 Comparison with State-of-the-art Methods
Table 7 compares the quantitative results on four challenging benchmark datasets, including ADE20K val, LIP val, PASCAL-Context val and COCO-Stuff test. As for ADE20K val, it is observed that prior to this paper, Mask2Former [12] with Swin-Large backbone [41] achieves the state-of-the-art performance hitting  $57.30\%$  mIoU. By incorporating Mask2Former with IDRNet, it yields  $58.22\%$  mIoU which surpasses other state-of-the-art segmentation methods. Specific to LIP val, we can observe that among the previous SOTA methods, UPerNet+MCIBI++ with Swin-Large backbone obtains the state-of-the-art result, i.e.,  $59.91\%$  mIoU. Our method, UPerNet+IDRNet with Swin-Large backbone, outperforms it by  $1.26\%$  and reports the new SOTA result,  $61.17\%$  mIoU. With regard to PASCAL-Context val, our method, UPerNet+IDRNet with Swin-Large backbone, surpasses all the competitors, i.e.,  $0.50\%$  over UPerNet-MCIBI++,  $5.51\%$  over Segmenter,  $8.68\%$  over SETR,  $9.71\%$  over OCRNet, to name a few. On the subject of COCO-Stuff test, we can observe that our UPerNet+IDRNet with Swin-Large backbone reaches  $50.50\%$  mIoU which is  $0.23\%$  higher than previous state-of-the-art method UPerNet+MCIBI++.
4.5 Extend Deletion Diagnostics to Object Detection
In this section, we apply deletion diagnostics mechanism to the object detection frameworks to further demonstrate the effectiveness and generalization of our method.
Implementation Details. Our experiments are conducted on MS COCO dataset [48]. All algorithm implementations are based on PyTorch [54] and MMDetection [6], and we follow the default settings in MMDetection for model training and testing.
Table 7: State-of-the-art results on ADE20K, COCO-Stuff, LIP and PASCAL-Context dataset.
Method	Backbone	Stride	ADE20K (train / val)	COCO-Stuff (train / test)	PASCAL-Context (train / val)	LIP (train / val)
PSPNet [83]	ResNet-101	8×	43.29	-	47.80	-
EMANet [46]	ResNet-101	8×	-	39.90	53.10	-
OCNet [80]	ResNet-101	8×	45.45	-	-	54.72
ACNet [35]	ResNet-101	8×	45.90	40.10	54.10	-
APCNet [31]	ResNet-101	8×	45.38	-	54.70	-
OCRNet [79]	ResNet-101	8×	45.28	39.50	54.80	55.60
ISNet [40]	ResNet-101	8×	47.31	41.60	-	55.41
UPerNet+MCIBI++ [41]	ResNet-101	8×	47.93	41.84	56.82	56.32
OCRNet [79]	HRNetV2-W48	4×	45.66	40.50	56.20	56.65
SETR [86]	ViT-Large	16×	50.28	-	55.83	-
Segmenter [62]	ViT-Large	16×	53.63	-	59.00	-
UPerNet [70]	BEiT-Large	16×	57.00	-	-	-
UPerNet [70]	Swin-Large	32×	53.50	-	-	-
UPerNet+MCIBI++ [41]	Swin-Large	32×	54.52	50.27	64.01	59.91
MaskFormer [13]	Swin-Large	32×	55.60	-	-	-
Mask2Former [12]	Swin-Large	32×	57.30	-	-	-
K-Net+UPerNet [82]	Swin-Large	32×	54.30	-	-	-
SegNeXt-L [30]	MSCAN-Large	32×	52.10	47.20	60.30	-
Segformer [71]	MiT-B5	32×	51.80	46.70	-	-
UPerNet+IDRNet (ours)	Swin-Large	32×	54.67	50.50	64.51	61.17
Mask2Former+IDRNet (ours)	Swin-Large	32×	58.22	49.96	61.65	61.86
Table 8: Performance improvements on two popular object detection frameworks and MS COCO benchmark dataset after leveraging deletion diagnostics to build object relations.
Method	Backbone	Schedule	AP	\( AP_{50} \)	\( AP_{75} \)	\( AP_s \)	\( AP_m \)	\( AP_l \)
Faster R-CNN [56]	ResNet-50-FPN	1×	37.4	58.1	40.4	21.2	41.0	48.1
Faster R-CNN+IDRNet (ours)	ResNet-50-FPN	1×	38.2 (+0.8)	59.7	41.3	22.0	42.0	49.2
Mask R-CNN [32]	ResNet-50-FPN	1×	38.2	58.8	41.4	21.9	40.9	49.5
Mask R-CNN+IDRNet (ours)	ResNet-50-FPN	1×	39.7 (+1.5)	60.8	43.0	22.6	43.2	52.5
Experimental Results. As illustrated in Table 8, the proposed deletion diagnostics mechanism is integrated into two popular object detection frameworks, i.e., Faster R-CNN [56] and Mask R-CNN [32]. We can observe that intervention-driven relation building for objects brings  $0.8\%$  and  $1.5\%$  AP improvements for the two detection frameworks, respectively.
This work studies how to utilize deletion diagnostics to model pixel relationships. The key contribution consists of two parts: (1) we simplify pixel relation modeling as object relation modeling and (2) we introduce deletion diagnostics to make the networks build object relationships by perceiving changes in the outputs of the objective functions. Compared to similarity-/multi-scale-driven context modeling paradigms, there are fewer human priors in our context modeling paradigm. Experimental results demonstrate the effectiveness of our proposed paradigm qualitatively and quantitatively.
The work described in this paper was partially supported by grants from the National Key R&D Program of China (2022ZD0118101), the National Natural Science Fund (62201483) and the Research Grants Council of the Hong Kong Special Administrative Region, China (T45-401/22-N).
[1] H. Bao, L. Dong, S. Piao, and F. Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021.
[2] M. Berman, A. R. Triki, and M. B. Blaschko. The lovasz-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4413–4421, 2018.
[3] G. Bertasius, J. Shi, and L. Torresani. Semantic segmentation with boundary neural fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3602-3610, 2016.
[4] H. Caesar, J. Uijlings, and V. Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1209-1218, 2018.
[5] Y. Cao, J. Xu, S. Lin, F. Wei, and H. Hu. Gcnet: Non-local networks meet squeeze-excitation networks and beyond. In Proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0–0, 2019.
[86] S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6881-6890, 2021.
[87] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641, 2017.
[88] X. Zhu, H. Hu, S. Lin, and J. Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9308-9316, 2019.
[89] Z. Zhu, M. Xu, S. Bai, T. Huang, and X. Bai. Asymmetric non-local neural networks for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 593-602, 2019.