SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The performance on fine-grained mIoUs is relatively low.  $\mathrm{mIoU^D}$ ,  $\mathrm{mIoU^C}$  and  $\mathrm{mIoU^K}$  are all averaged on a per-class basis. Their values typically follow the ranking:  $\mathrm{mIoU^D} \geq \mathrm{mIoU^C} \geq \mathrm{mIoU^K}$ . The more fine-grained a metric is, the more challenging the problem it presents, since it is less merciful to individual mistakes. Nevertheless, we observe that the results for  $\mathrm{mIoU^C}$  and  $\mathrm{mIoU^K}$  are usually very similar. This observation indicates that  $\mathrm{mIoU^C}$  can serve as a reliable proxy for instance-wise metrics when instance-level labels are unavailable, e.g. PASCAL VOC/Context, etc.
Improvements in fine-grained mIoUs do not always extend to corresponding worst-case metrics. The worst-case metrics generally fall substantially below their corresponding mean metrics. For example, on PASCAL Context, the value of  $\mathrm{mIoU}^{\mathrm{C}}$  ranges from  $46.00\%$  to  $60.17\%$ , yet all models have  $0\% \mathrm{mIoU}^{\mathrm{C}^{1}}$ . The presence of extremely challenging examples poses obstacles to the application of neural networks in safety-critical tasks. It would be an interesting future research to explore ways of enhancing a model's performance on these hard examples.
Certain images consistently yield low image-level IoU values across various models. As presented in Table 13, most models agree on worst-case images across multiple datasets. Several challenging factors, such as difficult lighting conditions and the presence of small objects, can contribute to low IoU scores (as seen in Mapillary Vistas, depicted in Figures 19 - 21). Besides, mislabeling in the dataset may also cause models to produce an image-level IoU of 0 (see Appendix B.3).
DL3+: 98.51%
DL3:  $98.28\%$
UNet:  $45.77\%$
DL3+: 42.90%
DL3:  $9.48\%$
Figure 4: From left to right: ground-truth label, prediction and per-image IoU of class Truck with DeepLabV3+, DeepLabV3 and UNet using ResNet101 backbone, respectively. The class Truck is represented in dark blue and highlighted within white dashed circles.
UNet:  $51.76\%$
Table 3: Comparing DeepLabV3+, DeepLabV3 and UNet on Cityscapes using ResNet-101 backbone. Red: the best in a column. Green: the worst in a column.
Method	IoU D truck	mIoU D	IoU C truck	mIoU C	mIoU I
DeepLabV3+	84.94 ± 0.97	80.90 ± 0.16	55.26 ± 0.79	70.17 ± 0.16	76.21 ± 0.05
DeepLabV3	81.48 ± 1.43	80.07 ± 0.06	53.87 ± 0.94	69.09 ± 0.06	75.26 ± 0.03
UNet	65.58 ± 0.59	77.12 ± 0.17	53.00 ± 0.61	68.78 ± 0.09	75.36 ± 0.06
Image-level histograms vary across datasets for the same architecture, but are similar across architectures for the same dataset. The same architecture yields significantly different shapes of image-level histograms across various datasets. We believe it has to do with the coverage of a dataset (the fraction of classes appear in each image, discussed in Appendix B.2): a dataset with low coverage (e.g. ADE20K and COCO-Stuff) often results in a spread-out histogram with fat tails. On the other hand, although it is possible that a model that has a medium score on all images obtains the same mIoU<sup>1</sup> as a model that performs exceedingly well on some images and very poorly on others, we find that different architectures generally produce similar histograms for the same dataset.
Fine-grained mIoUs exhibit less bias towards large objects. For example, in Cityscapes, the class Truck has high dataset-level size imbalance (see Figure 4). As demonstrated in Table 3, compared with DeepLabV3-ResNet101, UNet-ResNet101 obtains a low value on  $\mathrm{IoU}_{\mathrm{Truck}}^{\mathrm{D}}$  (15% lower) and  $\mathrm{mIoU}^{\mathrm{D}}$  (3% lower), since it has a limited reception field and struggles with large objects such as trucks that are close to the camera. On the other hand, it is capable of identifying small objects. As a result, its performance on  $\mathrm{IoU}_{\mathrm{Truck}}^{\mathrm{C}}$ ,  $\mathrm{mIoU}^{\mathrm{C}}$  and  $\mathrm{mIoU}^{\mathrm{I}}$  is similar to that of DeepLabV3-ResNet101.
Architecture designs and loss functions are vital for optimizing fine-grained mIoUs. We have seen from previous findings that architecture designs play an important role in optimizing fine-grained mIoUs. The influence of loss functions is also significant. We leave discussions of architecture designs in section 5.2.1 and loss functions in section 5.2.2. These insights contribute to the development of best practices for optimizing fine-grained mIoUs.
5.2 Best Practices
In this section, we explore best practices for optimizing fine-grained mIoUs. In particular, we underscore two key design choices: (i) the aggregation of multi-level features is as vital as incorporating modules that increase the receptive field, and (ii) the alignment of the loss function with the evaluation metrics is paramount.
5.2.1 Architecture Designs
Our study reveals the importance of aggregating multi-scale feature maps for optimizing fine-grained metrics. DeepLabV3 and PSPNet seek to increase the receptive field through the ASPP module [6] and the PPM module [58], respectively. The large receptive field helps to capture large objects, resulting in high values on  $\mathrm{mIoU}^{\mathrm{D}}$ . However, these methods lack the fusion of multi-scale feature
maps and lose the ability to perceive multi-scale objects due to aggregated down-sampling. On the other hand, UNet does not incorporate a separate module to increase the receptive field and therefore underperforms on  $\mathrm{mIoU}^{\mathrm{D}}$ . Nevertheless, it performs comparably to DeepLabV3 and PSPNet on fine-grained metrics. The aggregation of multi-scale feature maps in UNet is critical for detecting small objects and achieving high values on fine-grained metrics<sup>3</sup>.
This observation is further confirmed by comparing DeepLabV3 with its successor, DeepLabV3+. The sole architectural difference between them is the incorporation of a feature aggregation branch in DeepLabV3+. Although they produce similar results on mIoU $^{\text{D}}$ , DeepLabV3+ typically outperforms DeepLabV3 on mIoU $^{\text{C}}$ .
It is encouraging to see that state-of-the-art methods, such as DeepLabV3+, UPerNet, and SegFormer, all incorporate modules to increase the receptive field and to aggregate multi-level feature maps. Such design choices align well with the objective of fine-grained mIoUs. We also advocate for the design of more specialized modules for perceiving small objects which can further improve the performance on fine-grained mIoUs.
5.2.2 Loss Functions
Aligning loss functions with evaluation metrics is a widely accepted practice in semantic segmentation [47, 2, 56, 14, 57, 50, 51]. The conventional approach involves combining the cross-entropy loss (CE) and IoU losses [2, 56, 50, 51], which extend discrete  $\mathrm{mIoU}^{\mathrm{D}}$  with continuous surrogates. With the introduction of fine-grained mIoUs, it is straightforward to adjust the losses to optimize for these new metrics. Specifically, we explore the Jaccard metric loss (JML) [50] and its variants for optimizing  $\mathrm{mIoU}^{\mathrm{I}}$  and  $\mathrm{mIoU}^{\mathrm{C}}$ . From a high-level, JML relaxes set counting (intersection and union) with simple  $L^1$  norm functions. More details of JML are in Appendix F.
In Table 4, we compare DeepLabV3+-ResNet101 trained with various loss functions on Cityscapes and ADE20K, respectively. Specifically, (D, I, C) represents the fraction of mIoU $^{\text{D}}$ , mIoU $^{\text{I}}$  and mIoU $^{\text{C}}$  in JML. For example, we consider the CE baseline:  $(0, 0, 0)$  and variants of JML to only optimize mIoU $^{\text{D}}$ :  $(1, 0, 0)$  and a uniform mixture of three:  $(\frac{1}{3}, \frac{1}{3}, \frac{1}{3})$ . The main takeaways are as follows:
The impact of loss functions is more pronounced when evaluated with fine-grained metrics. Specifically, we notice improvements on  $\mathrm{mIoU}^{\mathrm{C}}$  over CE reaching nearly  $3\%$  on Cityscapes and  $7\%$  on ADE20K. We also observe substantial increments on  $\mathrm{mIoU}^{\mathrm{I}}$ . Roughly speaking,  $\mathrm{Acc} \geq \mathrm{mIoU}^{\mathrm{D}} \geq \mathrm{mIoU}^{\mathrm{C}}$  in terms of class/size imbalance, while CE directly optimizes Acc. Therefore, it becomes more crucial to choose the correct loss functions when evaluated with fine-grained metrics.
There exist trade-offs among different JML variants. For example, compared with other JML variants, (1, 0, 0) achieves the highest mIoU $^{\text{D}}$  in both datasets, but have low values on mIoU $^{\text{I}}$  and mIoU $^{\text{C}}$ . We use (0, 0.5, 0.5) as the default setting in our benchmark, but alternate ratios may be considered depending on the evaluation metrics.
A notable observation is that CE surpasses (0, 1, 0) in terms of  $\mathrm{mIoU}^{\mathrm{D}}$  on Cityscapes. Interestingly, many segmentation codebases, including SMP [25] and MMSegmentation [8], calculate the Jaccard/Dice loss on a per-GPU basis. When training on GPUs with limited memory, the per-GPU batch size is often small. As a result, the loss function tends to mirror the behavior of (0, 1, 0), potentially leading to sub-optimal results as measured by  $\mathrm{mIoU}^{\mathrm{D}}$ .
Table 4: Comparing different loss functions using DeepLabV3+-ResNet101 on Cityscapes and ADE20K. (D, I, C) represents the fraction of mIoU $^{\text{D}}$ , mIoU $^{\text{I}}$  and mIoU $^{\text{C}}$  in JML, respectively. Red: the best in a column. Green: the worst in a column.
(D,I,C)	Cityscapes	ADE20K
mIoU\(^D\)	mIoU\(^I\)	mIoUC	mIoU\(^D\)	mIoU\(^I\)	mIoUC
(0,0,0)	80.67 ± 0.36	74.57 ± 0.09	67.61 ± 0.12	45.01 ± 0.13	53.73 ± 0.09	39.66 ± 0.13
(1,0,0)	81.22 ± 0.27	75.34 ± 0.09	68.97 ± 0.19	47.21 ± 0.26	57.74 ± 0.11	46.69 ± 0.25
(0,1,0)	80.29 ± 0.28	76.28 ± 0.02	69.97 ± 0.09	45.83 ± 0.19	58.37 ± 0.03	45.55 ± 0.25
(0,0,1)	80.75 ± 0.24	76.21 ± 0.08	70.49 ± 0.19	46.55 ± 0.54	58.74 ± 0.09	46.97 ± 0.23
(\frac{1}{3},\frac{1}{3},\frac{1}{3})	81.13 ± 0.10	76.19 ± 0.05	70.08 ± 0.15	46.92 ± 0.25	58.51 ± 0.02	46.77 ± 0.07
(0,\(\frac{1}{2}\),\(\frac{1}{2}\))	80.91 ± 0.17	76.21 ± 0.05	70.18 ± 0.16	46.32 ± 0.33	58.79 ± 0.04	46.52 ± 0.30
This might explain the efficacy of UNet for medical datasets. Typically, these datasets exhibit small lesion areas but lack extremely large objects that necessitate a large receptive field.
6.1 Limitation
We propose three fine-grained mIoUs:  $\mathrm{mIoU^I}$ ,  $\mathrm{mIoU^C}$ ,  $\mathrm{mIoU^K}$ , along with their associated worst-case metrics. These metrics bring significant advantages over traditional  $\mathrm{mIoU^D}$ . However, they are not without shortcomings, and users should be aware of these when employing them in research or applications. Specifically,  $\mathrm{mIoU^I}$  and  $\mathrm{mIoU^C}$  fall short when it comes to addressing instance-level imbalance. Meanwhile,  $\mathrm{mIoU^K}$  serves merely as an approximation of the per-instance metric, rather than an exact representation. Beides, conducting image-level analyses, such as image-level histograms and worst-case images, proves challenging with  $\mathrm{mIoU^C}$  and  $\mathrm{mIoU^K}$ . Although such capabilities are accommodated by  $\mathrm{mIoU^I}$ , it has an inherent bias towards classes that are more frequently represented across images.
Additionally, it is important to note that our conclusions are based on a specific training setting, including choices of optimizer, loss function, and other hyperparameters. There is potential for varied findings when altering this setting, especially when it comes to the choice of the loss function.
Finally, our experiments primarily focus on natural and aerial datasets, excluding medical datasets from our analysis. Besides, while our core emphasis is on addressing size imbalance in relation to IoU, the concept of calculating segmentation metrics in a fine-grained manner could be extended to other metrics, such as mAcc, the Dice score, the calibration error [19], etc. We plan to address these limitations in future work.
6.2 Suggestion
While  $\mathrm{mIoU}^{\mathrm{I}}$  might show a bias towards more common classes, it remains invaluable for users to analyze the dataset with image-level information, as well as provide a holistic evaluation of various segmentation methods. Given that  $\mathrm{mIoU}^{\mathrm{C}}$  is a good proxy of  $\mathrm{mIoU}^{\mathrm{K}}$ , and considering many datasets lack instance-level labels, we believe  $\mathrm{mIoU}^{\mathrm{C}}$  adequately serves as an instance-level approximation and could be used in most cases.
Regarding the worst-case metrics, we examine three variants:  $\mathrm{mIoU}^{\mathrm{C}^{\bar{q}}}$ ,  $\mathrm{mIoU}^{\mathrm{C}^{5}}$  and  $\mathrm{mIoU}^{\mathrm{C}^{1}}$ . We recommend the use of  $\mathrm{mIoU}^{\mathrm{C}^{\bar{q}}}$  in applications where worst-case outcomes are prioritized. We provide evaluations using both  $\mathrm{mIoU}^{\mathrm{C}^{5}}$  and  $\mathrm{mIoU}^{\mathrm{C}^{1}}$  for the sake of comprehensiveness, but believe that adopting either one is sufficient. While utilizing  $q = 1$  might be suitable for larger datasets like ADE20K to emphasize these exceptionally difficult examples, it could introduce significant variance for smaller datasets such as DeepGlobe Land. For the latter, a value of  $q = 5$  or even larger might be more appropriate. With these considerations, dataset organizers can choose the metric that best fits their context.
In conclusion, this study provides crucial insights into the limitations of traditional per-dataset mean intersection over union and advocates for the adoption of fine-grained mIoUs, as well as the corresponding worst-case metrics. We argue that these metrics provide a less biased and more comprehensive evaluation, which is vital given the inherent class and size imbalances found in many segmentation datasets. Besides, these fine-grained metrics furnish a wealth of statistical information that can guide practitioners in making more informed decisions when evaluating segmentation methods for specific applications. Furthermore, they offer valuable insights for model and dataset auditing, aiding in the rectification of corrupted labels.
Our extensive benchmark study, covering 12 varied natural and aerial datasets and featuring 15 modern neural network architectures, underscores the importance of not relying solely on a single metric and affirms the potential of fine-grained metrics to reduce the bias towards large objects. It also sheds light on the significant roles played by architectural designs and loss functions, leading to best practices in optimizing fine-grained metrics.
We acknowledge support from the Research Foundation - Flanders (FWO) through project numbers G0A1319N and S001421N, and funding from the Flemish Government under the Onderzoeksprogramma Artificièle Intelligentie (AI) Vlaanderen programme. The resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the FWO and the Flemish Government.
M.B.B. is entitled to stock options in Mona.health, a KU Leuven spinoff.
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer Normalization. *NeurIPS Deep Learning Symposium*, 2016.
[2] Maxim Berman, Amal Rannen Triki, and Matthew B. Blaschko. The Lovasz-softmax loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Union Measure in Neural Networks. CVPR, 2018.
[3] Patrick Bilic et al. The Liver Tumor Segmentation Benchmark (LiTS). MIA, 2023.
[4] Gabriel J Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and Recognition Using Structure from Motion Point Clouds. ECCV, 2008.
[5] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCO-Stuff: Thing and Stuff Classes in Context. CVPR, 2018.
[6] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking Atrous convolution for semantic image segmentation. arXiv, 2017.
[7] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with Atrous separable convolution for semantic image segmentation. ECCV, 2018.
[8] MMSegmentation Contributors. MMSegmentation: OpenMMLab Semantic Segmentation Toolbox and Benchmark, 2020.
[9] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. CVPR, 2016.
[10] Gabriela Csurka, Diane Larlus, and Florent Perronnin. What is a good evaluation measure for semantic segmentation? BMVC, 2013.
[11] Dengxin Dai and Luc Van Gool. Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime. ITSC, 2018.
[12] Ilke Demir, Krzysztof Koperski, David Lindenbaum, Guan Pang, Jing Huang, Saikat Basu, Forest Hughes, Devis Tuia, and Ramesh Raskar. DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images. CVPR Workshop, 2018.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR, 2021.
[14] Tom Eelbode, Jeroen Bertels, Maxim Berman, Dirk Vandermeulen, Frederik Maes, Raf Bisschops, and Matthew B. Blaschko. Optimization for medical image segmentation: Theory and Practice When Evaluating With Dice Score or Jaccard Index. TMI, 2020.
[15] Adam Van Etten, Dave Lindenbaum, and Todd M Bacastow. *SpaceNet: A Remote Sensing Dataset and Challenge Series.* arXiv, 2018.
[16] Mark Everingham, Luc van Gool, Chris Williams, John Winn, and Andrew Zisserman. The PASCAL Visual Object Classes Challenge 2007, 2007.
[17] Mark Everingham, Luc van Gool, Chris Williams, John Winn, and Andrew Zisserman. The PASCAL Visual Object Classes Challenge 2008, 2008.
[18] Mark Everingham, Luc Van Gool, Christopher K I Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes (VOC) Challenge. *IJCV*, 2009.
[19] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. ICML, 2017.
[20] Bharath Hariharan, Pablo Arbelaez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. ICCV, 2011.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
[22] Nicholas Heller et al. The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 challenge. MIA, 2021.
[23] Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units (GELUs). arXiv, 2016.
[24] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv, 2017.
[25] Pavel Iakubovskii. Segmentation models pytorch, 2019.
[26] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ICML, 2015.
[27] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollar. Panoptic Segmentation. CVPR, 2019.
[28] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. LISA: Reasoning Segmentation via Large Language Model. arXiv, 2023.
[29] Tsung-Yi Lin, Piotr Dálár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. CVPR, 2017.
[30] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C Lawrence Zitnick, and Piotr Dólar. Microsoft COCO: Common Objects in Context. ECCV, 2014.
[31] Chang Liu, Henghui Ding, and Xudong Jiang. GRES: Generalized Referring Expression Segmentation. CVPR, 2023.
[32] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. CVPR, 2022.
[33] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. *ICLR*, 2019.
[34] Lena Maier-Hein et al. Metrics Reloaded: Recommendations for image analysis validation. arXiv, 2023.
[35] Sachin Mehta and Mohammad Rastegari. MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer. ICLR, 2022.
[36] Sachin Mehta and Mohammad Rastegari. Separable Self-attention for Mobile Vision Transformers. TMLR, 2023.
[37] Bjoern H. Menze et al. The multimodal brain tumor image segmentation benchmark (BRATS). TMI, 2015.
1 Introduction 1
2 Fine-grained mIoUs 2
3 Worst-case Metrics 5
4 Advantages of Fine-grained mIoUs 5
5.1 Main Results 6
5.2 Best Practices 8
5.2.1 Architecture Designs 8
5.2.2 Loss Functions 9
6 Discussion 10
6.1 Limitation 10
6.2 Suggestion 10
7 Conclusion 10
A Models 15
A.1 Segmentation Methods 15
A.2Backbones 16
B Datasets 16
B.1 "Thing" and "Stuff" 17
B.2 Dataset Coverage 18
B.3 Dataset Auditing 18
C Implementation Details 19
D NULL in Fine-grained mIoUs 20
E Other Per-instance mIoUs 21
F The Jaccard Metric Loss 22
G Results 22
G.1 Cityscapes 23
G.2 Nighttime Driving 28
G.3 Dark Zurich 33
G.4 Mapillary Vistas 38
G.5 CamVid 43
G.6 ADE20K 48
G.7 COCO-Stuff 53
G.8 PASCAL VOC 58
G.9 PASCAL Context 63
G.10 DeepGlobe Land 68
G.11 DeepGlobe Road 73
G.12 DeepGlobe Building 78
We select segmentation methods that are included in MMSegmentation [8] and backbones that are supported by timm [52] for our benchmark study. An overview of 15 selected models are presented in Table 5. Specifically, DeepLabV3-ResNet101 and PSPNet-ResNet101 are the only models that leverage an extra module to increase the reception field, but does not utilize multi-scale features. Conversely, UNet-ResNet101 is the only model that does not have an extra module to increase the reception field, but merges hierarchical features. A brief introduction of each segmentation method and backbone is below.
Table 5: An overview of 15 selected models. FLOPs (G) is computed with an input size of  $512 \times 1024$ . Inference latency (ms) is measured with the same input size on a NVIDIA A100. Training memory is estimated using a ground-truth size of  $8 \times 19 \times 512 \times 1024$  (batch_size, num_classes, H, W), also on a NVIDIA A100. RF: if the model adopts an extra module to increase the reception field except aggregated downsampling. MS: if the model utilizes multi-scale features. Yes*: SegFormer does not adopt an extra module to increase the reception field but it has a large reception field enabled by the MiT backbones.
Method	Backbone	#Params (M)	FLOPs (G)	Latency (ms)	Memory (GB)	RF	MS
DeepLabV3+ [7]	ResNet101 [21]	62.58	528.32	26.47	42.68	Yes	Yes
DeepLabV3+ [7]	ResNet50 [21]	43.59	372.74	17.92	29.72	Yes	Yes
DeepLabV3+ [7]	ResNet34 [21]	22.57	194.52	8.11	10.80	Yes	Yes
DeepLabV3+ [7]	ResNet18 [21]	12.47	109.90	5.50	8.69	Yes	Yes
DeepLabV3+ [7]	EfficientNetB0 [46]	4.65	34.52	10.16	23.96	Yes	Yes
DeepLabV3+ [7]	MobileNetV2 [44]	2.86	25.05	6.45	18.64	Yes	Yes
DeepLabV3+ [7]	MobileViTV2 [36]	5.73	52.59	26.81	21.32	Yes	Yes
UPerNet [53]	ResNet101 [21]	85.40	517.95	20.20	22.63	Yes	Yes
UPerNet [53]	ConvNeXt [32]	117.21	254.43	30.64	22.87	Yes	Yes
UPerNet [53]	MiTB4 [54]	91.22	545.34	43.60	38.97	Yes	Yes
SegFormer [54]	MiTB4 [54]	63.17	163.01	36.08	36.53	Yes*	Yes
SegFormer [54]	MiTB2 [54]	26.52	87.16	17.65	21.73	Yes*	Yes
DeepLabV3 [6]	ResNet101 [21]	87.10	714.73	26.20	33.99	Yes	No
PSPNet [58]	ResNet101 [21]	67.96	532.31	23.90	33.98	Yes	No
UNet [42]	ResNet101 [21]	95.06	448.25	22.78	28.16	No	Yes
A.1 Segmentation Methods
UNet [42] is a classic encoder-decoder architecture for semantic segmentation. The encoder is a feature extractor and the decoder consists of a series of up-convolution operations followed by concatenation with the high-resolution features from the encoder path, and two regular convolutions. A crucial feature of UNet is the skip connections between layers at the same level in the encoder and the decoder. These connections allow the network to use both the context information from the low-level features and the spatial information from the high-resolution features, thereby enabling precise localization.