Revisiting Evaluation Metrics for Semantic Segmentation: Optimization and Evaluation of Fine-grained Intersection over Union
Authors: Zifu Wang
Maxim Berman
Amal Rannen-Triki
Philip Torr
Devis Tuia
Tinne Tuytelaars
Luc Van Gool
Jiaqian Yu
Matthew B. Blaschko
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Semantic Segmentation

Abstract: Semantic segmentation datasets often exhibit two types of imbalance: \textit{class imbalance}, where some classes appear more frequently than others and \textit{size imbalance}, where some objects occupy more pixels than others. This causes traditional evaluation metrics to be biased towards \textit{majority classes} (e.g. overall pixel-wise accuracy) and \textit{large objects} (e.g. mean pixel-wise accuracy and per-dataset mean intersection over union). To address these shortcomings, we propose the use of fine-grained mIoUs along with corresponding worst-case metrics, thereby offering a more holistic evaluation of segmentation techniques. These fine-grained metrics offer less bias towards large objects, richer statistical information, and valuable insights into model and dataset auditing. Furthermore, we undertake an extensive benchmark study, where we train and evaluate 15 modern neural networks with the proposed metrics on 12 diverse natural and aerial segmentation datasets. Our benchmark study highlights the necessity of not basing evaluations on a single metric and confirms that fine-grained mIoUs reduce the bias towards large objects. Moreover, we identify the crucial role played by architecture designs and loss functions, which lead to best practices in optimizing fine-grained metrics. The code is available at \href{https://github.com/zifuwanggg/JDTLosses}{https://github.com/zifuwanggg/JDTLosses}.

Revisiting Evaluation Metrics for Semantic Segmentation: Optimization and Evaluation of Fine-grained Intersection over Union
Zifu Wang 1+
Maxim 2erman
Amal Rannen-Triki"
Philip H.S. Torr'
Devis Tuia
Tinne Tuytelaars
Lue Van Gool 1,6,7 16.7
Jiaqian Yu
Matthew B. Blaschko'
ESAT-PSI, KU Leuven, Leuven, Belgium
Google, Zürich, Switzerland
Google DeepMind, London, United Kingdom
University of Oxford, Oxford, United Kingdom
5 EPFL., Lausanne, Switzerland
ETH Zürich, Zürich, Switzerland 7 INSAIT, Sofia, Bulgaria Samsung Research, Beijing, China
Abstract
Semantic segmentation datasets often exhibit two types of imbalance: class imbal- ance, where some classes appear more frequently than others and size imbalance, where some objects occupy more pixels than others. This causes traditional evalua- tion metrics to be biased towards majority classes (e.g. overall pixel-wise accuracy) and large objects (e.g. mean pixel-wiss wise accuracy and per-dataset mean intersection over union). To address these shortcomings, we propose the use of fine-grained mloUs along with corresponding worst-case metrics, thereby offering more holis- tic evaluation of segmentation techniques. These fine-grained metrics offer less bias towards large objects, richer statistical information, and valuable insights into model and dataset auditing. Furthermore, study, where we train and evaluate 15 modern neural networks with the proposed metrics 12 diverse natural and aerial segmentation datasets. Our benchmark study highlights the necessity of not basing evaluations on a single metric and confirms that fine-grained mloUs reduce the bias towards large objects. Moreover, we identify the crucial role played by architecture designs and loss functions, which lead best practices in optimizing fine-grained metrics. The code is available
undertake extensive benchmark
Introduction
Every metric reflects a certain property of the result and choosing the right one is important to emphasize those that care about. However, this choice may not be easy, many metries come with certain biases [34]. For semantic segmentation, the overall pixel-wise accuracy (Ace) is believed unsuitable due to its bias towards majority classes, especially given that semantic segmentation datasets typically have long-tailed class distributions (18). The mean pixel-wise accuracy (mAcc) is introduced to counter this issue by averaging pixel-wise accuracy across all classes, and it eventually became the official evaluation metric in PASCAL VOC 2007 [16], which was one of the earliest challenges that showcased segmentation leaderboard. However, mAce does not account for false positives, leading to over-segmentation of the results. Consequently, in PASCAL VOC 2008 [7]
*Corresonndncce to: zifu.wang @ @uuleuven.be
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

(a)
(b)
(c)
(d)
Figure An illustration of size imbalance. The class Car is depicted blue and the regions are highlighted within white dashed circles. (a) A car whose side mirror takes up around 2,000 pixels (from Cityscapes). (b) car that takes up around 1,000 pixels (from Cityscapes). (c) A car whose front wheel takes up around 600,000 pixels (from Mapillary Vistas). (d) A car that takes up around 1,000 pixels (from Mapillary Vistas).
the official metric updated to the per-dataset mean intersection union (mloU Since then, PASCAL VOC (18] has consistently used m and this metric has been adopted by multitude of segmentation datasets 38
2 39
59 5
12 15 43
mloUD, true positives, false positives and false negatives are accumulated across pixels over the whole dataset and therefore this metric suffers several notable shortcomings [10
It is biased towards large objects, which can pose a significant issue given that most semantic segmentation datasets inherently have considerable size imbalance (see Figure T This bias is particularly concerning in safety-aware applications with critical small objects such
autonomous driving 4
39
11
43
and medical imaging (37 22 49 3
It fails to capture valuable statistical information about the performance of methods on individual images or instances, which impedes comprehensive comparison.
It is challenging link the results from user studies to those obtained with per-dataset metric, since humans generally assess the quality of segmentation outputs at the image level.
In this paper, we advocate fine-grained mloUs for semantic segmentation. In line with previous studies [10] 55 28 31.. we calculate image-level metrics. When instance-level labels are available 39 59 J. we propose to compute approximated instance-level scores. These fine- fine-grained metrics diminish the bias toward large objects. Besides, they yield wealth of statistical information, which not only allows for a more robust and comprehensive comparison, but also leads to the design of corresponding worst-case metrics, further proving their importance in safety-critical applications. Furthermore, we show that these fine-grained metrics facilitate detailed model and dataset auditing.
Despite the clear advantages of fine ime-grriied mloUs over mloU", these metrics are seldom considered in recent studies within the segmentation community. We bridge this gap with large-scale benchmark. where we train and evaluate 15 modern neural networks on 12 natural and aerial segmentation datasets, spanning a variety scenarios including (i) street scenes 4
43), (ii) "thing" and "stuff" [18 38 59 5], (iii) aerial scenes [2] Our benchmark study emphasizes the peril of depending solely on single metric and verifies that fine- ene-grained mloUs mitigate the bias towards large objects. Moreover, we conduct an in-depth analysis of neural network architectures and loss functions, leading to best practices for optimizing these metrics.
In sum, we believe no evaluation metric is perfect. Thus, we advocate comprehensive evaluation of segmentation methods and encourage practitioners report fine-grad -eee-graindd metrics and corresponding worst-case metrics, complement to D We believe this is particularly crucial in safety-critice applications, where the bias towards large objects can precipitate catastrophic outcomes.
2 Fine-grained mloUs
In this section, we review the traditional per-dataset mean intersection over union, followed by our proposed fine-grained lmage-level and instance-level metries.
m Intersection over union (IoU) is defined as
TP
IoU
(1)
TP F FN'

---

where TP, FP and FN represent true positives, false positives and false negatives, respectively. Per- dataset IoU computes these numbers by accumulating all pixels over the whole dataset. In particular, for class it is defined as
TP.
IoU
Ei=1 (TPr, + F. + +NN,.)
where / is the number of images in the dataset; TP FP,. and FN are the number of truu-rousiii true-positive, false-positive and false-negative pixels for class in image i, respectively. The per-dataset mean IoU then calculated as
mloU = 1 2 IoU
(3)
where C' denotes the number of classes for the dataset.
Compared to pixel-wise accuracy (Acc) and mean pixel-wise accuracy (mAcc), mIoU aims to mitigate class imbalance and take FP into account. It is widely used in the evaluation of semantic segmentation models, but has several shortcomings. Most notably, is biased towards large objects in the dataset, due dataset-level accumulations of TP, FP and FN. To address this issue, resort to fine-grained mloUs at per-image (mloU', mloU?) and per-instance (mloU*) levels, as detailed below.
we compute per-image scores. particular, for each image and class per-image-ppe--clae score is calculated
mloU!. Following 10 55
28
31]
TPec
loU;,e TPi.c + FP,. + FN
(4)
Then the per-image IoU for image is defined as
lOU! Eg 1fIoU NULLHOUU,
(5)
Ee LIIOU; NULL}
where we only sum over loU;,e that is not NULL (discuss below). Averaging these per-image scores yields
mloU = Σ loU!.
(6)
Using image-level metrics presents challenge: not all classes appear in every image, leading to NULL values and potential ambiguities. In multi-clat segmentation, we denote the value as NULL if the class is missing from the ground truth, regardless of it appearing in the prediction or not. In contrast, for binary segmentation, the value is set to instead of NULL when the prediction includes the foreground class, but the ground truth does not. More detail of the definition is in Appendix D
mloU". As the left panel of Figure 2 shows, due to the presence of NULL values, averaging these per-image-per-clas scores by class first and then by image (mIoU*) is different from averaging them by image first and then by class. A drawback of mloU" is that is biased towards classes that appear more images"] To address this bias, we first average these per- ieagi--ma---eeeeeeee scores by image and define the per-image IoU for class
loU = Ele (loU, F NULL)HOU,e
(7)
El_ 1(loU,, A NULL)
Similar to m we then average these per-class values to obtain
C 1 - LoU
(8)
This is different from Acc that biased towards classes that take up more pixels.

---

I
FP
C
0.85
0.94
0.56
0.73
0.68
FN1
TP1 FP
C2
0.42
NULL
0.59
0.88
NULL
FN₂ TP3 FP
C2
NJLL
0.79
NULL
NULL
NUDL
Figure 2: Left: mloU' first averages per-image-per-class scores by class and then by image. mloU first averages them by image and then by class. mloU is biased towards classes that appear in more images, e.g. C₁. Right: Given two ground-truth instances (red) and 2 (blue), and the prediction (green),
compute TP TP2, FN1, FN2. We propose approximate FP1 and FP2 by distributing FP proportional to the size of each instance.
mloU*. IoU is known for its variecvariance. However, this property only makes sense at an instance level. When instance-level annotations are available, for example, for "thing" classes in panoptic segmentation (27] ("thing" classes are with characteristic shapes and identifiable instances such as truck; see a more detailed discussion of "thing" and "stuff" Appendix B.I., we can compute a even more tineggriine metric.
Specifically, for an image i and a class c, we have binary mask H W from the prediction of semantic segmentation network and an instance-level annotation H x W x Ki,. from panoptic segmentation, where Kg the number instances of class that appear image i. The key insight, as illustrated in the right panel of Figure is to realize that TP and FN for each instance can be computed exactly, and we are only left with image-level FP. We propose to approximate instance-level FP by distributing image-level FP proportional to the size of each instance, with other variants discussed in Appendix Therefore, for class c in image i, we compute the per-instance IoU as
TPi.kkk
loU
(9)
TPi..k + FNi.c,k
St,ck FPi.c
Ex-1 Steck
such that FP,,c is the total number of salse-positive pixels of class c in image i, and St.c.k TP,,, + cN is the size of instance k. Having these per-instance IoUs, we can again calculate per-class score as
El= Ekir IoU,,e. El Kic
IOUK
(10)
With slightly abuse of notation, we have
C Σ
mloUk
loU
(11)
0
cml
such that for classes with instance-level labels, we calculate loUK according to Eq. 10). and for classes without instance-level labels, such as "stuff" classes (classes that amorphous and do not have identifiable instances such as sky, see Appendix B.I). we compute it as Eq.
To conclude, similar to m we average the scores per-class basis for m and mloUk therefore mitigating class imbalance. Additionally, mloU" and m reduce size imbalance from dataset-level to image-level, and m further computes the score at an instance level. On the other hand, scores of mloU! are finally averaged on a per-image basis; therefore it will be biased towards classes that appear in more images. However, mloU! leads to a single averaged score per image (while c has scores per class), making per-image analysis (e.g. image-level histograms and worst-case images) feasible as we will present in section 5.1

---

Worst-case Metrics
is essential for safety-critical applications to evaluate a model's worst-case performance. Adopting fine-grained mloUs allows us compute worst-case metries, where only images/instances that model obtains the lowest scores are considered. In this section, we focus on mloU as an example. However, the same idea can be applied to both mloU' and mloUk
For each class e, we first sort loU;,e by images such that IoU1, S ... S loUr,. where Ie ΣL 1 i,ooU,e * NULL]. We then only compute the average of those scores that fall below the q-th quantile:
1
lOU max (1, LL x 9%))
IoU
(12)
For the ease of comparison, we want to derive single metric that considers different 4%. Thus, we partition range into 10 quantile thresholds {10, 20,..., 90, 1003. We then average the scores corresponding to these thresholds:
IOU
Σ
IOU
(13)
10
(E(10,...,100) 100)
Consequently, the scores with lower values will be given more weights to the final metric. Additionally, consider loU and loU to evaluate model's performance extremely hard cases. Having these per-class scores, we can average them to obtain the mean metrics: mloU<* mIoU and mloUe
4 Advantages of Fine-grained mloUs
In summary, we advocate these fine-grained mloUs as complement of mloU because they present several notable benefits:
Reduced bias towards large objects. Unlike mloUD fine-grained metrics compute TP, FP and FN at an image an instance level, thus decreasing the bias towards large objects. To further illustrate this on real datasets, we can compute the ratio of the size of the largest object to that of the smallest object for each class, evaluated at both dataset (r.P) and image levels (r-)):
maxx & St.ck
r/ max max; St.ck min; Sko,k
(14)
min; Sk..k
These ratios can serve indicator of size imbalance at either the dataset or image " level. We average log ff for "thing" and "stuff" classes separately, and note that r/ = for all "stuff" classes. In Figure 5 (Appendix B.I. we present the numbers for Cityscapes, Mapillary Vistas, ADE20K, and COCO-Stufi The size imbalance is considerably reduced at the image level, especially for "stuff" classes. As a result, fine-grained mloUs can be less biased towards large objects.
A wealth statistical information. For example, fine-grained metrics enable to compute worst-case metries image and instance levels as discussed in the previous section. We can also plot a histogram for m from which we can extract various statistical information and identify worst-case images. These worst-case images can be instrumental in examining the specific scenarios in which a model underperforms, i.e. model auditing. Additionally, employing fine- ine-gndined scores facilitates statistical significance testing. Altogether, this yields a more robust and comprehensive comparison between different methods.
Dataset auditing. Images that consistently yield low image-level score across various models can be examined. These low scores could potentially be attributed to mislabeling. Furthermore, the presence of discrepancies between image- and object-level labels can result in an abnormal m values (NaN) because the denominator will become zero. We present the identified mislabels in Appendix B.3
5

---

Experiments
We provide large-scale benchmark study, where we train 15 modern neural networks from scratch and evaluate them 12 datasets that cover various scenes. More details of 15 models and 12 datasets considered in our benchmark study can be found in Appendix A and Appendix respectively. The total amount of compute time takes around NVIDIA A100 year.
In our benchmark, we emphasize the importance of training models from scratch (training details are in Appendix C) instead of relying on publicly available pretrained checkpoints. This choice is motivated by several factors:
Analysis. By training models from scratch, we aim to analyze how specific training strategies lead to high results fine- aee-grained metries. This provides insights into the training process and enables a better understanding of the factors affecting performance.
Completeness. While there are pretrained models available for popular datasets like Cityscapes and ADE20K, there is scarcity of checkpoints for other datasets. Training models from scratch ensures more comprehensive evaluation across different datasets, providing a more complete perspective on model performance.
Fairness. Publicly available pretrained checkpoints are often trained with different hyperpa- rameters and data preprocessing procedures. Training models from scratch ensures fairness in comparing different architectures and training strategies, as they all start from the same initialization and undergo the same training process.
Performance. We leverage recent training techniques to improve results. By training models from scratch with these techniques, we aim to achieve better performance compared to other publicly available checkpoints. We provide comparison of our results with those of MMSegmentation [8] in Table Jnn Table
33
50 which have been shown
Statistical significance. Publicly available pretrained checkpoints often stem from single run, which may lack statistical significance. By training models from scratch, we can perform multiple runs and obtain statistically significant results. This ensures more reliable evaluation and avoids potential misinterpretations.
Table 1: Comparing the performance of DeepLabVV+-ResNeel100 with MMSogmentation. All results are m Red: the best in a column.
Danseet
Cityscapes
ADF20K
VOC
Context
MMMSggmenttton
B0 97
45.47
78.82
33.20
D.17
$1.07
Table 2: Comparing the performance ADE20K with MMSegmentation. All results m Red: the best in a column.
Model
D131-R101
DB2
UPN-R101
UPN-CCom
Seg-MiTB4
PSP-R101
MMSegmentatin
43.47
43.82
48.46
Our
46.32 0.33
36.85 4 0.34
0.11
51.08 = 0.42
50.23 0.26
45.69 = 0.33
5.1 Main Results
Complete results, including tabular entries, image-level histograms and worst-case images are in Besides, we rank 15 models by their averaged ranking across 10 datasets, excluding Nighttime Driving and Dark Zurich. As depicted in Figure 3 we contrast the rank of m with (a) m (b) mloU!, (c) m (d) m Furthermore,
Appendix G
count the number of times each worst-case image appears for each model, and present the count of three most common worst-case images for each dataset in Table 13 (Appendix G.. The key findings summarized below:
No model achieves the best result across all metrics and datasets. obtains the highest score for most of the metrics, possibly due to the ImageNet-22K pretraining of ConvNeXt, while other backbones are pretrained on ImageNet- However, (PPerNet-MTTB. and Se.Forerr-MiTB-. outperform "PerNet-ConvNer metrics focus on a certain property of the result, is important have a comprehensive comparison using various metrics.
usually
many worst-case metrics. Since different

---

URN
Sogg
R.10
UPN-RRO
DCC
PSP-RISI
PLI+ANN
LNet
UNDER101
DCS
DCA+EB0
PL11-334
DL3--334
DL1--818
MB2
(a)
(b)
URRN-DORY
15
Ri01
1IN--R001
JSP
DV3-MV2
R10]
PLS-
(c)
(d)
Figure 3:
Contrasting the rank of mloU with (a) mloUD, (b) mloU', (c) mloue (d) mlou Models below the green dashed line have a higher rank of m than that of the counterpart.
Models do not overfit particular metric specific range of image-level values. As shown in Figure we witness only minor local rank discrepancies when comparing mloU with D and notably, no rank differences arise when comparing mloU with mloU!. general, we observe a monotonic trend. This proves that models are not tailored to favor a particular metric. The rank differences between mloU and mloUC are also minimal. This implies that the advancements on m are global and models not overfit to specifie range image-level values. However, we identify a larger rank drop of 11 mloUC", indicating that UNet may suffer from more challenging examples.
The performance on fine-grained mloUs is relatively low, mloUD m and mloU are all averaged on a per-class basis. Their values typically follow the ranking: mloUD > mloU 2 mloUK The more fine-grained a metric is, the more challenging the problem it presents, since it less merciful to individual mistakes. Nevertheless, we observe that the results for mloU and mloU are usually very similar. This observation indicates that m can serve as reliable proxy for instace-wwise metrics when instance-level labels are unavailable, e.g. PASCAL VOC/Context, etc.
Improvements in fine-grained mloUs do not always extend to corresponding worst-case metrics. The worst-case metrics generally fall substantially below their corresponding mean metrics. For example, on PASCAL Context, the value of mloU ranges from 46.00% 60.17% yet all models have 0% C The presence of extremely challenging examples poses obstacles to the application of neural networks in safety-critical tasks. It would be an interesting future research to explore ways of enhancing a model's performance on these hard examples.
Certain images consistently yield low image-level IoU values across various models. As presented Table [3 most models agree on worst-case images across multiple datasets. Several challenging factors, such as difficult lighting conditions and the presence of small objects, can contribute to low IoU scores (as seen in Mapillary Vistas, depicted in Figures 19 2T). Besides, mislabeling in the dataset may also cause models to produce image-level IoU of 0 (see Appendix B.3)
7

---

Label
DL3+: 98.51%
DL3: 98.28%
UNet: 45.77%
Label
DL.3+ 42.90%
DL33 9.48%
UNet: 51.76%
Figure 4: From left to right: ground-truth label, prediction and per-image IoU of class Truck with DeepLabV3+, DeepLabVa and UNet using ResNet101 backbone, respectively. The class Truck is represented in dark blue and highlighted within white dashed circles.
Table 3:
Comparing DeepLabV3+, DeepLab V3 and UNet on Cityscapes using ResNet-101 backbone.
Red: the best column. Green: the worst in column.
Method
lnU
mlot!"
bell Ioll
mlot"
mlot!
Deept abV3-
84.94
80.50 + 0.16
55.26
76.21
DespLabi33
81.48 + 1.43
0.06
53.87
76.26 0.03
UNet
65.58 + 0.59
77.12 0.17
53.00 + 0.01
68.78 # 0.09
76.36 +: 0,06
Image-level histograms vary across datasets for the same architecture, but are similar across architectures for the same dataset. The same architecture yields significantly different shapes of image-level histograms across various datasets. We believe it has do with the coverage of dataset (the fraction of classes appear in each image, discussed in Appendix B.2): 2): dataset with low coverage (e.g. ADE20K and COCO-Stufi often results in a spread-out histogram with fat tails. On the other hand, although it is possible that a model that has medium score on all images obtains the same mloU' model that performs exceedingly well on some images and very poorly on others, we find that different architectures generally produce similar histograms for the same dataset.
Fine-grained mloUs exhibit less bias towards large objects. For example, in Cityscapes, the class Truck has high dataset-level size imbalance (see 4. As demonstrated in Table 3 compared with DeepLabV3-ResNeell
UNet-ResNet101 obtains a low value on RoUPu (15% lower) and m (3% lower), since it has a limited reception field and struggles with large objects such as trucks that are close to the camera. On the other hand, it is capable identifying small objects. As result, its performance on loU Truck. m and mloU' is similar to that of DeepLab C ResNet101.
Architecture designs and loss functions are vital for optimizing fine-grained mloUs. We have seen from previous findings that architecture designs play an important role in optimizing fine-grained mloUs. The influence of loss functions is also significant. We leave discussions of architecture designs in section 5.2.1 and loss functions section [52.2] These insights contribute to the development of best practices for optimizing fine-grained mloUs.
5.2 Best Practices
In this section, we explore best practices for optimizing fine-grained mloUs. In particular, we under- score two key design choices: (i) the aggregation of multi-level features as vital as incorporating modules that increase the receptive field, and (ii) the alignment of the loss function with the evaluation metries is paramount.
5.2.1 Architecture Designs
Our study reveals the importance aggregating multi-scale feature maps for optimizing fine-grained metrics. Deepl abV3 and PSPNet seek to increase the receptive field through the ASPP module and the PPM module [58], respectively. The large receptive field helps to capture large objects, resulting in high values D However, these methods lack the fusion of multi-scale feature

---

maps and lose the ability to perceive multi-scale objects due to aggregated down- www.lsam On the other hand. UNet does not incorporate a separate module to increase the receptive field and therefore underperforms on mloUD. Nevertheless, it performs comparably to DeepLabV33 and PSPNet on fine-grained metrics. The aggregation of multi-scale feature maps in UNet is critical for detecting small objects and achieving high values on fine-grained metric
This observation is further confirmed by comparing De with its successor, V3+, The sole architectural difference between them is the incorporation of a feature aggregation branch in DeepLabV3+. Although they produce similar results on mloU", DeepLab typically outperforms DeepLabV3 on mloU
It is encouraging all incorporate modules to increase the receptive field and to aggregate multi-level feature maps. Such design choices align well with the objective of fine-grained mloUs. We also advocate for the design of more specialized modules for perceiving small objects which can further improve the performance on fine-grained mloUs.
that state-of oftte--f-the-eee the-hhe-rtt methods, such DeepLabV3+, UPenNet, and SegFormer,
5.2.2 Loss Functions
Aligning loss functions with evaluation metrics is widely accepted practice in semantic segmentation
47
56
57
50 ST. The conventional approach involves combining the cross-entropy loss (CE)
and IoU losses 2
56
50 51, which extend discrete mloU with continuous surrogates. With the
introduction of fine
fine-graine grained mloUs, it is straightforward to adjust the losses to optimize for these new metrics. Specifically. we explore the Jaccard metric loss (JML) [50) and its variants for optimizing mloU' and mloUS. From a high-level, JMI. relaxes set counting (intersection and union) with simple 1 norm functions. More details of JMI. are in Appendix F
In Table 4 we compare DeepLabbV3+ Restee trained with various loss functions on Cityscapes and ADE20K, respectively. Specifically, (D, I, C) represents the fraction of mloUD, mloU and mloU in JML. For example, we consider the CE baseline: (0, 0, 0) and variants of JML only
1 (). The main takeaways are as optimize m (1, O, 0) and a uniform mixture of three: (5 3 follows:
The impact of loss functions is more pronounced when evaluated with fine-grained metrics. Specifically, we notice improvements on mloU over CE reaching nearly 3% on Cityscapes and 7% on ADE20K. We also observe substantial increments on mloU'. Roughly speaking, Acc 2 mlol > mloU in terms of class/size imbalance, while CE directly optimizes Acc. Therefore, it becomes more crucial to choose the correct loss functions when evaluated with fine-grained metrics.
There exist trade-off's among different JML variants. For example, compared with other JML variants, (1, O, 0) achieves the highest mloU in both datasets, but have low values on mloU and mIoUS. We use (0, 0.5, 0.5) as the default setting in our benchmark, but alternate ratios may be considered depending on the evaluation metrics.
A notable observation is that CE surpasses (0, 1. 0) in terms of mloU on Cityscapes. Interestingly. many segmentation codebases, including SMP 25 and MMSegmentation [8) calculate the Jac- card/Dice loss on per-GPU basis. When training on GPUs with limited memory, the per-GPU batch size is often small. As result, the loss function tends to mirror the behavior of (0, 1, 0), potentially leading to sub-optimal results as measured by mloUD.
Table 4: Comparing different loss functions using D on Cityscapes and ADE20K. (D, I. C) represents the fraction of 5 mloU' and m in JML, respectively, Red: the best in a column. Green: the worst in a column.
(), L,))
Gaywages mlol"
ADEZOK
mlol"
mloo"
m
mlot"
mlel"
(0, 0,
80.67
0.36
74.57
: 0.00
67.61 + 0.12
45.01
1 0.13
53.73
2 0.09
39.66
0.13
(1,0,0) 0)
81.22 2 0.27
75.34
+ 0.00
68.97
0.19
47.21
2 0.26
57.74 2 0.11
46.69 = 0.25
L, 0)
80.29 1: 0.28
76.28
1 0.02
69.97
0.00
45.63
1 0.19
58.37
45.35
0.25
46.55
0.54
0.009
46.97
( +. +)
81.13
4 0.10
76.19
+: 0.05
70.08
0.15
46.92
+ 0.25
58.61
2 0.02
46.77
= 0.07
[0. 2.
80.91
0.17
76.21
0.05
70.18
0.16
46.32
0.33
58.79
0.04
46.32 0.30
'This might explain the efficacy of UNet for medical datasets. Typically, these datasets exhibit small lesion areas but lack extremely large objects that necessitate large receptive field.

---

Discussion
6.1 Limitation
We propose three fine-grained mloUs: mloU", mloUC, mloU along with their associated worst-case metrics. These metrics bring significant advantages over traditional mloUD. However, they are not without shortcomings, and users should be aware of these when employing them in research or applications. Specifically, mloU! and mloUC fall short when it comes to addressing instance- level imbalance. Meanwhile, mloU serves merely as approximation of the per-instance metric, rather than an exact representation. Beisdes, conducting image-level analyses, such as image-level histograms and worst-case images, proves challenging with mloUC and mloU Although such capabilities are accommodated by mloU', has an inherent bias towards classes that are more frequently represented across images.
Additionally, it is important note that our conclusions based specific training setting, including choices of optimizer, loss function, and other hyperparameters. There is potential for varied findings when altering this setting, especially when it comes to the choice of the loss function.
Finally, our experiments primarily focus on natural and aerial datasets, excluding medical datasets from our analysis. Besides, while core emphasis is addressing size imbalance in relation to IoU, the concept of calculating segmentation metrics a fine- grained manner could be extended to other metrics, such mAec, the Dice score, the calibration error [19]. etc. We plan address these limitations in future work.
6.2 Suggestion
While mloU* might show bias towards more common classes, it remains invaluable for analyze the dataset with image-level information, as well as provide a holistic evaluation of various segmentation methods. Given that mloU is good proxy mloUK, and considering many datasets lack instance-level labels, we believe mloU adequately serves as an instance-level approximation and could be used most cases.
Regarding the worst-case metrics, we examine three variants: m mloUC* and mloUC*, We recommend the use of mloUC* in applications where worst-case outcomes are prioritized. We provide evaluations using both mIoUC* and mloUS! for the sake of comprehensiveness, but believe that adopting either one is sufficient. While utilizing ADE20K to emphasize these exceptionally difficult examples, it could introduce significant variance for smaller datasets such as DeepGlobe Land. For the latter, a value = or even larger might be more appropriate. With these considerations, dataset organizers can choose the metric that best fits their context.
1 might be suitable for larger datasets like
Conclusion
In conclusion, this study provides crucial insights into the limitations of traditional per-dataset mean intersection over union and advocates for adoption of fine-grained mloUs, as well as the corresponding worst-case metries. We argue that these metrics provide less biased and more comprehensive evaluation, which is vital given the inherent class and size imbalances found in many segmentation datasets. Besides, these fine-grained metrics furnish a wealth of statistical information that can guide practitioners in making more informed decisions when evaluating segmentation methods for specific applications. Furthermore, they offer valuable insights for model and dataset auditing, aiding in the rectification of corrupted labels.
Our extensive benchmark study, covering 12 varied natural and aerial datasets and featuring 15 modern neural network architectures, underscores the importance not relying solely on a single metric and affirms the potential of fine-grained metrics to reduce the bias towards large objects. It also sheds light on the significant roles played by architectural designs and loss functions, leading to best practices in optimizing fine-graines metrics.
10

---

Acknowledgments and Disclosure of Funding
We acknowledge support from the Research Foundation Flanders (FWO) through project numbers G0A1319N and SO0142IN, and funding from the Flemish Government under the Onderzoekspro- gramma Artificille Intelligentie (AI) Vlaanderen programme. The resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the FWO and the Flemish Government.
M.B.B. 3. is entitled to stock options in Mona.health, a KU Leuven spinoff.
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer Normalization. NeurIPS Deep Learning Symposium, 2016.
[2] Maxim Berman, Amal Rannen Triki, and Matthew Blaschko. The Lovasz-softmax loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Uniion Measure in Neural Networks. CVPR, 2018.
[3] Patrick Bilic et al. The Liver Tumor Segmentation Benchmark (LiTS). MIA, 2023.
[4] Gabriel J Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and Recognition Using Structure from Motion Point Clouds. ECCV, 2008.
[5] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCO-Stull Thing and Stuff Classes in Context. CVPR, 2018.
[6] Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking Atrous convolution for semantic image segmentation. arXiv, 2017.
[7] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with Atrous separable convolution for semantic image segmentation. ECCV, 2018.
[8] MMSegmentation Contributors. MMSegmentation: OpenMMLab Semantic Segmentation Toolbox and Benchmark, 2020.
[9] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. CVPR, 2016.
[10] Gabriela Csurka, Diane Larlus, and Florent Perronnin. What good evaluation measure for semantic segmentation? BMVC, 2013.
[11) Dengxin Dai and Lue Van Gool. Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime, ITSC, 2018.
[12] like Demir, Krzysztof Koperski, David Lindenbaum, Guan Pang, Jing Huang, Saikat Basu, Forest Hughes, Devis Tuia, and Ramesh Raskar. DeepGlobe 2018: A Challenge Parse the Earth through Satellite Images. CVPR Workshop, 2018.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR, 2021.
[14] Tom Felbode, Jeroen Bertels, Maxim Berman, Dirk Vandermeulen, Frederik Maes, Raf Biss- chops, and Matthew B. Blaschko. Optimization for medical image segmentation: Theory and Practice When Evaluating With Dice Score or Jaccard Index. TMI, 2020.
[15] Adam Van Fiten, Dave Lindenbaum, Todd M Bacastow. SpaceNet: Remote Sensing Dataset and Challenge Series, arXiv, 2018

---

[16] Mark Everingham, Luc van Gool, Chris Williams, John Winn, and Andrew Zisserman. The PASCAL Visual Object Classes Challenge 2007, 2007.
[17] Mark Everingham, Luc van Gool, Chris Williams, John Winn, and Andrew Zisserman. The PASCAL Visual Object Classes Challenge 2008, 2008.
[18]
Mark Everingham, Lue Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes (VOC) Challenge. IJCV, 2009.
[19] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. ICML, 2017.
[20] Bharath Hariharan, Pablo Arbelaez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. 2011.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
[22] Nicholas Heller et al. The state of the art in kidney and kidney tumor segmentation in contrast- enhanced CT imaging: Results of the KiTS19 challenge. MIA, 2021.
[23] Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units (GELUs). arXiv, 2016.
[24] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto. and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv, 2017.
[25] Pavel lakubovskii. Segmentation models pytorch, 2019.
[26] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ICML, 2015.
[27] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollar, Panoptic Segmentation. CVPR, 2019.
(28) Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. LISA: Reasoning Segmentation via Large Language Model. arXiv, 2023.
[29] Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. CVPR, 2017.
[30] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common Objects in Context. ECCV, 2014.
[31] Chang Liu, Henghui Ding, and Xudong Jiang. GRES: Generalized Referring Expression Segmentation. CVPR, 2023.
[32] Zhuang Liu, Hanzi Mao, Chao-Yua Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. CVPR, 2022.
[33] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. ICLR, 2019.
[34] Lena Maier-Hein et al. Metrics Reloaded: Recommendations for image analysis validation. arXiv, 2023.
[35] Sachin Mehta and Mohammad Rastegari. Vobi Light- General-purpose, and Mobile-friendly Vision Transformer. ICLR, 2022.
[36] Sachin Mehta and Mohammad Rastegari. Separable Self-attention for Mobile Vision Trans- formers. TMLR, 2023.
[37] Bjoern Menze al. The multimodal brain tumor image segmentation benchmark (BRATS). TMI, 2015.
12

---

[38] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam- Cho, Seong-Whan Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The Role of Context for Object Detection and Semantic Segmentation in the Wild. CVPR, 2014.
[39] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulò, and Peter Kontschieder. The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes. ICCV, 2017.
[40] Sebastian Nowozin. Optimal Decisions from Probabilistic Models: the Intersection-over. -vver-nnn Case. CVPR, 2014.
[41] Md Atiqur Rahman and Yang Wang. Optimizing intersection-ovee-umioo in deep neural networks for image segmentation. ISVC, 2016.
[42] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI, 2015,
[43] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Guided Curriculum Model Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation. ICCV, 2019.
[44] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. MobileNetV2: Inverted Residuals and Linear Bottlenecks. CVPR, 2018.
[45] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. CVPR, 2016.
[46] Mingxing Tan and Quoc V Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ICML, 2019.
[47] Vladimir N Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All You Need. NeurIPS, 2017.
[49] Ruchika Verma et al. MoNuSAC2020: A Multi-Organ Nuclei Segmentation and Classification Challenge. TMI, 2021.
[50] Zifu Wang. Xuefei Ning, and Matthew B. Blaschko. Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labels. NeurlPS, 2023.
[51] Zifu Wang, Teodora Popordanoska, Jeroen Bertels, Robin Lemmens, and Matthew B. Blaschko. Dice Semimetric Losses: Optimizing the Dice Score with Soft Labels. MICCAI, 2023.
[52] Ross Wightman. Pytorch Image Models, 2019.
[53] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified Perceptual Parsing for Scene Understanding. ECCV, 2018.
[54] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- Former: Simple and Efficient Design for Semantic Segmentation with Transformers. NeurIPS, 2021.
[55] Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip H.S. Torr. LAVT: Language-Aware Vision Transformer for Referring Image Segmentation. CVPR, 2022.
[56] Jiaqian Yu and Matthew B. Blaschko. The Lovász Hinge: Novel Convex Surrogate for Submodular Losses. TPAMI, 2018.
[57] Jiaqian Yu, Jingtao Xu, Yiwei Chen, Weiming Li, Qiang Wang, Byung In Yoo, and Jae-Joon Han. Learning generalized intersection over union for dense pixelwise prediction. ICML, 2021. preeitiisereeiittionn
[58] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid Scene Parsing Network. CVPR, 2017.
[59] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene Parsing Through ADE20K Dataset. CVPR, 2017.
[60] Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. ICLR, 2017.
13

---

Contents
Introduction
2
Fine-grained mlous
3
Worst-case Metrics
Advantages Fine-grained mloUs
5
Experiments
Main Results
5.2
Best Practices
5.2.1
Architecture Designs
8
5.2.2
Loss Functions
6
Discussion
10
Limitation
10
6.2
Suggestion
10
Conclusion
10
A Models
15
A.I I
Segmentation Methods
15
Backbones
16
Datasets
16
B.1 "Thing" and "Stufi"
17
B.2 Dataset Coverage
18
B.3 Dataset Auditing
18
Implementation Details
19
NULL in Fine-grained mloUs
Other Per-instance mloUs
21
The Jaccard Metric Loss
22
Results
22
G.I
Cityscapes
23
G.2
Nighttime Driving
28
Dark Zurich
33
G.4
Mapillary Vistas
38
G.5
Cam Vid
43
14

---

G.6 ADE20K
48
G.7 COCO-Stufi
53
G.8
PASCAL VOC
58
G.9
PASCAL Context
63
G.
DeepGlobe Land
68
DeepGlobe Road
73
G. 12 DeepGlobe Building
78
Models
We select segmentation methods that are included MMSegmentation 8 and backbones that are supported by timm [52] for benchmark study. An overview of 15 selected models presented in [3] Specifically,
13-ResNet101 and PSPNet-ResNet110 are the only models that leverage extra module to increase the reception field, but does not utilize multi-scale features. Conversely, UNet-ResNet 101 is the only model that does not have an extra module to increase the reception field, but merges hierarchical features. A brief introduction of each segmentation method and backbone is below.
Table 5: An overview of 15 selected models. FLOPs (G) is computed with input size of 512 x 1024. Inference latency (ms) is measured with the same input size on a NVIDIA A100. Training memory is estimated using ground-truth size of 8 x 19 x 512 x 1024 (batch_s _iize, num_ ham__clsses, H, W), also on NVIDIA A A100. RF: if the model adopts an extra module to increase the reception field except aggregated downsampling. MS: if the model utilizes multi-scale features. Yes": SegFormer does not adopt an extra module to increase the reception field but it has a large reception field enabled by the MiT backbones.
FL.P%
RF
DespLabbV3
k
ResNet101
21
62.58
528.32
26.47
42.08
Yes
Yas
DespLabV3+
7
ResNet50
21
43.59
372.74
17.92
29.72
Yes
Yes
DeepLabV3+
ResNet34
20
22.57
194.52
8.11
Yes
Yes
DespLab/V3
0
ResNet18
5.50
Yes
DeepLabV3+
7
BitccienNeii0
46
4.03
34.52
10.16
23.96
Yes
Yes
DespLabV3+
7
MobileNeV2
41
2.86
25.05
6.15
18.64
Yes
Yes
DespLabiv3
V2
26.11
Yes
UPerNex
Reshett01
Yes
1PeerNer
53
Com:N:Xi
117.21
254.43
30.64
22.87
Yes
Yes
UPerNer
53
MiTB4
54
91.22
545.34
43.00
33.97
Yes
Yes
Segiormer
5D
Mi114
54
36.08
SegFormar
MiTR2
Desplabb'
Reshet101
$3.10
714.73
26.20
33.99
Yes
No
PSPNet 5g
ResNett01
21
67.96
$32.31
23.90
33.98
Yes
No
UNet 22
ResNet101
A.1
Segmentation Methods
a classic encoder-decoder architecture for semantic segmentation. The encoder is feature extractor and the decoder consists of series of up-convolution operations followed by concatenation with the high-resolution features from the encoder path, and two regular convolutions. crucial feature of UNet is the skip connections between layers at the same level in the encoder and the decoder. These connections allow the network to use both the context information from the low-level features and the spatial information from the high-resolution features, thereby enabling precise localization.
UNet 422
DeepLab V3 [6] attaches the Atrous Spatial Pyramid Pooling (ASPP) module to the output of the feature extractor. ASPP applies atrous (dilated) convolutions at a variety dilation rates in parallel, allowing the model to handle objects of diverse sizes in the image.
V3+ builds upon DeepLabV3 and retains the use of the ASPP module. primary enhancement lies in incorporating encoder-decoder structure where the decoder upsamples the low-level features from the encoder and combines them with high-level features generate detailed segmentations.
15

---

PSPNet [58] adopts the Pyramid Pooling Module (PPM) to capture different levels of details in the scene. The idea behind this module to apply pooling operations at varying scales and concatenate the output alongside the original feature map.
UPerNet [53]
applies encoder-decoder structure. The encoder takes inspiration from the Feature Pyramid Network (FPN) [29], creating pyramid of features extracted at multiple scales. The decoder merges multi-level feature maps and adopts PPM at the lowest FPN level bring useful global prior representations.
SegFormer [54] features a lightweight decoder that solety consists of MLP layers for upsampling and fusing multi-level features. It avoids computationally intensive components, and the key for this simplified design is the large receptive field provided by the hierarchical transformer encoder.
A.2 Backbones
ResNet [21 is a type of CNN that uses shortcut connections, "skip connections". These shortcut connections enable the network to learn an identity function, ensuring that the output of the block is at least as good as its imput, and hence the term "residual", This identity function makes it easier for the network to learn and allows gradients to flow directly through the network during backpropagation, which can alleviate the problem of vanishing gradients in deep networks.
ConvNeXt [32 is a pure convolutional model, inspired by the design of vision transformers 13 It consists of several macro designs (stage ratios, patchified stem, etc), micro-designs (replacing ReI.U with GELU 233 substituting BN (26) with LN [1] etc), modern training techniques (AdamW
33 label smoothing 45
etc) and other designs (large kernel sizes, inverted bottlenecks, etc) to
modernize ResNets.
EfficientNet [46] is l'amily of CNN architectures innovatively designed for model scaling. Efficient- Net's central insight is recognizing the interactive nature of the network width, depth, and resolution. Therefore, they need to be scaled together to maintain model efficiency. The key is to adopt neural architecture search [60] approach to find the best scaling combinations.
MobileNetV2 [44] is designed to be lightweight, efficient network architecture optimized for speed and size, making particularly well-suited for mobile and embedded vision applications. adopts depth-wise separable convolutions from MobileNetV] (24] to reduce computational costs. Additionally, it leverages linear bottleneck layers to prevent nonlinearities from destroying too much information and inverted residuals blocks to facilitate gradient propagation across multiple layers.
specifically optimized for semantic segmentation tasks. One key aspect of MiT is its hierarchical structure. The model comprises several stages, forming multi-scale structure that effective for semantic segmentation. Another noteworthy feature of MiT is its relatively lightweight nature compared to some other vision Transformers, making it more practical for real-world deployment.
MiT (54 a type of vision transformer
[
Mobile V2 36 is a lightweight and low-latency hybrid network for mobile vision tasks. follows the macro-architecture of Mobile (35), which adopt a hybrid design of MobileNetV2 blocks 44 and self-attention operations [48]. Its main innovation is a separable self-attention that
reduces the computational complexity of self-attention from Onz²) to O(n).
B Datasets
Our benchmark include 12 datasets that cover various domains including: (i) street scenes 4 9 39
43), (ii) "thing" and "stuff" [18] 38, 59 5), (ii) acrial scenes [12]. The number of images in the dataset ranges from 50 (e.g. Nighttime Driving and Dark Zurich) to more than hundred thousand (e.g. COCO-Stuffy, the number of classes varies from binary (e.g. DeepGlobe Road and DeepGlobe
Building) to over hundred (e.g. ADE20K and COCO-Stuff). T ppessents overview of 12
selected datasets. Following is a brief introduction to each.
Cityscapes 9 is a large-scale dataset that focuses on semantic understanding of urban street scenes. contains diverse set of stereo video sequences recorded in street scenes from 50 different cities, with high quality pixel-level annotations of 5,000 frames (2,975 for training, 500 for validation and
16

---

Table An overview of 12 selected datasets.
Dilaeet
Scene
Plrann
PVal
#C lacoes
Coverage (%)
Ciyycapes
Libba
2975
500
19
6d. 22 +: 15.88
Nighitine Driving
Usban
24.50
+ 16.28
Mapillary Vistas
10
Usban
18,000
2,000
65
30.18
22.23
CamVid 4
Ushan
469
232
11
89.53
9.18
ADEE2K
50
"Thing" "sutt"
20,210
COCO-Smilf
"Thing"
118.267
4.82
PASCAL VDX
18
"Thing" -cely
10.582
1,449
21
11.59 + 25.54
PASCAL Contest
38
"Thing" & "stutt"
4,996
5,104
60
9.60
41.77
Deeppilobse
Acrial
647
Doeptilobe
4,981
100
DeupGlobbe Buikding
12
Aerial
8.476
2,117
2
88. 07
+: 23.30
1,525 for testing). In addition, there are 20,000 coarsely annotated frames also included in the dataset. In this study, utilize the finely annotated training and validation images exclusively.
Nighttime Driving [[I] and Dark Zurich [43] are datasets that provide images of various road scenes captured during the nighttime. They are designed to facilitate research in autonomous driving,
and specifically addresses the challenges associated with low-light and nighttime conditions. They incorporate fine pixel-level Cityscapes (9] labels.
Mapillary Vistas 39 is a large-scale street-level imagery dataset used primarily for semantic segmentation tasks. It is one of the most diverse publicly available datasets of street-level imagery, collected from numerous cities around the world under a variety of weather and lighting conditions. While images are mainly taken from mobile devices, in general the dataset spans wide range of different camera types, including head- car-mounted ones. This diversity in viewpoints useful for training more robust machine learning models that can generalize well to novel scenarios.
Cam Vid 4 is of the pioneering datasets used for semantic segmentation. The dataset comprises of footage captured from the perspective of a driving automobile, providing a vision of the road scene ahead. has been collected over different times of the day, offering an excellent selection of varying lighting conditions and appearances.
ADE20K (59] is widely used dataset for scene parsing. One unique feature of the ADE20K dataset its high diversity in object classes. The dataset has annotations for over 150 object categories, and includes a diverse of scenes from both indoor and outdoor environments. This diversity makes the dataset challenging and well-suited for developing and benchmarking algorithms for semantic segmentation.
COCO-Stuff (5] is an extension of the original COCO 30) dataset which is renowned for its diverse set of high-quality, real-world images containing complex everyday scenes of common objects in their natural context. augments the COCO dataset by adding pixel-level annotations for "stuff" classes. Thus, it contains the same images as the original COCO dataset and is well-suited for semantic segmentation tasks.
PASCAL VOC [18 is composed of images collected from the Flickr photo-sharing web-site. The labels include diverse set of classes, including various types of vehicles, animals, and indoor objects. The original dataset contains 1,464 images for training. 1,449 images for validation, and a further 1,456 images for testing. Extra annotations are provided in (20), leading to total of 10,582 training images.
PASCAL Context (38 an extension the popular PASCAL VOC dataset 18 where only object classes of interest were annotated, leaving a large portion of pixels marked as "background" or "unlabeled". In the PASCAL Context dataset, the goal is to provide comprehensive understanding of the scene, thus all pixels in an image are assigned a label, giving the "context" of the objects.
DeepGlobe contains high-resolution satellite images from around the globe, covering various geographical locations, features, and phenomena. offers three distinct satellite image understanding datasets: land cover classification, road extraction, and building detection. With this rich assortment of large-scale satellite imagery, DeepGlobe aids in propelling advancements geospatial analysis.
B.1 "Thing" and "Stuff"
The differences between "thing" and "stuff" encompass various aspects 5
17

---

3,50
'Thingg "Srff"
3,25
Avanged logir3tr)) "Thing""
3,00
2.75
16
Avennged logr2
2.50
2.25
2.00
Chyscpps
Mapillary
ADEZ0K
COCD
Figure 5: Results of averaged log for "thing" and "stuff" classes on Cityscapes, Mapillary Vistas, ADE20K and COCO-Stuff.
Shape: "thing" has characteristic shapes (e.g. person, car, phone), whereas "stuff" is amorphous (e.g. sky, road, water).
Size: "thing" occurs in characteristic sizes with little variance, whereas "stuff" is highly variable in size.
Parts: "thing" possess identifiable parts, whereas "stuff" does not. For example, fragment of sky is still considered sky, but wheel alone does not constitute a car.
Instances: "stuff" is typically uncountable and has no clearly defined instances.
Texture: "stuff" is typically highly textured.
The characterization of "thing" and "stuff" is usually a decision made by the dataset creator. How- ever, many datasets do not define which classes are "thing" or "stuff" (e.g. Cityscapes, Mapillary Vistas, ADE20K, etc). In this work, for datasets that provide instance-level annotations (Cityscapes, Mapillary Vistas, ADE20K, COCO-Stuff), we treat classes that have instance-level labels as "thing" classes, and as "stuff" classes otherwise.
B.2 Dataset Coverage
For each dataset, calculate the proportion of distinct classes present in each image to the total number of classes within that dataset. We define the coverage of dataset as the mean and the normalized standard deviation over all images. A dataset cxhibiting high mean and low normalized standard deviation is considered to have high coverage, implying low variations in contexts.
Urban-scene datasets, such Cityscapes, Mapillary Vistas, and Cam Vid, generally demonstrate high coverage. In contrast, "thing" and "stuff" datasets like ADE20K, COCO-Stuff, and PASCAL Context typically exhibit lower coverage they incorporate a wide range of indoor and outdoor scenes. Datasets with low coverage often showcase more spread-out image-level histograms (see Appendix G) with fat tails, indicating a diverse range of image-level loU values.
B.3 Dataset Auditing
Images that consistently yield a low image-level score across different models can be inspected. These low scores could potentially be attributed to mislabeling within the images. For example, in COCO-Stufi instances in "000000064574" and "000000053529"
erroneously labeled as "stuff" (as shown in Figures 3T 33 This mislabeling leads to most models registering an image-level IoU of O.
Additionally, the presence of inconsistencies between image- and object-level labels can lead NaN values during the computation of mK because the denominator in Eq. (9) will become zero. This provides a way to identify mislabels within the dataset. As demonstrated in Table we find these mislabels exist in Cityscapes and ADE20K. Specifically, only single mislabeled image is in the validation set (highlighted in red), and only a single mislabeled object is "stuff" (highlighted in green).
18

---

Table Identified mislabels in Cityscapes and ADE20K. ID: object ID. Reason: 1 means that the object is present in the object-level label but absent in the image-level label; 2 means that the object included the image-level label but missing from the object-level label. Red: the image in the validation set. Green: the object is "stuff"
Duisse
File
Cluss
10
Reason
Chyscopes
stmolboury
029577
Torain
22
Chyssapes
hanover. 000000 053604
Rider
25000
Chyyscpes
aschen 000022 000019
Car
26000
ADEDO
ADE N0000899
3407634
ADEDOX
ADE Irain. 0000099
Doer
3407624
ADEZOK
ADI: train 00010938
Docr
3407624
ADE2OK
ADE train, 00005926
Scat
14745351
ADEDOK
00018960
14745351
ADEDI
ADE 00018960
15859465
ADE20K
ADE. train, 00015355
Seat
14745351
ADEZOK
ADE. train, 000011166
Scot
14745351
ADEDOK
00010454
14745351
ADEZOX
ADE Irain, 00008027
Seat
14745351
ADEZOK
ADE train 00016771
Seot
133614009
2
ADEZOK
ADEL SOI__rrinn. 00012611
14745331
ADEZOK
ADE, 00012611
16116992
ADEDOX
ADE. train 00002139
Seni
14745351
ADEZOK
ADE. train, 00005924
Sear
15204121
2
ADEZOK
ADE. train, 00016746
Scat
44745351
ADEZOK
00008022
14745351
ADEZOX
ADIE Irain, 00008022
Seat
14410752
ADEZOK
ADE train 00019174
Seot
1203-4752
2
ADEZOK
ADEL 00019174
163855749
ADE 00019174
14673932
ADEZOX
ADE Irain, 00019174
Seat
17696785
ADEZOK
ALDEL train, 00019177
Seat
12909092
2
ADEZOK
ADEL 00013522
Scat
14745351
ADEZOK
00001.34
13369599
ADEZOX
ADDL Irain, 00001387
Booth
13309599
ADEZOK
ADEE train, 00001.367
Booth
11 1-469155
2
ADEZOK
ADE train, 00001.392
Booth
13369599
ADE_
11796735
ADEZOK
ADIE Irain, 00018451
Booth
12364599
ADEZOK
ALDEL train, 10015050
Booth
13369599
ADEZOK
ADIL ADii_rrain 00001573
Booth
12369599
ADI 00001573
11-669055
ADMEOO
ADDIL train, 00001573
Booth
15769215
2
ADEZOK
ALDE train, 00001575
Booth
12993861
2
ADEZOK
ADE. train 00001573
Booth
11993599
ADEL
15275519
ADEZOK
ADIE Jrain, 00012787
Booth
12369599
1
ADEZOK
ADEE train, 300127877
Booth
14945216
2
ADEZOO
ADEL 00012287
Booth
11999999
ADI 00012787
Booth
12715263
ADEZOK
ADE train, 00012713
Booth
13369599
ADEZOK
AUDEL train, 000144369
Booth
12369599
ADEZOK
ADE 00005268
Booth
13369599
ADE
Booth
13369599
ADEZOX
ADH__rainn_001009
Booth
13369599
C Implementation Details
Training details. We use AdamW B3] with weight decay of 0.01. The learning rate starts from 1e-6 and linearly warms up during the first 1% iterations to the initial learning rate which is 6e-5 for ResNet101/50, ConvNeXt, MiTB4/B2, and 6e-4 for ResNet34/18, 88 EfficientNetBO, MobileNetV2, MobileViTV2 The learning rate is then decayed in a "poly" policy with an exponent of 1. The number of training iterations and the crop size for each dataset is summarized in Table All models are trained with a combination of the cross-entropy loss and the Jaccard metric loss (JML) [50), balancing with weights of 0.25 and 0.75, respectively. JMI. optimizes for mloU! and m with equal weights, as discussed in Section S.222 Data augmentations including (i) random scaling in the range of [0.5, 2], and (ii) random horizontal flipping with probability of O.S.
All results are based on three independent runs and are presented in the format of standstandard deviation. We do not apply any inference-time augmentation such as multi-scale inference and flipping.
Dataset splits. For Nighttime Driving and Dark Zurich, we evaluate models trained on Cityscapes on their test sets, following MMSSemmenatiin X For CimVid, we merge "train" and "val" into the training set, and "test" as the test set. For DeepGlobe, since it does not provide an official data split, we select the first 80% (sorted by indexes) of training images as the training set and the rest as the test set. For all other datasets, adhere to the official training/validation splits.
19

---

In general, our implementations (models, dataset preprocessing, training recipes) closely follow MMSegmentation [8], except that we adopt a smaller number of training iterations to save the computational costs and utilize recent training techniques 50 33 for superior performance. As demonstrated Table Table 2 we either match or exceed the results of MMSegmentation with less training iterations (see Table 8). Also note that our training hyper-parameters are not optimized for mloU (see Table
Table 8: The number of training and warmup iterations and the crop size for each dataset. MMSeg-
mentation: the number of training iterations in MMSegmentation [8]
Dosaset
MMSSegmetaatoo
Macration
Wammup
Crop Size
Cityscapes
2
80,000
40,000
410)
512
1024
Nightime Driving
1024
Dark Zarich
512 1024
Mapillary Visiz
39
160,000
1,600
512 1024
CamVid 1
20,000
200
512 X 512
ADF20K
160,000
800
512
COCO Stut-Stll [5]
160,000
80,000
800
512 X S12
PASCAL VOC [8]
40,000
40,000
400
512 X 512
PASCAL. Context
1S1
80,000
40,000
400
512 X 612
DeepGlube
X 512 X 512
DeepClahe Road
Fi
40,000
400
DeepGlobe Building
R [5]
40,000
400
512 X 512
D NULL in Fine-grained mloUs
For each image, based on the presence or absence of a class in both the prediction and the ground truth, we can identify four cases:
1. The class is present in both the prediction and the ground truth.
2. The class is absent in both the prediction and the ground truth (true negatives).
3. The class is absent in the prediction but present in the ground truth (false negatives).
4. The class is present in the prediction but is absent in the ground truth (false positives).
In multi-class segmentation, for cases and 3, we compute the per-image-pert placeaass score as usual. In particular, the score for case is O. For cases 2 and 4, define the score NULL. In binary segmentation, if it is viewed as 2-class segmentation, then we can refer to the definition above. Otherwise, if it is considered foreground-backken background segmentation, the definition for cases 1 and 3 remains consistent with the multi-class setting. For case 2, we define the score as and for case 4, we define the score as 0. A summary is shown in Table 9
Notably, the multi-class setting, our definition differs from Csurka et al. [T]. their definition, the score is 0 for case 4. Our modification aims preventing the metrie from overly reacting to minor false positives, c.g. a few pixels are mispredicted as a car absent from the ground truth. In safety-critical settings, false negatives should have higher importance, e.g. overlooking a present in the ground truth. Overemphasizing such minor false positives might inadvertently diminish the significance of false negatives. A concerning scenario could be when the ground truth contains only small number of classes, yet each unrelated class has few pixels erroneously present in the prediction. It is pivotal to note that in our definition, these mispredicted pixels still incur penalties they are false negatives with respect to the corresponding ground-truth classes.
Besides, we do not want to discriminate between false positives solely based on whether the predicted class is present in the ground truth or not. Generally speaking, mispredicting Road as either Car or Truck should incur similar penalty. However, according to the definition of [10], if Car is present in the ground truth and Truck is not, mispredicting Road as Car may only result a mild penalty, whereas mispredicting Road as Truck will incur full penalty.
To further illustrate the differences between the definition of [00 and ours, consider an example image composed of just 4 pixels, with a total of 6 classes in the dataset, as presented in Table To We compare the two ways of computing the score, as detailed in Table II When calculating the score as prescribed by [10], false positives that appear in the prediction but not in the ground truth incur a considerable penalty. Consequently, the mean loU value for this image only 0.25 under their definition, but is 0.5 under ours.
20

---

we present the two ways for calculating mloU using DeepLabV3++ResNelO00. evident that, when the score derived following [0]. there is more aggressive penalty, resulting in generally lower value.
For more empirical comparison, in Table 12
In the end, although present definition with the belief that it offers advantages over 10 in certain scenarios, we understand that each metric will have its pros and cons. It is essential to note that our aim is not assert that our metries are the best only way, but to highlight this distinction. Ultimately, end users could select the version most aligned with their specific requirements.
Table 9: The definition of loU;, in multi-class and binary segmentation.
Multa
Binay
loU,.e
loU,
NULL
NULL
Table 10: Predicted and ground-truth classes of an example image composed of just 4 pix- els, with total number of 6 classes in the dataset.
Table 11: The two ways of computing IoU,, for the example image.
03
Dur 0.5
0.5
WOLL
WILL
KULL
RULL.
Fred
Table 12: The two ways of computing mloUC using DeepLabbV3+-ReNeet000
Mapilary
ADEZOK
10
56.14 = 0.43
23.96 = 0.19
20.25 = 0.06
20.98 + 0.00
62.30 + 0.58
25.92 +: 0.07
40.14 1.89
70.18 = 0.16
46.23 0.12
46.32 = 0.30
41.98 4 0.04
79.07 + 0.188
55.89 + 0.07
53.76.20.40
E Other Per-instance mloUs
Recall that have instance-level T, FNi,e, and image-level FP.,,. We can solve minimization problem:
TPick
(15)
TP: Tiic + FNi.o. + +FP Fii.e
FP P.ickkk FP
FP...
(16)
Alternatively, we can replace the minimization with maximization. Denote the resulting instance-level values as loUming,á,k and loU loUmax max c, respectively. They provide a lower/upper bound on the value of IoUk,c.k:
LoUmin IoU,,., I IoU
max c,
(17)
Compared to distributing image-level FP according to the size of each instances (as defined in Section 2). IoU c.i.k and loU IoU c, may yield results misaligned with our intuition. For example, consider case with two instances where TP: 1, TP2 10. FN1 FN2 0, FP; FP2 = 2. Then we have
10
+
(18)
I + FP
10 + FP
which is minimized when FP; 2, FP. instance when there is an instance imbalance.
0. Thus, IoUminest will distribute FP the smaller
Consider another example with two instances such that TP, TP2 10, FN, FN2 FP + FP, = 10. Then we have
10
10
(19)
10 FP
10 + FP
which maximized when FP, = 10, FP2 = 0 or FP, = O, FP, = 10. Therefore, IoU C.k will
distribute FP to one instance when instances are of the equal size.
21

---

The Jaccard Metric Loss
IoU is defined over sets that are discrete and therefore cannot directly optimized by neural networks.
Given two sets 2, {0, 13", the Jaccard metric loss (JML) (50] rewrites IoU as a function of set difference:
IoU
(20)
x yl
[
|
and realizes that when z, y E [0, 1] set operations can be relaxed with norm functions:
[c
| 1, lyl 11r/la, |rrAyl ||
y/|
(21)
Taking these together, the loss is defined as:
|| + 2/11 || 3/11
JML =
(22)
|| + w|1l + || y/
JML has the property that it identical to the widely used soft Jaccard loss (SJL) 40 41 when y {0, 13", and it is metric on [0, 1]". Therefore, it is compatible with soft labels, e.g. 0.5, while SJL is not [50].
JML extends discrete IoU with continuous surrogate. It was used to optimize mloUD, but it is straightforward to adjust optimize fine-grained mloUs by simply modifying how IoU values are aggregated during the loss computation.
G Results
In this section, we provide detailed tabular results, image-level histograms, and examples of the worst- case images for each dataset. In Table [3 we present the count of three most common worst-case images for each dataset.
Table 13: The count of three most common worst-case images for each dataset.
Duiset
83
Cityszapes
Nightime Driving
Dark Zurich
2
Mapillry Vistas
ADE20K
COCO-Sut
PASCAL
DeepGlohe Land
13
DeepGlibe Raad
DozpCilobe
22

---

G.1 Cityscapes
Table 14: Results: Cityscapes, Red: the best for each metric. Green: the worst for each metric.
miou!
mlou!!
miou! mloyee mlotyk
mloU! mloye! miot
mlot/
Method
Rackhone
miat mlot
mistyk3
76,21
0.05
70.32
0.06
59.17 0.38
51.62 0.71
LbVp3abV33
Re
70.18 + 0.16
02.64
0.19
17.13 0.13
6.00 + 0.00
09.77
16.11
75.33 1 0.12
69.31
0.189
57.90 0
51.30 0.91
DeepLabb33
ResNet50
69.10 + 0.30
51.65
0.42
15.58
16.69 0.40
4.67 + 0.47
79.34 +: 0.36
86.85
0.42
96. 0.06
76.24 + 0.07
69.19 + 0.12
$7.50 1: 0.55
80.39 = 2.30
Deeplabb33
ResNe134
0.46
78.61 + 0.14
86.
0.12
96.18
0.04
74.23 1 0.04
65.07 0.08
56.12 E 0.36
49.65 0.73
DeepLabb33
66.91 4: 0.18
48.35 0.25
15.32 0.17
5.00 0.00
77.07 1: 0.17
85.31 0.38
95.96 +: 0.03
DeepLabb2e
BfficienNerB00
+
17.43 0.15
66.98 + 0.03
48.45 0.06
16.04 0:21
6.67 + 0.47
77.40 + 0.13
85.65 0.12
96.07 0.04
DeepLabbV3
MohileNerV2
66.79 + 0.09
49.
0.09
15.43 0.31
0.00
66.38 + 0.10
47.75 a 0.08
14.58 0.21
5.00 + 0.00
75,40 1: 0.43
84.28 0.44
95.78 0.03
75.06 +: 0.06
69.12 0.12
58. 0.38
51.33 0.96
DeepLabb33
MobieVi1V2
67.99 + 0.27
68.47 + 0.25
51.18 0.25
49.80 0.28
17.80 0.15
16.47 0.14
6.00 0.00
6.33 + 0.47
75.73 1: 0.04
69.68 0.07
$8.09 0.22
51.61 0.95
UParNer
ResNet1B1
09.24 + 0.17
51.68 0.19
17.45 0.D8
4.67 + 0.47
68.78
16.47
0.82
77.13 0:13
71.25 1 0.19
60.41 = 0.40
53.86 = 0.81
UPerNet
ComNNeXt
71.22 4: 0.22
53.87 0.33
19.48 0.44
6,00 + 0.00
81.81 + 0.11
0.07 88.30
96.62
0.02
76.52 0.16
70.82 0.23
00.50 1 0.64
56.09 I 1.67
UPerNeet
0.33
80.99 + 0.20
88.10 0.23
96.61 = 0.04
76.78 0.07
71.05 0.13
60.12 0.43
55.15 = 1.36
SogHormer
70.14 0.19
52.
0.18
18.22 0.14
6.33 0.47
80.75 + 0.19
87.96 0.17
96.55 0.01
Seyfoormer
MITB2
68.63 0.14
0.16
50.41 0.12
17.41 0.36
6.33 0.47
79.96 + 0.05
87.40 0.00
96.36 0.01
DespLabv
ResNer101
68.51 1: 0.12
69.10 0.07
51.60 0.12
50.08 1 0.15
15.79 0.05
16.91 0.09
4.67 1 0.47
4.67 0.47
80.07 1 0.06
87.77 0.13
95.25 0.02
76.18 +
69.2T
58.83 0.29
PSPNet
ResNet101
66.58 + 0.14
69.05 + 0.15
51.53 0.21
50. 18 0.18
15.84 0.09
16.94 0.08
6.00 + 0.00
5.00 + 0.00
75.36 1 0.06
69.45 0.07
58.16 0.20
52.19 = 0.994
UNet
ResNet101
66.31 1: 0.20
08.79 + 0.10
51.28 0.08
0.22
16.59 + 0.16
15.66 0.17
5.00 + 0.00
5.00 1 0.00
77.12
0.04
96.05 0.00
23

---

20
20-
25-
15
15
20
100
Deepl.abV3+-Reseet
DeepLaab3++RReie Net5
DecpLabbV3+RResNtttt min: 44.54, max: 93.57
min: 44.76, max: 94.66 mean: 76.21, std: 7.31
min: 43.73, max: 94.52
mean: 75.33, std: 7.44
mean: 75.24, std: 7.46
skew: -0.53, kurtosis: 0.65
skew: -0.50, kurtosis: 0.37
skew: -0.52, kurtosis: 0.50
25
25
20-
1m
DeepLabV3+-ResNelle
DeepLab VeeplaabV+-EffcieennNNe
Deepplab33-MMobieNNN
min: 48.19, max: 93.15 mean: 74.22, std: 7.60
min: 44.92, max: 94.18
min: 44.92, max: 93.93
mean: 74,60, std: 7.48
mean: 73.60, std: 7.61
skew: -0.45, kurtosis: 0.25
skew: -0.43, kurtosis: 0.26
skew: -0,51, kurtosis: 0.30
20
22
20
15-
2-
20
42
120
20
300
DeepLabV3+MMoblee min: 44.05, max: 94.31
UPerNet-ResNeen
JPPerNetCCovNNe..
min: 45,71, max: 93,94
min: 48.59, max: 95.03
mean: 75.06, std: 7.32
mean: 75.72, std: 7.49
mean: 77.13, std: 7.24
skew: -0.51, kurtosis: 0.49
skew: -0.52, kurtosis: 0.41
skew: -0.48, kurtosis: 0.28
20
50-
25.
15
15-
15
10
20
UPerNet-MiTB4 min: 52.55, 94.65
SegFormer-MiTH MBTB
SegFormer-MTTB2
min: 49.05, max: 93.95
min: 51.16, max: 94.06
mean: 76.52, std:
mean: 76.78, 7.07
mean: 75.66. std: 7.18
skew: -0.38, kurtosis: -0.10
skew: -0.48, kurtosis: 0.19
skew: -0.42, kurtosis: 0.09
25-
25-
25.
20.
CeepLabV3-ResNeelle min: 48.09, max: 94.62
PSPNet-ResNetlOf min: 47.54, max: 94.16
UNel-ResNetIO1 min: 45.16. max: 94.10
75.26, std: 7.32
mean: 75.18, std: 7.30
75.36, std: 7.34
skew: -0.47, kurtosis: 0.23
skew: -0.42, kurtosis: 0.04
skew: -0.46, kurtosis: 0.38
Figure 6: Histograms: Cityscapes.

---

DeepLab ResNet101: lindau 000045 000044500001900000 1atgs_00000199_ttgggehii (loU) 43.56).
VieLabb33+ReeNet555 lindau_000045, 000010001 leftImg&bit (IoU) 41.71).
DecpLabV3+-ResNet34 aodduu40000444_110000000 1aa___00000100011111 (IoU) = 41.34).
DeepLab 3 lindau 1000044_000019leffllme (loU! 43.90).
Bailau_000055,_01100 000019 1000000000ttllnnn0 _efflmggttitt (loU! 42.24).
Figure 7: Worst-case images: Cityscapes 1 1/3. Left: image. Middle: label. Right: prediction.
25

---

DeepLab MobileNetV2: lindau 000034_000015 leftImgöbit (loU? 41.51).
DeepLab V3+ Vobile lindau 000000000000 000045_000019 1ettllgggtttttii (loU) 42.11).
UPerNet-ResNet101: lindau 1ee_000000000000 1e000019lefllmgStiii (IoU} = 44.28).
UPerNet-ConvNeke lindau_00008s 00000004_000011_eeefeenhhhe (loU! 44.72).
UPerNet-MiTB4: 0000510000911111111111 1000199lefllm@8bil (loU! 44.25).
Figure 8: Worst-case images: Cityscapes 2 2/3. Left: image. Middle: label. Right: prediction.
26

---

SeeFormer-MiBB.. lindau 000045_000011 leffomgieettlmg (loU? 45.04).
SigFormer-MiB22 lindau 100034_000015 letillg@sttttti (IoU! 46.96).
DeepLabV3-Res 10000340000010 Jeftlm@Sbit (IoU) = 44.04).
RePNet-ResNet101: kinda_00000450000 1000445_00011_eettlgggg (loU! 45.87).
UNet-ResNet101: lindau (iiiaa_000055_.0111 100045_0000190000000 1ww00000000l00ggg000011 leftlmg@bit (loU! 42.14).
Figure 9; Worst-case images: Cityscapes 3/3. Left: image. Middle: label. Right: prediction.
27

---

G.2 Nighttime Driving
Table 15: Results: Nighttime Driving. Red: the best for each metric. Green: the worst for each metric.
miat! mlou
miou!! mlot
miot! mlou
mlat miou
Method
Hackhone
mlou
mAkec
Aec
43.02 = 2.28
31.63 + 2.07
13.68 = 3.54
10.89
4.59
DeepLabV34
ResNet101
25.93
1.68
1.77 40.29
2.43
60.80
0.80
36.46 1: 0.99
28.1
14.01 3.20
13.36 = 3.21
DeepLabb33e
ResNet50
1.00
1.41
26.85
1.17
18.30
1 0.30
4.07 1.04
3 1.03
DoepLabbV3
ResNe134
20.49 4: 2.05
6.24 + 0.98
0.84 1.02
0.67 + 0.94
29.66 # 0.96
20.25
+ 0.23
7:22 0.99
6.36 + 0.38
DeepLabV24
ResNet18
19.27 + 1.44
8,33 + 0.56
0.82 0.32
0.33 + 0.47
14.20
34.06 =2 2:34
23.63
2.02
7:55 2:75
5.71
DisplabbV3
23.66 + 1.42
10.04 + 1.22
0.69 1 0.40
0.00 + 0.00
19.45 +: 1.79
40.71
1.76
61.71
4.52
0.71
DespLaabV34
MohileNerV2
14.96 + 1.44
4.78 + 0.60
0.04 0.05
0.00 + 0.00
12.67 + 1.48
31.88
3.21
33.22
3.82
DeepLabV3e
MobileVIVV2
31.88 1.38
16.49
0.82
4.32 1.03
3.67 1:25
25.88 + 0.8
43.99 1 1.50
71.15 + 1.90
28.96 + 24.24 + 1.76
1.99 1.11
47.87
1.62
63.55
5.89
62.34 1 1.21
53.
0.94
0.68 $2.33
41.86
UPerNeet
42.10 + 0.21
63.38
0.66
82.65
0.33
61.27 = 1.15
52.21
1.16
1.16 39.71
39.10 1.17
UPerNeet
41.59 +: 0.85
83.34
61.45 1 1.53
51.21 1 1.78
$7,90 12 3.07
36.07 = 3.43
SegHormer
M1184
50.89 1: 2.45
40.91
1 2.29
24.63 1: 1.78
23.47 = 1.18
SegFormer
MITR2
41.33 +: 2.03
25.21
1.32
9.61 1.611:00 0.16
9.00 + 0.00
42.21 + 3.33
31.12
3.79
16.23 3.87
15.08 = 3.54
DespLabV3
ResNet101
30.71 + 2.58
2.27 14.88
2.46 1.22
1.67 +: 1.25
40.48 1:
29.50
13.87 0.55
11:37
PSPNet
Re
30.22 + 1.98
2.10 45.63
14.81
2.57
1.78
1.67 + 1.70
23.65 +: 1.17
2.74
61.03 4.07
9:31
UNet
ResNer1B1
26.72 +: 2.38
1:70 40.33
12.31
2.56
1.36
1.67 + 1.25
21.63
1.17
2.14
63.95 :: 3.74
28

---

20
100
DeepLabV34+ Ne1
DeepLab 3
min: 10.90, max: 80.01
min: 13.68, max: 76.32
min: 6.60, max: 76.78
mean: 43.02, std: 14.41
mean: 38.46, std: 13.00
mean: 28.85, std: 14.06
skew: 0.24, kurtosis: -0.18
skew: 0.53, kurtosis: 0.22
skew: 0.96, kurtosis: 1.17
DbepLabV3+-ResNeae min: 5.35, max: 70.27
DeepLabV34--obbieeNen
min: 7.95, max: 66.73
min: 0.71, max: 64.58
mean: 29.66, std: 12.10
mean: 34.06, std: 13.00
mean: 20.10, std: 13.82
skew: 0.72, kurtosis: 1.12
skew: 0.31, kurtosis: -0.33
skew: 1.02, kurtosis: 0.91
130
20
300
DeepLabV3+ MatDV-3--Mbileiiiii MobileViTV2 min: 18.36, max: 74.31
UPerNet-ResNetUU ResNet101
UPerNet-ConVNeXX min: 43.08, max: 87.03 00
min: 16,50, max: 75.09
mean: 43.04, std: 12.29
mean: 39.39, std: 12.86
mean: 62.54, std: 11.46
skew: 0.44, kurtosis: -0.07
skew: 0.50, kurtosis: -0.04
skew: 0.25, kurtosis: -0.66
20
UPTTNet-MFTB4 min: 39.69, 83.31
SegFormer-MiTHe MiTB4 min: 39.44, 83.36
SegFomer-MMTB2
min: 26.66, max: 72.75
mean: 61.27, std: 11.19
mean: 61.45, std: 12.34
mean: 50.88, std: 11.89
skew: 0.1 kurtosis: -0.98
skew: -0.06, kurtosis: -1.06
skew: -0.18, kurtosis: -0.89
DeepLabV3 1eepLab
PSPNeI-ResNeiIP min: 14.53, max: 74.87
UNel-ResNet101
min: 16.55, max: 78.74
min: 8.99, max: 72.76
mean: 42.21, std: 14.04
40.48, std: 13.78
36.86, std: 14.05
skew: 0.46, kurtosis: -0.22
skew: 0.28, kurtosis: -0.39
skew: 0.24, kurtosis: -0.20
Figure 10: Histograms: Nighttime Driving.
29

---

DeepLab V3+ ResNet101: 0 frame 1171 Jeftlmgbit (loU) = 6.57).
0 f_frame_ 1171 Leftlmggbit (IoU) = 9.59).
Deeplai Lab Resseet:: 0_frame_eaaaeeee O_frame__287_eeeee 0et___0288_eeee8ii11 (loU) = 2.42).
V 0_ 0_frame_ 1171 Jeftlmg@bii (IoU) 4.82).
DeepLabV3++ BeepLaabV++EffcieentNee EfficientNeeBB: 0. frame 1171 Jeftlmmbtt (loU) 1.53).
Figure 11: Worst-case images: Nighttime Driving 1 3.3 Left: image. Middle: label. Right: prediction.
30

---

DeepLab MobileeeNet 0 frame _07e4_03344__0 lertlm@git (loU) = 0.17).
DeepLab - 0 0_frame_ 1138 lef_lletllgti (IoU) 13.87)
Reseeee- 0 (aaime_1816_eeffhee (loU) 11.13).
UPerNet-ConNNeet:: 0 (atne__043__eefllmgee (loU) 41.24).
UPenNet-MiTB4: 0 f.frmme_ 2548 leftImg&bit (loU} 37.89).
Figure 12: Worst-case images: Nighttime Driving 2/3. Left: image. Middle: label. Right: prediction.
31

---

MerFormer-MiBBB.. 0, frame 2548 leftlmm@mttti (loU) 32.86).
SegFormer-MiTB2: 00 0_frame_ 1687 left_liiit (loU) 22.12).
Deeplab V3- 0 1171 lef (loU) = 11.02).
PSPNet-ResNet101: 0 o_frame_Ilee 1171 Je7__efllg@@ (loU) 9.58).
#Net-ResNet101. 0_frame 0461 lefttmggtttttit (loU 3.58).
Figure 13: Worst-case images: Nighttime Driving 3/3. Left: image. Middle: label. Right: prediction.
32

---

G.3
Dark Zurich
Table 16: Results: Dark Zurich. Red: the best for each metric. Green: the worst for each metric.
miot!! miou mlotpo
mlot!! miot
miot!! mlou
mlot! miou
Method
Backhone
mAde
21.36 + 0.36
16.66
0.33
5.58
U.4.1
8.17 + 0.50
DeepLabV3e
ResNet101
18.41 0.89
1: 0.32
2.07
0.09
1.33 0.47
16.08 + 0.44
11.66 0.50
6.03 0.88
5.77 0.84
Dosplaaby33
ResNer50
13.44 + 0.52
5.00 + 0.21
0.57 0.10
0.00 + 0.00
12.32 + 0.13
5.18 + 0.15
4.337
2,98 0.64
DeepLabV3e
ResNet34
9.87 0:226
3.54 0.18
0.84 0.4.1
0.33 1: 0.47
8.42
T-86
4.19 0.31
DeepLabb34
8esNetl
0.50
3.54 0.16
0.92 0.228
0.67 1 0.47
8.38
26.04 0.50
33.48 2.66
DeepLabbV3
EfficeenNeetill
13.42 0.75
6.49 + 0.23
1.08 0.64
0.33 + 0.47
11.21 + 0.79
29.31
1.74
38.19 + 1.70
DeepLabV3e
MobileNetV2
6.95
2.37 + 0.24
0.12.1 0.10
0.00
6.54 6.54 0.93
24.14 +: 1.85
25.17 2 4:332
17.02
6.75 2.42
2218
DeepLabV3e
1.31
11.51 + 1.55
20.13 2.03
41.76 2 6.32
17.29 1.14
13:22 1.19
6.87 1.83
6.52 2.13
13.39
0.24
35 64
1.18
41.75 1: 2:22
26.03 + 1.26
20.68
1.44
12.90
1.78
12:28 1.50
UPerrer
ComoNcXi
21.98 + 1.34
0.73
61.71
1.93
21.76 + 1.77
16.64
1.06
8.17 1.26
7.27 + 1.24
UPerNeet
MITB4
18.56
1.22
1 0.65
0.21 1.74
1.00 0.00
24.33 + 1.17
18.68 : 1.08
8.59 0.62
6.95 + 0.65
SegFormer
M1TB4
21.01 + 0.88
9.35 + 0.66
1.92 0.21
1.00 + 0.00
23.75 4 0.49
18.55 1: 0.49
9.73 0.116
8.30 0.80
Seyfoorer
MiTD2
19.85 + 0.31
6.44 + 0.48
1: 1.91
1.33 + 0.47
19.56 +
1.72
6.55
6.83
DesplabbV33
101
17.35 + 1.63
6.68 - 0.93
1.49 0.74
1.00 1 0.82
15.38 + 1.22
35.65 2.53
46.86 : 3.32
PSPNet
ResNer101
17.09 + 1.32
6.70 0.75
1.21 4:11:000 0.39
0.67 + 0.47
15.61 + 0.78
2.12 36.83
45.22 + 3.05
ResNet101
14.34 +: 1.15
6.02 + 0.40
2:23 0.12
+ 0.00
13.21 + 0.56
34.30 1.80
45.02 = 1.27
33

---

20
100
D min: 9.59, max: 35.04
DeepLabbV3+-8B Restee0
DecpLabb3++ResNettt
min: 6.33, max: 30.30 mean: 16.08, std: 5.54
min: 3.51, max: 27.02
mean: 21.36, std:
mean: 12.33, std: 5.24
skew: 0.38, kurtosis: 0.05
skew: 0.38, kurtosis: -0.30
skew: 0.68, kurtosis: 0.27
DeepLabV3+-ResNell min: 4.65, max: 22.38 mean: 11.28, std: 4.28
DeepLabbV3--EficienNNee
DeepLabV34-MobiieNNee
min: 7.17, max: 33.83 mean: 16.86, std: 5.30
min: 1.36, max: 18.21 mean: 9.49, std: 4.21
skew: 0.90, kurtosis: 0.44
skew: 0.68, kurtosis: 0.88
skew: 0.02, kurtosis: -0.86
47
130
200
D3+pLabV3+ Mobile min: 8.26, max: 28.04
min: 6,86, max: 29.11
min: 13.57, max: 44.89
mean: 17.02, std: 4.58
mean: 17.29, std: 5.09
mean: 26.03, std: 7.25
skew: 0.17, kurtosis: -0.36
skew: 0.35, kurtosis: -0.23
skew: 0.57, kurtosis: -0.31
20
UPerNet-MrTBA min: 7.55, max: 35.92
SegFormer-
SegForeer-MiTB. min: 9.11, max: 36.02
min: 7.40, max: 38.94
mean: 21.76, std: 6.27
mean: 24.33, 6.59
mean: 23.75, std: 6.28
skew: 0.10, kurtosis: -0.42
skew: -0.33, kurtosis: -0.24
skew: -0.16, kurtosis: -0.72
DeepLabV3-ResNeet min: 6.49, max: 34.28
Neeeel-ResNet101 min: 6.68, max: 31.36
UNet-ResNet101 min: 5.67, max: 26.99
19.56, std: 6.52
mean: 19.79, std: 5.60
16.59, std: 5.19
skew: 0.19, kurtosis: -0.20
skew: 0.03, kurtosis: -0.46
skew: 0.16, kurtosis: -0.29
Figure 14: Histograms: Dark Zurich.
34

---

CeepLabV3++ReNNetllo GOPRO356 frame frame_000377 000378 rgbb anon (loU) 7.06).
DeepLab V3+- ResNeet::: GOPR0356 frame_ 00000900rgb_aaaaaea (loU! = 5.02).
COPR0356_frame__00
DeepLab GOPR0356_frame framee0000378 aat_:00008_rr__aaaa rgg__anonnnnnnoaaa (loU) = 2.45).
DeepLabV3+-ResNedd Re GOPR0356_free (0RR0356_fram___00
(ooU) = 3.17).
DeepLab EfficideNi- GOPR0356 _non (loU) = 4.38).
Figure 15: Worst-case images: Dark Zurich /3. Left: image. Middle: label. Right: prediction.
35

---

DeeplabV3t-MbbieNeeW:: GOPRO356 2a000366ffrae___0 Tamee_00073 ___bb_rb____ anon (loU; = 0.53).
DeepLabYY++-oobieevirzz GOPR0356 GOPRO356_frame__en__ 00000000000__oaammea (loU! 3.05).
Ueseeet- GOPR0356_frame, ttrame_000378 1117788rbb_aaaa rgb_anon (loU 4.09).
UPerNet-ConvNeXtt GOPR0356_frame (aaimee00039__a000 anon (loU? 10.12).
UPerNet-MiTB4: GOPR0356 frame
frame_ 0000330003333300000 rgb__ngor anon (loU) 5.96).
Figure 16: Worst-case images: Dark Zurich 2/3 3.. Left: image. Middle: label. Right: prediction.
36

---

SegFormer- MiTB4: GOPR0356 frame frame_000482 rgb anon (loU) 6.20).
SegFormer-MiTB2. GOPR0356fframe Brrame_00048__rg__aaa (loU! 7.29).
Deeptabb..-eeNNet001 GOPR0356_frame trame_000335_____ 0003755rgb_anon (loU) = 2.73).
PSPNet-ResNetlO1: GOPR0356_frame gaane__00375_rgie 100055_rbb_aa__ (loU) = 4.76).
UNet-ResNet1O1: GOPR0356 frame, 000378 rgb anon (loU) 4.90).
Figure 17: Worst-case images: Dark Zurich 3 3.. Left: image. Middle: label. Right: prediction.
37

---

G.4 Mapillary Vistas
Table 17: Results: Mapillary Vistas. Red: the best for each metric. Green: the worst for each metric.
miou!
mlou!!
miou!
mloU! mloyc! miot!
mlot/
mlouee
Method
Backhone
mist
mistyk3
mlotyk
mlot!"
62.47 + 0.04
56.01
0.04
44.61
0.15
38.47
0.21
LbVp3ab
Re
46.23 + 0.12
27.17 0.07
3.06
0.02
0.00 + 0.00
2.54
47.76
0.03
60.61 1: 0.03
53,99 1 0.05
42.58 = 0.12
36.67 = 0.12
DeepLabb33
ResNet50
43.31 + 0.03
25. 10 0.03
2.05 0.00
0.00 + 0.00
44.30 +: 0:13
56.03 0.14
90.76 0.03
58.35 + 0.05
51.47 +: 0.05
39.75 =2 0.17
$3.99 = 0.54
Deeplabb33
ResNe134
0.01
2.26
38.74 + 0.09
0.14 49.59
90.22 0.03
56.35 1 0.08
0.09 49.56
31.75 0.23
31.11 0.33
DeepLabb33
35.03 0.08
18.93
0.08
2.13 0.03
0.00 0.00
36.58 1 0.26
0.30 47.09
89.63 1: 0.08
DeepLabb2e
BfficienNetB00
0.15
2.72
37.11 + 0.13
20.38 0.03
2.37 0.04
0.00 + 0.00
39.89 + 0.39
0.34 50
90.64 0.06
DeepLabbV3
MohileNerV2
34.97 0.13
19.87 0.10
2.29 0.01
0.00
33.95 + 0.11
18.14 0.09
2.07 + 0.02
0.00 + 0.00
35.59 1: 0.05
45.60 0.09
89.33 0.03
58.28 =: 0.02
51.47 0.04
40.05
34.97 0.15
DeepLabb33
MobieViLV2
37.21 + 0.01
38.22 + 0.03
22.32 0.02
20 0.02
2.75 0.02
2.36 + 0.01
0.00 + 0.00
0.00 + 0.00
61.88 1 0.08
55.19 0.08
43.86 0.12
37.29 0.22
UParNer
ResNet1B1
46.03 + 0.08
45.15
26.60 0.D4
2.47
3.01 0.02
0.D0 + 0.D0
0.00
65.26 1 0.01
59.12 0.03
48.55 0.07
42.90 = 0.26
UPerNet
ComNNeXt
51.95 + 0.04
30.91 01.9152 0.06
3.53 1:33-00 0.02
0.00 + 0.00
55.68 + 0.12
0.14 67.63
92-41 0.04
64.48 1 0.01
58.29 1 0.02
47.66 12 0.15
42.39 a 0.37
UPerNeet
2.53
54.22 + 0.09
66.05 0.11
92.41 = 0.01
64.17 1 0.02
57.07 0.02
47.43 Σ 0.00
42.50 0.27
SogHormer
49.02 1: 0.08
27.
0.08
2.87 0.08
0.33 0.47
54.10 + 0.04
65.96 0.13
92.38 - 0.03
Seyfoormer
MITB2
45.75 + 0.04
+ 0.07
25.18 0.05
2.53 0.02
3.15
0.00 0.00
51.02 + 0.13
62.55 0.24
91.91 0.03
DespLabv
ResNer101
46,00 1: 0.08
47.13 +: 0.02
27. 44 0.01
25. 0.04
3.06
2.55
0.01
0.02
0.00 1 0.00
0.00
0.00
49.32 1: 0.33
61.35 0.27
91.41 0.03
61.85 0.07
0.08
0.37
PSPNet
ResNet101
46.85 0.18
27.19 0.14
3.05 0.03
0.00 + 0.00
45.85 - 0.12
25.
0.09
2.54 0.03
0.00 + 0.00
57.29 1: 0.20
50.30
0.28
38.45 0.38
31.93 0.38
UNet
ResNet101
35.34 1: 0.32
36.03 + 0.33
20.81 0.28
0.27
2.12 0.05
2.36 0.08
0.00 + 0.00
0.00 0.00
33.30
0.22
38

---

300
102
100
80
02-
60
89
60
40
40
40
00
20
300
Deepl.abV3+-ResNete min: 30.46, max: 91.24 mean: 62.47, std: 8.25
DeepLab R min: 23.90, max: 90.44
min: 25.47, max: 91.59
mean: 60.61, std: 8.45
mean: 58.35, std: 8.77
skew: -0.05, kurtosis: 0.33
skew: -0.03, kurtosis: 0.31
skew: -0.01, kurtosis: 0.21
200
DeepLabV34-ResNee min: 25.02, max: 90.91 mean: 56.55, std: 8.92
DeepLab Seeplaab3---fficieenneeBBB
DeepLabV344Mobbille
min: 31.16, max: 90.16
min: 24.55, max: 88.36 mean: 55.46, std: 8.92
mean: 58.28, std: 8.72
skew: 0.02, kurtosis: 0.24
skew: 0.05, kurtosis: 0.14
skew: 0.04, kurtosis: 0.24
60-
20
2-
20
42
60
122
20
200
DoepLabV3+-Mobilee min: 29.06. max: 91.82
OPerNet-ConvNeeX min: 33.91, max: 92.64
min: 29 29.68, max: 92.29
mean: 58.28, std: 8.73
mean: 61 61.68, std: 8.30
mean: 65.26, std: 7.79
skew: 0.07, kurtosis: 0.21
skew: -0.02, kurtosis: 0.35
skew: -0.07, kurtosis: 0.22
300
10c
:00
30
82
80
30
60
45-
20
20
UPerNet-MiTB4 min: 39.82, max: 92.88
SegFormer-MiTW
SegForeer-MFTB2
min: 38.01, max: 91.99
min: 32.94, max: 91.26
mean: 64.48, std: 7.89
mean: 64.17, std: 7.90
mean: 62. std: 8.12
skew: 0.00, kurtosis: 0.20
skew: 0.01, kurtosis: 0.20
skew: 0.06, kurtosis: 0.24
25
CeepLabVV-ResNettll min: 29.29, max: 91.85
PSPNet-ResNetlO1 min: 27.73, max: 91.76
UNet-ResNet101
min: 21.06, max: 89.13
62.33, std: 8.19
mean: 61.85, std: 8.17
$7.29, std: 8.88
skewness: -0.02, kurtosis: 0.38
skew: -0.03, kurtosis: 0.38
skew: -0.03, kurtosis: 0.27
Figure 18: Histograms: Mapillary Vistas.
39

---

DeepLabV3+ V3r--eeetRRRR1111111 bJSGIVPKHWUUAPMPFI19I A (loU)) 28.85).
D3+ ResNett: DISGIVPKHHNUUAAMIFIHH A (loU) 23.41).
(3 bYSGIPPKHW-UAApMiFIH A (loU! 23.42).
(3pplabb33+ReNNetlle bYSGIVVKKENUUAAAMIPPHHAAA A (loU! 24.09).
WUSGIVPKHHUUAPPPFFF DeepLabVV++-EffiieeetteBBB
A (loU! 26.10).
Figure 19: Worst-case images: Mapillary Vistas 1 3. Left: image. Middle: label. Right: prediction.
40

---

D MobileNetV2: 2UUUEESNN (loU) 24.02).
DeepLab 2UUUUBSYGGXCCCWOHHWYYA (IoU) 28.26).
UPerNet-ResNetlO:: bYSCIVPKIH.UU9AMMFII9 A (loU) = 27.10).
UPeNNet-ConNNeXt bVSCIVVKKHHUU9APMFF99..A (loU) 33.15).
UPerNet-MiTBA: HIWWWI_pxNjQ7HHI _Kg (loU) = 37.43).
Figure 20: Worst-case images: Mapillary Vistas 2 Left: image. Middle: label. Right: prediction.
41

---

SegFormer-MiTB4 I-RM@p4XIllayyLXXNPAgg (loU) 36.03).
SegFormer- MiTBB: ZZYHIarrbniSmmKKKHHWaAa (loU) 30.82).
DeepLabV3-ResNette bISGIVPKHWUUAApiFFI9h A (IoU) = 25.70).
ReeNet-ResNet101: bYSGINPKIINUUAAAMFF99..AA A (loU! 26.92).
UNet-ResNet101: buSSGIVPKKHWUUAAMMII9 A (loU) 17.85).
Figure 21: Worst-case images: Mapillary Vistas 3 / 3. Left: image. Middle: label. Right: prediction.
42

---

G.5
CamVid
Table 18: Results: CamVid. Red: the best for each metric. Green: the worst for each metric.
miot!! miou mlotpo
mlot! miou
miot!! mtog
mlot! miou
Method
Backhone
mAde
79.97 + 0.12
74.48
0.20
65.19
0.18
62:46
0.06
DeepLabV3e
ResNet101
79.48
67.72
37.01
0.94
25.00
0.82
79.22 + 0.11
73.58
0.22
63.77 = 0.61
60.59 = 0.50
Dosplaaby33
ResNet50
76.69 + 0.16
0.27 66.50
35.00 = 0.68
23.33
0.94
79.16 + 0.24
0.60 72.92
0.44
62.06 1.11
58.66 1.84
DeepLabV3e
ResNet34
78.71 +: 0.32
66.
33.63 +: 1.15
22:33 1: 1.70
95.42
78.12 0.03
0.05
0.15
57.11
DeepLabb34
8esNetl
77.55 0.03
64.44 0.04
32. +: 0.27
18.67
1:25
80.79 0.13
87.89 0.11
95.19 0.01
DeepLabbV3
EfficeenNeetill
78.15 + 0.32
65.59
0.40 88.33
34.51 0.67
21.00
0.82
80.97 + 0.38
0.30
95.17 0.08
DeepLabV3e
MobileNaaV2
77.39 + 0.12
0.21 87.64
32.39 0.35
20.33
80.30 + 0.16
0.03
95.08 = 0.09
61.70
DeepLabV3e
0.00
80.62 + 0.18
88.06
0.26
94.91 0.26
79.76 1 0.01
74.22
0.08
65.47 0.07
63.18 0.34
83,00 1. 0.13
89.17
0.08
95.74
0.02
80.02 + 0.13
75.40
0.18
0.59 66.30
63.36
0.02
UPPrrer
ComoNcXi
84.01 0.10
0.12 90.61
0.04 95.96
80.53 + 0.09
75.19 + 0.20
65.56 0.76
62.82
1:32
UPerNeet
MITB4
80.09 1: 0.13
68.48
0.65 31.77
24.33
1:25
80.04 + 0.13
74.61 + 0.19
64.95 = 0.44
61.53
0.58
SegFormer
MMTB4
79.52 = 0.12
0.19 67.88
36.94 0.55
23.00
0.82
79.31 3 0.05
73.45 0.10
64 0.27
62.01 1.04
Seyfoorer
MiTD2
76.84 1: 0.08
66.73 0.13
35.77 : 0.40
23.00 2: 0.82
78.81
0.10
64.12 0.37
61.73
DesplabbV33
ResNet101
78.36 +: 0.02
66.63 0.05
35. 0.39
22.67
2.05
82.43 + 0.08
89.15 0.05
95.50 0.04
PSPNet
ResNer101
78.12 0.14
66.32
0.26
33.74 0.47
23.67
0.47
82.13 + 0.12
89.14 0.01
95.46 + 0.05
ResNet101
79.19 =: 0.35
0.59
35.29 0.64
23.00 1.41
82.84 -: 0.31
89.26 0.26
95.63 = 0.03
43

---

12
12
20
300
Deeplabb33+RRsNeell min: 63.47, max: 92.76 mean: 79.97. std:
RessLabb
min: 60.44, max: 92.38 mean: 79.22, std: 7.02
min: 59.57, max: 92.39 mean: 79.15, std: 7.53
skew: -0.25, kurtosis: -0.66
skew: -0.16, kurtosis: -0.58
skew: -0.32, kurtosis: -0.61
10
12
20
DeepLabVV+-ResNelle
DeepLabV3+-Eficienttee
min: 57.14, max: 92.45 mean: 78.12, std: 8.04
min: 57.27, max: 92.33
min: 56.03, max: 92.20
mean: 78.62, std: 7.73
mean: 77.89, std: 8.07
skew: -0.21, kurtosis: -0.79
skew: -0.23, kurtosis: -0.74
skew: -0.19, kurtosis: -0.81
20
42
20 JPPrrNetConvNeeX min: 62.46, max: 92.34 mean: 80.52, std: 6.27
200
DeepLadV3+-MooleeWiTTV min: 59,59, max: 92.39
MobileViTV2
ResNet101
min: 62.88, max: 92.61
mean: 78.58, std: 7.99
mean: 79.77, std: 6.84
skew: -0.26, kurtosis: -0.83
skew: 18,, kurtosis: -0.74
skew: -0.36, kurtosis: -0.42
12
10.
20
UPerNet-MiTB4 min: 62.45, max: 92.92
SegFormer-Mi MiTB4
SegFormer-MiTB:
min: 62. max: 92.66 62.64 mean: 80.04, 6.72
min: 61.87, max: 92.70
mean: 80.54. std:
mean: 79.31, std: 7.22
skew: -0.30, kurtosis: -0.43
skew: 10 kurtosis: -0.45
skew: -0.27, kurtosis: -0.71
20.
15-
10-
CeapLaab3-RReNeettiee min: 61.58, max: 92.22
NSPNet-ResNetlO1 min: 61.96. max: 91.91
UNet-ResNet101 min: 62.01. max: 92.59
78,81. std: 6.78
mean: 78.63, std: 6.82
79.69, std: 7.14
skew: -0.22, kurtosis: -0.55
skew; -0.16, kurtosis: -0.62
skew: -0.24, kurtosis: -0.74
Figure 22: Histograms: Cam Vid.
44

---

DeepLabV3+ Neeel: 0001TP -004770 (loU 59.90).
MeppLabV3+-ReNNettth (loU) 58.81).
Seq05VD 103570 (loU) = 54.26).
DeepLab V Seg05VD_103 103570 (loU) = 56.06).
DeepLab DeepLabV3+-EficceennNeeee EfficifiieiiieeeBo 0001TP 007560 (loU! 53.86).
Figure 23: Worst-case images: Cam Vid /3. Left: image. Middle: label. Right: prediction.
45

---

Vibilaab 0001TP 007470 (loU) 0001700007400
52.72).
D VITv2: 000ITP 000ITP_006930 (IoU) = 57.51).
Seq05VD 10024 (loU) = 62.33).
"PerNet-ConvNeen Seq05VD 100270 (ooU) 61.81).
UPerNet-MiTB4: Seg05VD f00240 (loU! 59.61).
Figure 24: Worst-case images: Cam Vid 2 3. Left: image. Middle: label. Right: prediction.
46

---

Seg Mooomer-MiTBA. 0001TP_007477 (loU) 59.96).
Seg Firmerr-MiBBB: 000ITP 000560 (loU) 58.83).
f00200 MeppLabb3-ResNaaaee Seq05VD 20270 (loU) = 60.78).
Seg05VD 103570 (loU) = 60.58).
Nestel1: 0001TP_007470 (loU) 60.12).
Figure 25: Worst-case images: Cam Vid 3 3. Left: image. Middle: label. Right: prediction.
47

---

G.6 ADE20K
Table 19: Results: ADE20K. Red: the best for each metric. Green: the worst for each metric.
miot!! miou
mlot!!
miot!! mlou mlou
mlot! miol mlou
Method
Backhone
miou
mlou mloU
mlou
mákec
Acc
56.79 I 0.04
45.
0.05
25.18
0.22
16.74 0.87
DeepLabb3e
45.84 0.30
22.72 0.35
1.78 0.30
1.00 0.00
46.32 + 0.33
60.96 + 0.51
81.25 = 0.03
1: 57.15
0.14
23.71
15.46 0.42
DeepLabV2e
ResNer50
43.73 + 0.39
21.16 0.11
1.48
1.40 4.10
0.67 + 0.47
0.47
44.14 + 0.31
59.41
0.51
80.16 0.10
55.58
DeepLabbV3
ResNer34
40.65 =: 0.09
40.02 1: 0.14
18.97 0.09
0.02
1.63 0.10
1.64 0.100
1.00 1 0.00
+ 0.00
40.50 +: 0.17
0.23 54.94
79.49
53.78 = 37.98 + 0.17
0.11
0.80
DeepLabV2e
ResNet18
37.02 0.17
17.43
18.31 = 0.09
0.11
1.40 + 0.09
1.41 0.98
1.00 + U.D0I
1.00 + 0.00
54.17 0.16
40.90
0.17
19:40 0.32
DespLabb33
EffiiietNeeBB
38.63 0.13
18.49 0.11
1.32 + 0.07
1.00 + U.D0
38.23 1: 0.07
1.30 0.08
1.00 0.00
52.85 + 0.13
39.92 2: 0.12
19.94 0.18
11.25 = 0.29
DeepLabb3e
MobileNetV2
36.42 + 0.04
17.49 0.09
1.37
1.00 + 0.00
36.85
55.05 1 0.11
42.01 1 0.14
21.73
14:32 0.44
DeepLabbV3
MobieViTV2
1.22
41.36 +: 0.43
55.07 0.52
79.34 = 0.06
57.85 + 0.01
45.
0.07
24.22 2 0.30
14.91 0.85
0.11
44.85 0.27
21.99 0.25
L.77 0.16
1.00 0.00
45.89 + 0.11
59.65
0.35
80.97 - 0.06
UPerNeet
Com-NeXi
47.58
0.17
24.08
0.08
2.14
1.98 0.03
1.00 0.00
01.08 + 0.42
63.34
0.50
83.05 0.07
UPerNet
MITB4
47.97 + 0.13
48.58 0.10
25.47
24.49
1: 0.15
0.08
2.16 0.12
2.36 0.09
1.67 0.47
1.00 1 0.00
50.25 + 0.15
0.22 64.13
83.23 0.12
+ 0.07
0.08
Seyfoorerr
M1TB4
48.10 +: 0.09
25.29
0.12
2.32 0.16
1.33 + 0.47
47.47 + 0.12
0.16 24.25
2.09 0.15
1.00 + 0.00
56.28 1: 0.03
45.72 0.06
25.45 0.02
1T.02 0.21
SogHorrer
MI182
45.18 1: 0.07
45.79 4: 0.09
23.59 0.18
22.71 0.118
2.37 0.18
2.52 0.18
1.67 0.47
1.67 0.47
47.18
81.72
67.27 +: 0.06
44.60 0.09
24.09 = 0.23
15.67 = 0.43
DespLabV3
ResNet101
44.73 4: 0.11
22.72
0.19
1.89 0.17
1.00 + 0.D0
46.08 = 0.08
60.20
0.25
$1.07 0.08
57.37 + 0.06
$4.90
0.D3
24.79 = 0.18
16.81 = 0.36
ResNet101
44.86
1.79 0.14
0.00
0.00
45.69 0.33
59.42 0.57
$1.00 0.14
56.86 4.000. 0.05
44.15
24.05 0.31
14.84 0.48
0.02
19.83 0.05
1.42 0.02
1.00 1 0.00
40.78 -: 0.07
54.58 0.15
79.24 0.04
48

---

300
102
100
80
02-
60
89
60
40
40
20-
0
70
100
min: 11.20, max: 99.80
min: 11.78, max: 99.93
min: 1.88, max: 99.78
mean: 58.79, std: 16.13
mean: 57.15, std: 16.14
mean: 55.58, std: 16.48
skew: 0.03, kurtosis: -0.19
skew: 0.08, kurtosis: -0.15
skew: 0.09, kurtosis: -0.19
200
DeepLabV34-ResNee
DeepLabV344Mobieee
min: 2.20, max: 99.74
min: 3.19, max: 99.79
min: 0.00, max: 99.84
mean: 53.78, std: 16.44
mean: 54.17, std: 16.60
mean: 52.85, std: 16.57
skew: 0.17, kurtosis: -0.04
skew: 0.09, kurtosis: -0.12
skew: 0.19, kurtosis: -0.08
60-
40
20
2-
20
42
60
132
20
200
V3+ DeepLaV3+-MobleeViVV> min: 8.10, max: 99.78
UPerNet-ResNeee
min: 6.08, max: 99.69
min: 11.09, max: 99.89
mean: 55.05, std: 16.55
mean: 57.85, std: 16.02
mean: 60.62, std: 15.83
skew: 0.16, kurtosis: -0.21
skew: 0.07, kurtosis: -0.12
skew: 0.06, kurtosis: -0.29
300
102
:00
30
82
80
30
62
60
26
70
UPerNettMFTBA min: 6.72, max: 99.48
SegFormer MiTB4
SegFormer-MiTB:
min: 7.81, max: 99.62
min: 4. 4.64, 99.75
meam: 60.64, std: 16.17
mean: 60.29, std: 16.02
mean: 58,27, std: 15.97
skew: 0.00, kurtosis: -0.22
skew: 0.04, kurtosis: -0.16
skew: 0.09, kurtosis: -0.16
CegpLaab33RReNeetllee min: 11.16 max: 99.68
PSPNet-ResNetIO1 min: 11.51, max: 99.63
UNel-ResNet101 min: 1.31, max: 99.70
mean: 57.27, std: 16.14
57.37, std: 15.87
mean: 56.86, std: 16.17
skew: 0010, kurtosis: -0.12
skew: 0.12, kurtosis: -0.14
skew; 0.09, kurtosis: -0.17
Figure 26: Histograms: ADE20K.
49

---

V ADE val_00006626 (loU) 5.99).
DeepLabV3++ ADE val. 00000223 (loU! 6.91).
DeepLabW3t-ResNet34 ADE val 00001600 (loU! 0.00).
DeepLabV3+-ReNNeile ADE val 00000263 (IoU! 0.01).
BrnilaabV+--fficiennnneen ADE nol_00000263 (loU) 0.00).
Figure 27: Worst-case images: ADE20K /3. Left: image. Middle: label. Right: prediction.
50

---

DeeLabY3+ ADE val 00001600000 (loU) = 0.00).
DeepLabV3=-Mobbe-vVTV2 ADE val. .00000050 (loU) 4.27).
UPerNet-Res ADE val_ nll_00012 (IoU) = 0.06).
ADE val 00001197 (IoU) = 0.00).
UPerNet-MiTB4: ADE_ o0l_09001951 (loU) 0.58).
Figure 28: Worst-case images: ADE20K 2 3. Left: image. Middle: label. Right: prediction.
51

---

Se.Former-MiTBA. ADE val 000016000000000 (loU) = 1.14).
SegFormer-MiTB2: ADE val. 00000054 (loU) 2.21).
DecpLabV3-ResNeetue ADE val 00000626 (loU? 6.21).
PSPNet-ResNetOO1: ADE mel000099996 (loU) = 4.81).
UNet-ResNetlOl: ADE val 00000263 (loU) 0.00).
Figure 29: Worst-case images: ADE20K 3 3. Left: image. Middle: label. Right: prediction.
52

---

G.7 COCO-Stuff
Table 20: Results: COCO-Stuff. Red: the best for each metric. Green: the worst for each metric.
miot!! mioy mloapo
mlot!! miou
miot!! mtou
mlot! miou
Method
Backhone
mAde
48.71 + 0.06
35. 26
0.09
17.07
=: 0.22
8.45 + 0.39
DeepLabV3e
ResNet101
41.93 1: 0.04
22.48
0.04
1.33 0.07
0.00 0.00
46.91 + 0.04
33.39 0.06
15.40 0.23
7.34 0.53
DospLaaby33
ResNet50
39.66 + 0.04
20.75 0.08
1.19 0.04
0.00 + 0.00
43.32 + 0.04
29.54
0.08
11.76 =: 0:20
3.76
0.23
DeepLabV3e
ResNet34
34.39 +: 0.06
16.77 1: 0.03
0.56 1: 0.04
0.00 1: 0.00
+
66.12
41.60 = 0.05
0.15
DeepLabb34
8
32.33 1: 0.04
15.43 0.08
0.38 1: 0.03
0.00 1 0.00
34.18 + 0.18
47.90 0.17
64.78 0.05
DeepLabbV3
EffcieenNeebil
33.64 + 0.24
16.22 0.19
0.35 0.08
0.00 + 0.00
35.63 + 0.22
50.23 0.32
65.59 0.16
DeepLabV3e
MobileNat22
30.95
0.05
0.04
0.36 0.03
0.00
32.65 + 0.17
46,
0.09
63.87 = 0.08
44.54
DeepLabV3e
38.99 + 0.03
53.15 0.12
67.01 = 0.03
46.16 0.10
34.73 0.13
16.69 0.24
8.20 1 0.78
1.31
43.87
0.07
58.57
0.08
69.47 0.03
51.34 + 0.04
38.06
0.02
19.77 = 0.05
10.99 0.07
UPerrer
ComoNcXi
48.51 + 0.03
62.61
0.06
71.78 0.04
50.74 + 0.03
37.23 + 0.D5
19.11 t: 0.20
11.30
0.45
UPerNeet
MITB4
43.69 1: 0.07
23.62
0.05
1.52
0.00
0.00
50.46 + 0.17
37.02
0.17
19.02 0.03
11.52 = 0.13
SegFormer
M1TB4
43.33 = 0.24
23.37
0.20
1.47 0.08
0.00 + 0.00
48.74
0.02
35.28
0.01
17.65 0.06
10:12
0.37
Seyfoorer
MITD2
41.36 1: 0.00
21.89 0.04
1.46:10.00
0.00 1 0.00
0.02
48.21
34.91
17.12
DesplabbV33
ResNet101
41.37:10..1
21.96 0.07
L.34
0.00 + 0.00
14.29.00.0
59.29 0.14
69.63 0.09
PSPNet
ResNer101
41.30 0.04
21.94 0.07
1.30 0.07
0.00 + 0.00
44.19 + 0.05
59.04 0.01
69.51 +: 0.14
0.91
ResNet101
37.76 0.19
19.43 0.26
0.97 0.20
1 0.00
37.14 0.34
01.43.0..
64.68 = 0.14
53

---

700
202
200
230
135
130
200
102
200
20
100
DeepLab V3-
DecpLabV3++ResNee Resseett
min: 0.00, max: 99.89
min: 0.00, max: 99,94
min: 0.00, max: 99.51
mean: 48.71, std: 18.06
mean: 46,91, std: 18.26
mean: 43.31, std: 18.72
skew: 0.50, kurtosis: -0.07
skew: 0.54, kurtosis: -0.03
skew: 0.63, kurtosis: 0.08
DeepLabV34-ResNelll min: 0.00, max: 99.27
D Di-i.abbV---iiiieenneeBBBB
DeepLabV34- DeeepaaYY24-MobieNeeee
min: 0.00, max: 99.87
min: 0.00, max: 99.11
mean: 41.60, std: 18.82
mean: 42.60, std: 18.67
mean: 40.36, std: 18.87
skew: 0.68, kurtosis: 0.15
skew: 0.67, kurtosis: 0.09
skew: 0.71, kurtosis: 0.21
250
152
200
102
50
50
0
2-
20
42
120
20
200
V3+ MrbilleVi min: 0.00, max: 99.78
110Ne
min: 0.00, max: 99.97
min: 0.03, max: 99.94 99.94
mean: 44.54, std: 18.63
mean: 48. 16,16,,666 std: 18.07
mean: 51.34, std: 17.72
skew: 0.64, kurtosis: 0.05
skew: 0.52, kurtosis: -0.06
skew: 0.43, kurtosis: -0.15
200
201
200
230
155-
250
200
107
200
70
UPerNettMiTBB
SegFormer- MiT44
SegFormer-MiTB
min: 0.00, max: 99.96
min: 0.00, max: 99.99
min: 0.00, max: 99.76
mean: 50.74, std: 18.02
mean: 50.46. std: 17.94
mean: 48.75, std: 18.16
skew: 0.46, kurtosis: -0.20
skew: 0.46, kurtosis: -0.19
skew: 0.52, kurtosis: -0.11
230
PSPNet-ResNetIO0 min: 0.00, max: 99.98
ONtl-ResNetl01
min: 0.00, max: 99.94
min: 0.00, max: 99.05
48.21, std: 17.91
mean: 48.05, std: 17.88
45.80. std: 17.90
skew: 0.52, kurtosis: -0.07
skew: 0.53, kurtosis: -0.05
skew: 0.56, kurtosis: 0.02
Figure 30: Histograms: COCO-Stuff.
54

---

VecpLabV3+ReeNetlllu 40000004440 (IoU) 0.00).
DeepLabV3+ 000000064574 (IoU) 0.00).
DeepLab 000000027932 (IoU) = 0.00).
BetlaabV3+RReNetII. 000000064574 (IoU) = 0.00).
DeepLab DeepLablV-t-EfficeeetteBBBB 000000064574 (loU! 0.00).
Figure 31: Worst-case images: COCO-Stuff /3. Left: image. Middle: label. Right: prediction.
55

---

MoohealW 000000027932 (IoU) 0.00).
DeepLabY++-MobbleiiVV: 000000064574 (IoU) 0.00).
UPerNet-Reed ReeNet- 000000064574 (loU) 0.00).
UPenNet-Com/NeXt: 000000064574 (loU) = 0.00).
UPerNet-MiTB4: 000000064574 (loU) 0.00).
Figure 32: Worst-case images: COCO-Stuff 2/ 3. Left: image. Middle: label. Right: prediction.
56

---

SegFormer-MiTB4: 000000064574 (loU) 0.00).
SegFormer-MiTB2: 000000064574 (loU! 0.00).
DeepLabV3-Reshhe ResNet101: 000000053529 (IoU! = 0.00).
PSPNet-ResNer101: 000000053529 (IoU) = 0.00).
UNet-ResNet101: 000000064574 (loU) 0.00).
Figure 33: Worst-case images: COCO-Stuff 3 3.. Left: image. Middle: label. Right: prediction.
57

---

G.8
PASCAL VOC
Table 21: Results: PASCAL VOC. Red: the best for each metric. Green: the worst for each metric.
miot!! miot mloapo
mlot!! miou
miot!! mlou
mlot! miou
Method
Rockhone
mAde
86.95 + 0.06
75. 78
0.09
45.67
0.27
31.61 # 0.41
DeepLabV3e
ResNet101
79.07
62.71
17.32
1.27
1.67 0.47
85.80 + 0.12
74.12.0..1
45.54 0.41
30.95 = 0.42
Dosplaaby33
ResNet50
77.56 + 0.03
60.20 0.21
13.03 = 1.65
2.67 + 1.25
79.71
82.96 + 0.09
69.47 1 0.22
39.79 0.54
24:25 0.82
DeepLabV3e
ResNet34
73.16 1: 0.19
52.11 0.38
5.12::00.32
0.33 1: 0.47
94.64
81.95
0.18
38.89
0.37
DeepLabb34
8
71.63:1
50.42 0.22
4.04 1: 0.2.
0.00 E 0.00
73.30 0.22
83.86 0.23
93.16 0.04
DeepLabbV3
EfficeenNeetill
72.06 + 0.13
51.56 0.14
6.73 0.51
0.00 + 0.00
73.58 + 0.78
0.47 83.83
93.82 + 0.16
DeepLabV3e
MobileNetV2
70.06 + 0.10
48 38 0.25
3.37 0.44
0.67 0.47
72.05 + 0.24
0.41 82.75
92.89 = 0.05
DeepLabV3e
78.32
76.75 + 0.20
85.34 0.18
93.95 = 0.07
86.32 1 0.03
0.08 74.03
44.76 1 0.25
29.14 0.53
80.50 +: 0.15
0.07 87.70
95.19 0.04
86.30 + 0.11
0.27 78.14
49.92
0.64
33.31 0.64
UPPrrer
ComoNcXi
1.25
83.48
0.08
0.17 89.01
95.95
0.03
87.76 + 0.12
76,98
0.27
0.72 47.25
$2.96 # 0.19
UPerNeet
MITB4
80.02
0.54 63.68
16.30
3.00 1.41
87.54 + 0.15
76.74 0.28
46.88 = 0.32
30.82 = 0.82
SegFormer
MMTB4
79.80 + 0.30
63.89
0.51
20.05 1.67
4.67 + 1.70
86,26 1: 0.10
74.51
0.18
43.45 0.31
25.08 0.05
Seyfoorer
MiTD2
77.57 1: 0.23
60.23 1: 0.15
11.76 : 0.50
1.33 +:: 0.47
86.32 0.03
45.09
31.52
DesplabbV33
ResNet101
77.93 0.09
61.43 1: 0.21
17.67 : 0.67
5.33 + 0.94
81.24 0.09
88.27 0.23
95.30 0.01
PSPNet
ResNer101
17.84400.0
61.11 0.28
15.95 1.32
3.33 + 1.70
81.10 + 0.16
88
0.23
95.25
0.05
ResNet101
76.85
0.24
16.00
1.34
3.33 1.25
74.94 1.844.000. 0.33
83.62 2: 0.20
94.06 = 0.09
58

---

200
202
200
230
195
230
200
102
100
00
20
100
DeepLab V3-
min: 16.20, max: 99.93
min: 12.35, max: 99.96
min: 11.71, max: 99.85
mean: 86.95, std: 13.44
mean: 85.79, std: 13.97
mean: 82.96, std: 15.73
skew: -172, kurtosis: 2.70
skew: -1.54, kurtosis: 2.00
skew: -1.32, kurtosis: 1.25
DeepLabV3+RResNer min: 9.81, max: 99.84
DeepLabV344Mobilee
min: 0.64, max: 99.80
min: 9.76, max: 99.62
mean: 81.95, std: 15.98
mean: 82.22, std: 15.72
mean: 81.00, std: 16.48
skew: -1.24, kurtosis: 1.06
skew: -1333, kurtosis: 1.69
skew: -1.16, kurtosis: 0.80
250
152
200
102
50
50
2-
20
42
80
130
20
200
DoepLabV3+-Mobiie Mob-MlobbbeeV
CPerNet-ConvNee.
min: 11.81, max: 99.92 mean: 83,41, std: 15.61
min: 11.58, max: 99.95
min: 12.15, max: 99.89
mean: 86.52, std: 13.86
mean: 88.30, std: 12.48
skew: -1.35, kurtosis: 1. 1.24
skew: -1.74, kurtosis: 3.10
skew: -1.96, kurtosis: 4.20
200
201
200
230
155-
250
200
102
200
70
UPerNet-MiTBA min: 24.94, 99.97
SegFormer- MiTB44
RegFFomer-MiTB3
min: 8.19, max: 99.97
min: 9.36, 99.94
meam: 87.76. std: 12.97
mean: 87,54, std: 13.06
mean: 86.26, std: 14.29
skew: -1880, kurtosis: 2.98
skew: -1885, kurtosis: 3.54
skew: -1.75, kurtosis: 2.94
20
25
CeepLabbV-ReeNetlllle min: 13.19. max: 99.68
PSPNet-ResNetl01 min: 17.11, max: 99.70
UNel-ResNetl01
min: 7.14, max: 99.93
mean: 86.32, std: 13.71
86.23, std: 13.76
85.58, std: 14.16
skew: -1.71, kurtosis: 2.77
skew: -1.67, kurtosis: 2.56
skew: -1,59, kurtosis: 2.41
Figure 34: Histograms: PASCAL VOC.
59

---

C 2010_001322 (loU) 9.91).
A
DeepLabV3+ReeNett0: 2010_004951 (loU) 1.74).
DeepLab 13t--RReeeee44 2007_007498 (loU) = 6.92).
DecpLabV3+ ResNeel 2010_00490 004951 (loU) = 4.81).
1
DeepLab V Efficiiennettt 2010 004951 (loU! 0.35).
Figure 35: Worst-case images: PASCAL VOC 1 /3. Left: image. Middle: label. Right: prediction.
60

---

MeppLaaY++MobiiNeeV.. (IoU) 5.63).
DeepLabV3+-Mobieeiit:: 2000 008051 (loU) 4.42).
1
MeNNett-esNNt101: 2010 004951 (IoU) = 1.57).
2010_004951
UPerNet-ConvNee 2010 001327 (loU! 9.69).
UPerNet-MiTB4: 0011_003182 (loU) 7.39).
Figure 36: Worst-case images: PASCAL VOC 2 /3 Left: image. Middle: label. Right: prediction.
61

---

ScgFormer-MiTB4: 2011_003188 (loU) 7.36).
MigFo: 2011_000111 2011_0031221100 (loU) 7.34).
DecpLabbV3ResNee110:: 2008 008051 (IoU! = 4.72).
ESPNet-ResNetlone 2008 000763 (loU) = 9.23).
UNst-ResNe1101: 1008500051111 (loU) 5.04).
Figure 37: Worst-case images: PASCAL VOC 3 /3. Left: image. Middle: label. Right: prediction.
62

---

G.9 PASCAL Context
Table 22: Results: PASCAL Context. Red: the best for each metric. Green: the worst for each metric.
miot!! miou mlotto
mlot! miot
miot!! mlou
mlot! miol
Method
Rackhone
mAde
67.16 = 0.04
51.85
0.03
27.15
=: 0.19
14.81 # 1.28 0.00 0.00
DeepLabV3e
ResNet101
55.89 1: 0.07
34.93
0.08
3.24
0.29
65.42 + 0.07
49.91 0.08
24.97 0.14
12.35 = 0.75
Dosplaaby33
ResNet50
53.95 = 0.18
33.12
2.57 0.05
0.00 + 0.00
61.92 + 0.13
45.52 1 0.119
18.35 =: 0.63
3.80
0.97
DeepLabV3e
ResNet34
49.49 1: 0.20
27.84 0.10
0.40 0.004
0.00 1 0.00
77.45
60.39 = 47.37 1: 0.26
0.12
0.40
DeepLabb34
8esNetl
26.
0.28
0.45 0.00
0.00 + 0.00
46.59 +: 0.33
58.58 0.39
76.40 0.12
DeepLabbV3
EfficeenNeetil
48.13 + 0.05
26.66 0.11
D.42 + 0.08
0.00 + 0.00
47.90 + 0.18
60.02 0.04
77.31 + 0.16
DeepLabV3e
MobileNat22
+ 46.00
24.87
0.23
0.22 0.04
0.00
45.64 + 0.11
57.77 2 0.21
75.72 = 0.08
02.48
DeepLabV3e
49.96
50.38 + 0.36
61.95 0.40
78.63 = 0-21
66,42 1 0.10
0.09 50.85
25.75 0.36
13.32 0.92
55.65 0.10
0.17 66.19
81 0.06
70.36 + 0.09
0.07 55.78
31.27 = 0.29
17.88 0.80
UPerrer
ComoNcXi
G1.33 + 0.06
0.06
84.09 0.06
09.83 + 0.03
54.57 + 0.02
29.50 =: 0.20
16.10 0.36
UPerNeet
MITB4
1: 58.55
37.15
0.14
3.83 0.34
0.00 0.00
69.54 + 0.05
54.40 0.07
29.72 = 0.23
16.11 = 0.90
SegFormer
M1TB4
58.29 + 0.14
36.97 0.22
3.63 0.28
0.00 + 0.00
67.24 0.03
51.99 0.03
26.39 0.07
12.76 0:30
Seyfoorer
MiTD2
55.26 1 0.04
34.02 1 0.04
2.06 1 0.01
0.00 1 0.00
0.11
0.46
0.71
DesplabbV33
ResNet101
54.33 1: 0.17
33.94 14.41:0.1 0.13
2.96 1: 0.
0.00 + 0.00
55.80 + 0.08
0.22 66.
81.41 0.07
PSPNet
ResNer101
54.33 +: 0.03
33.89 0.08
3.03 0.10
0.00 + 0.00
55.52 + 0.11
66.03 0.07
$1.23 +: 0.06
ResNet101
53.88 1: 0.15
33.07 0.09
2.50 1:00::00 0.07
51.11 0.33
63.92 0.41
78.74 0.08
63

---

200
202
200
230
155
230
200
102
200
70
100
hepplabb3+-Reseette
V ResNet50
min: 0.00, max: 100.00
min: 0.22, max: 100.00
min: 0.00, max: 100.00
mean: 67.16, std: 18.97
mean: 65.42, std: 19.31
mean: 61.92, std: 20.22
skew: -0,12, kurtosis: -0.55
skew: -0.08, kurtosis: -0.55
skew: -0.07, kurtosis: -0.38
DeepLabV3+RResNee min: 0.00, max: 100.00 mean: 60.39, std: 20.48
min: 0.00, max: 100.00 mean: 60.96, std: 20.41
min: 0.00, max: 100.00
mean: 59.30, std: 20.42
skew: -0.01, kurtosis: -0.41
skew: 0.00, kurtosis: -0.45
skew: 0.04, kurtosis: -0.45
250
152
200
102
50
2
20
45
60
120
20
200
V3+ DocpLaVV3+-MolleWWiVV min: 0.00, max: 100.00 mean: 62.48, std: 20.64
NonNet
min: 0.19, max: 100.00
min: 0.09, max: 100.00
mean: 66.42, std: 19.39
mean: 70.36, std: 18.15
skew: -0.05, kurtosis: -0.51
skew: 10, kurtosis: -0.55
skew: -0.25, kurtosis: -0.42
200
201
200
230
155-
250
200
107
200
0-
70
UPerNettMFTBB
SegFormer- MiT44
SegFormer-MiTB.
min: 0.03, max: 100.00
min: 0.01, max: 100.00
min: 0.00, max: 100.00
mean: 69.83, std: 18.82
mean: 69.54, std: 18.76
mean: 67.24, std: 19.42
skew: -0.23, kurtosis: -0.52
skew: -0.21, kurtosis: -0.51
skew: -0.14, kurtosis: -0.50
230
Peetel-ResNetl01 min: 0.88, max: 100.00
UNet-ResNet101
min: 0.24, max: 100.00
min: 0.81, max: 100.00
mean: 65.84, std: 19.19
mean: 65.86, std: 19.20
mean: 65.55, std: 18.74
skew: -0.07, kurtosis: -0.55
skew; -0.08, kurtosis: -0.55
skew: -0.09, kurtosis: -0.45
Figure 38: Histograms: PASCAL Context.
64

---

DeepLabY3+RReNNelO0:: 2009_004291 (loU) 0.00).
DeepLab33+ 2009_00429 (IoU! 0.00).
Retssett3 2008 000_633763 (loU) = 0.00).
DeepLabV3+ 2008 001111 (loU) = 0.00).
D 2008 0006700000000 (loU! 0.00).
Figure 39: Worst-case images: PASCAL Context 1 /3. Left: image. Middle: label. Right: prediction.
65

---

DeepLabV3--MobieeNeee: (loU) 0.00).
DeepLabV3--MobieWTTV: 2008_00570 (loU) 0.00).
UPerNet-ResNetlOOe 2008_004736 (IoU! = 0.00).
UPerNet-ConVNeXt: 2008 005046 (loU) 0.00).
UPerNet-MiTB4: 2008 005046 (loU} 0.00).
Figure 40: Worst-case images: PASCAL Context 2 3. Left: image. Middle: label. Right: prediction.
66

---

ScgFormer-MiTB4: 0000000066 (loU) 0.00).
SegFormer-MiTB2: 2009_000300 (loU) 0.00).
C 2008_004700 004730073 (IoU) = 0.00).
PSPNet-ResNet1O1 2009_0000000 _00_91 (IoU) = 0.00).
UNst-ResNe:101: 000079 (loU) 0.08).
Figure 41: Worst-case images: PASCAL Context 3 /3, Left: image. Middle: label. Right: prediction.
67

---

G.10
DeepGlobe Land
Table 23: Results: DeepGlobe Land. Red: the best for each metric. Green: the worst for each metric.
miou! miouC mlot
miou!!
mioU! mloue
mloU! miouic
Method
Hackhone
mlot/
mAse
Aec
= 0.46
0.72
DeepLabbV3
ResNert01
68.76 + 0.40
31.62 + 0.28
2.50 0.18
0.00 + 0.00
71.27 + 0.21
82.80
0.16
87.21
0.08
DeepLabb3e
ResNer50
53.63 + 0.17
31.48
0.18
2.53 0.02
0.00 0.00
71.44 + 0.07
82.94
0.15
#7.23 + 0.09
DeepLabV3e
ResNe134
53.50 + 0.36
31.42
0-44
2.30 1: 0.17
0.00
70.76 + 0.24
0.24 82.93
86.85 = 0.15
+
DeepLabV3e
2.84
70.76 1: 0.12
0.24 82.68
86.81 0.05
56.15 1 0.10
47.16 0.13
30.29 0.22
18.41 O
DeeplabV34
71.67 + 0.21
0.22
87.62 0.13
56.61 + 0.80
45.45 U.77
28.86 #: 1.28
19.43 0.09
Des
MobileNetV2
53.25
2.89
0.00
70.34
58.64 = 0.29
47.48 0.37
31.49 = 1.15
23.25 3.07
DeepLabb34
MobieViTV2
55.44 1:
33.25
2.74 0.15
0.00
0.00
72.86
56.47 + 0.10
44.93
0.25
25.45 0.90
17.33 4.38
UPaiNer
ResNet101
52.72 + 0.20
30 0.21
2.31 0.12
0,00 + 0.00
58.85 1: 0.21
47.69
0.31
30.95 0.52
19.93 0.80
UPerNet
Com-NeXi
55.83 +: 0.29
33.68
0.32
2:75
0.24
0.00 1 0.00
72.01 + 0.18
83.
0.29
88.01 :: 0.10
59.31
0.25
1.04
UPerNer
MiT84
56.42 + 0.28
34.35 3: 0.40
5.28 0.06
0.00 + 0.00
73.21 + 0.11
83.75 0.14
88.15 0.10
SegFormer
M1TB4
56.44
0.30
34.26
0.36
3.23 0.03
0.00 0.00
73.36 + 0.25
84.03 + 0.47
88.17 #: 0.14
SegFormer
56.21 1: 0.28
34.14
0.27
3.24 0.20
0.00
73.22 + 0.22
83.69 1 0.13
88.18 0.11
28.46
DespLabV3
2.53
71.39 1: 0.22
0.25 82.53
87.30 0.20
57.03 0.63
45.
0.67
23.41 0.04
21.82 2.552.45
PSPNet
R1
71.02 1: 0.60
82.39
0.75
87.07
0.30
56.63 + 0.08
45.31
0.13
28.04 1: 0.28
19.69 0.48
UNet
ResNet101
53.13 0.07
0.09
2.48 0.17
0.00
0.00
70.71 +:
0.05
86.97 0.14
68

---

12
12
20
100
SeppLaab33-RRsNeete min: 21.66, max: 99.33
DeeplabbV3+RRsNee
min: 24.58, max: 99.70
min: 19.96, max: 99.22
mean: 57.26, std: 14.55
mean: 57.13, std: 14.49
mean: 56.84, std: 14.28
skew: 0.15, kurtosis: -0.38
skew: 0.23, kurtosis: -0.46
skew: 0.16, kurtosis: -0.26
20
12
20
DeepLabV3+-ResNee min: 19.41, max: 99.30
DeepLab Bee
DecpLabV34-Mobille
min: 18.44, max: 99.72
min: 19.43, max: 99.56
mean: 56.38, std: 14.31
mean: 58. 16,16, std: 14.27
mean: 56.61, std: 14.37
skew: 0.16, kurtosis: -0.31
skew: 0.18, kurtosis: -0.21
skew: 0.18, kurtosis: -0.36
20
42
300
DeepLabV3++ Mobileeiii min: 25.82, max: 99.94
UPerNet-Resse ResNet101
OPerNet-ConvNeXX
min: 19.71, max: 99.35
min: 19,93, max: 99.87
mean: 58.64, std: 14.31
mean: 56.47, std: 14.84
mean: 58.85, std: 14.36
skew: 0.14, kurtosis: -0.54
skew: 0.11, kurtosis: -0.26
skew: 0.10, kurtosis: -0.39
70
UPTTNet-MFTB4 min: 21.49, 99.84
Seg SegFormer-Me MiTB4 min: 23.32, max: 99.88
SegFomer-MMTB2
min: 20.27, max: 99.96
mean: 59.31, std: 14.49
mean: 59.38, std: 14.44
mean: 59.40, std: 14.53
skew: 0.09, kurtosis: -0.47
skew: 0.14, kurtosis: -0.51
skew: 0.05, kurtosis: -0.48
20.
12-
10-
C min: 20.32, max: 99,41
OSPNet-ResNetION min: 22.43, max: 99.13
UNet-ResNet101 min: 19.68, max: 99.31
mean: 57.31, std: 14.51
57.03, std: 14.41
56.65. std: 14.72
skew: 0.14, kurtosis: -0.31
skew: 0.22, kurtosis: -0.37
skew: 0.17, kurtosis: -0.38
Figure 42: Histograms: DeepGlobe Land.
69

---

ReeLLabV3+ReeNeelO01 161109_sa (IoU) 20.01).
DecpLabV3+ 161109 sat (IoU) 19.01).
DecpLabV3+RResNee 143794 sat (loU) = 15.76).
ReepaabW3+RReNNetll 161109 sat (IoU) = 19.00).
Effiiintt: 161109_sst sat (IoU) 18.34).
Figure 43: Worst-case images: DeepGlobe Land 1 3. Left: image. Middle: label. Right: prediction.
70

---

DeepLaVV3--MobiNNeY\:: 161109_sa (IoU 18.47).
DeepLabV>+-Nobileiiitz 161109_1 sat (IoU) 18.96).
UPerNct-ResNetlO0) 143794 sat (loU) 11.45).
UPoNNet-ComvNeet: 161 _at (loU) = 19.05).
UPerNet-MiTB4: 161109_ss sat (loU} 19.22).
Figure 44: Worst-case images: DeepGlobe Land 2 2/3. Left: image. Middle: label. Right: prediction.
71

---

SegFormer-MiTB4: 161109_sa sat (IoU) 18.89).
CegFormer-MiTB2 161109_00 sat (loU) 19.43).
De<pLabV3-ReNeelt 161109_ese sat (IoU) = 18.88).
OesNel-Re 161 109 sat (IoU) = 19.00).
1Net-ResNet101: 161109_ss sat (loU) 19.15).
Figure 45: Worst-case images: DeepGlobe Land 3 3.3. Left: image. Middle: label. Right: prediction.
72

---

G.11
DeepGlobe Road
Table 24: Results: DeepGlobe Road. Red: the best for each metric. Green: the worst for each metric.
miou! miouC mlot!"
miou!!
mioU! mloue
mloU! miou
Method
Hackhone
mlot/
mAse
Asc
=
2.10
+ 1.45
DeepLabbV3
ResNer101
64.59 + 0.88
58.15 = 1.19
26.09 = 2.10
8.00 + 1.63
65.60 + 0.45
88.68
0.52
98.22 0.01
DeepLabb3e
ResNer50
63.86 0.07
52. 56 0.08
25.95 0.38
6.00 4: 1:41
64.59 + 0.08
88.34
0.14
98.16 0.01
0.99
DeepLabV3e
ResNer34
63.49 = 0.17
51.77
0.22
23.13 0.50
4,00 1 0.82
64.51 + 0.06
88.21 2 0.11
98.16 = 0.01
02.74
22.62
DeepLabV3e
0.93
63.87 0.08
87.85 0.11
98.12 0.00
64.29 1 0.09
53.49
0.10
28.82 0.11
13.88 0.58
DeeplabV34
65.14 + 0.07
0.07
98. 0.00
62.25 + 0.24
50.47 0.33
22.55
0.73
6.95 + 0.78
Des
MobileNetV2
62.25
22. 22.58
0.73
63.47
0.00
64.65 + 0.13
0.24 53.58
27.50 = 0.83
12.66 = 1.71
DeepLabb34
Mob
64.65 1:
53.58
21.50 0.83
12:33
1.70
63.03 + 0.32
51.03 1.05
22.61 3.41
6.12 3.133
UPaiNer
ResNet101
63.06 + 0.52
51.03 1.05
22.61 3.41
4.67 + 3.30
06.71 0.04
55.82 0.12
29.37 0.82
10:13 2.14
UPerNet
Com-NeXi
66.71 + 0.04
55.62 1: 0.12
29.37 : 0.82
9.67 + 1.89
67.65 + 0.05
89. 88 0.10
98.33 0.00
65.05 +
1.21
26.65 2.79
UPerNer
MiT84
65.05 + 0.87
58.73 1.21
26.65 2.79
10.00 5.66
66.37 + 0.64
88 90 0.55
98.27 0.03
SegFormer
M1TB4
64.27 + 1.15
90
1.59
25.60
8.33 1: 4.50
65.61 + 0.87
88.61 + 0.72
98.23 + 0.04
64.74
28.66
SegFormer
64.74 +: 0.02
53.81
0.22
13.00 1.41
65.70 + 0.09
89.44 0.12
98. 19 = 0.00
01.82
+
DespLabV3
4.08
63.57 1: 1.53
88.16
0.81
0.09
81.13 2.600000 2.60
49.00
3:45
21.76 5.14
7:33 1/33-33 3.00
PSPNet
63.19 1: 1.73
87.49 1: 1.25
98.09
0.08
62.88 + 0.88
50.85
4: U.994
21.23 + 2.00
3.29
+ U.87
UNet
ResNet101
0.88
21
3.00 0.82
64.26
+: 0.56
86.77 0.71
98.21
0.00
73

---

60
60
50
02
60
40
40
20-
20
100
DeepLab 3343-333eeett
min: 0.00, max: 90.85
min: 0.00, max: 92.84
min: 0.00, max: 90.87
mean: 64.59, std: 14.06
mean: 63.87, std: 14.01
mean: 63.49, std: 14.41
skew: -1004, kurtosis: 1.94
skew: -1003, kurtosis: 2.10
skew: -1.15, kurtosis: 2.16
DeepLabV34-Resse Re- min: 0.00, max: 90.55
DeepLab E3e-EffcciinnnNe1e0
D
min: 0.00, max: 92.55
min: 0.00, max: 90.57
mean: 62.74, std: 14.54
mean: 64.29, std: 13.19
mean: 62.25, std: 14.48
skew: -1006, kurtosis: 1.82
skew: -0.92, kurtosis: 1.47
skew: -1.00, kurtosis: 1.66
20
2
20
42
60
120
20
200
DeepLabV3+ DoopLabV3+MMobleWiiVV min: 3.65, max: 92.31
UPerNet- Cottettt
min: 0.00, max: 91.65
min: 0.00, max: 93.26
mean: 64.65, std: 13.61
mean: 63.06, std: 14.70
mean: 66.71, std: 13.49
skew: -1.00, kurtosis: 1.63
skew: -0.98, kurtosis: 1.63
skew: -1,18, kurtosis: 2.49
60
50
65-
60
40
47-
40
20
70
UPerNet-MiTBB min: 0.00, max: 93.21
ScgFormer-MiTH MiTB4
SegFomer-MiTBB
min: 0.00, max: 92.73
min: 0.00, max: 92.51
mean: 65.05, std: 13.72
mean: 64.27, std: 13.81
mean: 64.74, std: 13.45
skew: -1.07, kurtosis: 1.82
skew: -1.66, kurtosis: 1.77
skew: -0.97, kurtosis: 1.60
C min: 0.00, max: 90.61
PSPNet-ResNetll min: 0.00, max: 89.29
UNet-ResNet101
min: 0.00, max: 91.50
61.82, std: 14.47
61.13, std: 14.42
62.88, std: 14.85
skew: -0.95, kurtosis: 1.49
skew: -0.86, kurtosis: 1.17
skew: -1.10, kurtosis: 2.04
Figure 46: Histograms: DeepGlobe Road.
74

---

ResNect0 167220_sss _aat (IoU) 0.00).
DeepLabV3+-ResNet5u 167220_sg _at (IoU) 0.00).
DeepLab BLLLLab3++ReeNet33 ResNet34: 106741 sat (IoU) = 0.00).
DeepLab V3+ 122985 sat (IoU) = 0.00).
DogoLaab3+Efffiientttt 167220_ sat (loU) 0.00).
Figure 47: Worst-case images: DeepGlobe Road /3 Left: image. Middle: label. Right: prediction.
75

---

WobiuabbttMMMbbbaNetttVV 122985_sae sat (IoU) 0.00).
DeepLabV3-MobbieWiiiz2 167220 (loU! 0.00).
UPenNet-ResNNttO0! 157077_ss _at (loU) 0.00).
UPorNet-Com/NeXt: 247410 sat (loU) = 0.00).
UPerNet-MiTB4: 167220 sat (loU) 0.00).
Figure 48: Worst-case images: DeepGlobe Road 2/ 2/3. Left: image. Middle: label. Right: prediction.
76

---

SegFormer-MiTB4: 167220_ss sat (IoU) 0.00).
SegFormer-MiTBB: 167220_ _at (loU! 0.00).
DeepLabV3 --r.aubV--ReeNNeOOO 167220 sat (loU) 0.00).
PePNet-ResNet101: 167220 sat (IoU) = 0.00).
UNet-ResNet101: 122985_sssse sat (loU) 0.00).
Figure 49: Worst-case images: DeepGlobe Road 3/3. Left: image. Middle: label. Right: prediction.
77

---

G.12
DeepGlobe Building
Table 25: Results: DeepGlobe Building, Red: the best for each metric. Green: the worst for each metric.
mlot?
miati mlou
mlot mlou
mlat" miou
Method
Backbone
mlot
mlot
mAce
Ace
77.54
0.18
68.40 = 0.33
0.12 0.11
0.00 + 0.00
De
ResNet101
0.12
0.00
80.04
83.66
0.11
96.94
0.01
17:24 2 0.19
59.01
2 0.35
0.15 0.18
0.00 1 0.00
V
ResNec30
0.10
13.73
= 0.33
55.86 = 0.55
0.00 2: 0.00
0.00
0.00
DeepllahV34 3bV3+
34
75.73
0.30
55.86 = 0.58
0.00 0.00
0.00 + 0.00
75.87 =2 0.45
65.56
= 0.90
0.00 = 0.00
0.00 + 0.
Deeplaab
ResNet18
75.37 4: 0.45
55.56
0.90
0.00 + 0.00
0.00 + 0.00
74.12
1.04
52.91
1.94
0.00 0.00
Despl 34/3+
EMiiienNerRO
74.12 = 1.04
52.91
1.94
0.00 = 0.00
0.00 + 0.00
80.01
0.22
93.61
0.05
96.75 0.04
2.08
0.00
Deepi abV DesplabbV3
MobileNerV2
74.66 = 1.09
54.34 = 2.08
0.00 = 0.00
0.00 + 0.00
79.20 0.D4
93.26
0.10
96.61 + 0.01
Vob1
74.19
0.62
52.89
1.21
0.00- 0.00
0.00 0.00
60.58 1: 0.10
93.73 0.03
96.88 1: 0.02
UPeeNer
ResNori01
80.85
U.14
93.46
0.07
96.34 + 0.02
79.21 1: 0.22
62.63
0.54
2.83 0.9.
0.00 I 0.00
UPerNet
ConyNeXi
0.00
$1.08
0.07
94.04
0.00
97.00 + 0.02
76.73
1.12
57.28
2.40
0.00 0.00
0.00 + 0.00
UPerNet
M6TB4
0.16
0.15
0.02
17.02 2 0.99
58.21 2 2.06
0.27 :: 0.39
0.00 1 0.00
Segtormer
ME134
0.00
15.77 = 0.03
55.67
= 0.04
0.00 1 0.00
0.00 1 0.00
SeyFormer
MrT92
75.77
0.03
55.67
0.04
0.00 0.00
0.00 + 0.00
77.21 =2 0.09
59.62
0.56
1.51 1.64
0.00 + 0.D0
DeepLabV3
ResNet101
77:21 d: 0.09
59.62
0.56
1.51 :: 1.64
0.00 + 0.00
77.13 0.30
50.00
0.55
0.37 0.28
PSPNet
ResNer1D1
77.13 0.30
58.09
0.55
0.37 = 0.28
0.00 + 0.00
80.32 0.31
93.48 0.33
96.83 0.04
0.00
UNer
ResNeriDI
76.03 = 0.31
58.11 0.67
0.00 + 0.00
0.00 + 0.00
0.12 80.87
93.77
0.13
96.92 : 0.01
78

---

700
002
300
400
402
100
200
302
200
200
295
200
200
00
20
100
Setppaabb3+RResNeeee
DeepLab DeeplLabV3+--BB V34--33es
min: 0.00, max: 100.00
min: 0.00, max: 100.00
min: 0.00, max: 100.00
mean: 77.55, std: 21.81
mean: 77.24, std: 21.99
mean: 75.73, std: 23.30
skew: -1992, kurtosis: 4.17
skew: -1.00, kurtosis: 4.06
skew: -1.79, kurtosis: 3.22
300
DeepLabV34-ResNell
DeepL.bbV3-EEficieeee
D
min: 0.00, max: 100.00
min: 0.00, max: 100.00
min: 0.00, max: 100.00
mean: 75.37, std: 23.35
mean: 74.12, std: 23.95
mean: 74.66, std: 23.25
skew: -175,, kurtosis: 3.09
skew: -1.71, kurtosis: 2.71
skew: -1.71, kurtosis: 2.99
500
400
002
300
302
200
202
200
102
2-
20
45
120
20
200
V3+ min: 0.00, max: 100.00
MoolLabDVVMMoileiiViVV MobileViTV2
UPerNet-ResNetlUe ResNet101
UPerNet- Cottettt
min: 0.00, max: 100.00 mean: 77.45, std: 21.95
min: 0.00, max: 100.00
mean: 79,19, std: 24.10
mean: 79.21, std: 20.45
skew: -1.74, kurtosis: 2.81
skew: -1,91, kurtosis: 4.07
skew: -2.13, kurtosis: 5.50
300
502
500
400
402
400
200
302
300
200
205
200
100
105
100
20
UPerNet-MFTB4 min: 0.00, max: 100.00
SegFormer-MiTe MiTB4
SegForeer-MiTB3
min: 0.00, max: 66.67
min: 0.00, max: 100.00
mean: 76.73, std: 22,51
mean: 51.68, std: 14.45
mean: 75.77, std: 24.10
skew: -1.94, kurtosis: 3.88
skew: -1.55, kurtosis: 4.41
skew: -1899, kurtosis: 3.37
DbepLabV3-ResNeelle min: 0.00, max: 100.00
PSPNet-ResNetI01
Ueet-RReNeel00
min: 0.00, max: 100.00
min: 0.00, max: 100.00
77.21, std: 21.09
77.13, std: 21.42
mean: 76.94, std: 22.53
skew: -1.88, kurtosis: 4.34
skew: -1.87, kurtosis: 4.08
skew: -1,92, kurtosis: 3.93
Figure 50: Histograms: DeepGlobe Building.
79

---

ResNeet10::: Khartoum Leee__009____0___ sat (IoU) 0.00).
DeepLabV3t-ResNene Khartoum, 1rctttoum_069900000 _at (loU) = 0.00).
DeopLabV3++Resseeee Khartoum 1069_006000 sat (IoU) = 0.00).
DeepLabV3+ Resseetl Khartoum 1069 sat (IoU) 0.00).
DeepLabV3+ Effieeeneticcci Khartoum 1069 sat (IoU) 0.00).
Figure 51: Worst-case images: DeepGlobe Building 1 / 3. Left: image. Middle: label. Right:
prediction.
80

---

DeepLabV3+ Khartoum 1069 sat (IoU 0.00).
DucpLabYS+-MobieeiiVy Khartoum 1107_sat (loU) 0.00).
UPerNet-ResNet1O1:: Khartoum 1069 sat (loU) = 0.00).
ConNNet- Khartoum 1069 sat (loU) 0.00).
UPerNet-MiTB4: Khartoum 1107_sa sat (loU! 0.00).
Figure 52: Worst-case images: DeepGlobe Building 2 / 3. Left: image. Middle: label. Right: prediction.
81

---

SegFormer-MiTB4: Khartoum 1108_si0 sat (loU) 0.00).
SegFormer-MiTB2: Khartoum 1108_sat (loU) = 0.00).
DeepLabV3-ResNeee Khartoum 1069__000000001000 sat (IoU) = 0.00).
ResNel-ResNet1O1: Khartoum 1069 sat (IoU 0.00).
UNst-ResNe Khartoum 1069 sat (IoU) 0.00).
Figure 53: Worst-case images: DeepGlobe Building 3 / Left: image. Middle: label. Right:
prediction.
82


Reviews:
[{"review_id": "N7NzgsTq5t", "reviewer": "Reviewer_ahvw", "summary": "", "questions": "", "limitations": "There is no discussion about limitations, but I also do not find any specific limitation about this work. ", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper analyze limitations of existing evaluation protocols thoroughly, and proposes new evaluation metrics that can effectively address the limitations.\n\n- I believe that the proposed fine-grained evaluation metrics are technically sound. This will open a new research direction of semantic segmentation. \n\n- Re-evaluation of existing semantic segmentation methods on various datasets is fruitful for future semantic segmentation research.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "WCw88yUvbx", "reviewer": "Reviewer_NFd4", "summary": "", "questions": "", "limitations": "Potential limitations are sparsely discussed (e.g., mIoUI bias towards more frequent classes). However, the proposed metrics have only few relevant limitations that I am aware of but solve the limitations of the current practice. The discussion could include the effect on the computation costs (evaluation time) and the design choice of not considering the non-gt classes.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The downside of the current practice and the resulting motivation for fine-grained metrics is well explained. Furthermore, the proposed metrics are clearly described and easy to understand. They cover a wide range of potential use cases, like worst case metrics for safety-critical applications. In the future, researchers may not evaluate only the default mIoU but more specific metrics that better represent their goal for a use case or dataset. Lastly, the experiments demonstrate the benefits of the proposed metrics and enable a more detailed comparison of different models and their architectural approaches. The authors use these insights to provide best practices for training segmentation models.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "BfYPD4l4b4", "reviewer": "Reviewer_gpCb", "summary": "", "questions": "", "limitations": "No, please provide more discussion on the limitations. I assume that the negative social impact can be minor as the new metric targets at improving safety-critical applications. But again, to ensure the safety, limitations of the metric should be discussed.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ The motivation of the new metric is clear and easy to follow.\n+ This paper provides a comprehensive study of the proposed metric, including its advantages, the benchmark study, and the architecture design as well as loss functions.\n+ The experiments are extensive, covering 12 varied natural and aerial datasets and featuring 15 modern neural network architectures, which can verify the claims of the proposed metric.\n+ The findings in Sec. 5.1 are interesting, and can possibly guide the design of novel algorithms.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "pNbaNQuvLO", "reviewer": "Reviewer_QtsN", "summary": "", "questions": "", "limitations": "1. The GitHuB page seems to refer to a completely different project (Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labels).\n2. Paper has minor contributions to research community due to trivial extensions to the conventional mIoU metric.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "Authors accomplished a comprehensive study for multiple segmentation models on many datasets using the proposed metrics.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This work proposes the use of fine-grained mIoUs along with corresponding worst-case metrics, offering a more holistic evaluation of segmentation techniques to address class and size imbalances in the data.

Most reviewers found this benchmark paper interesting and important. They commented that:

+ The paper is well written and easy to understand.
+ The fine-grained evaluation metrics are technically sound, although they are a simple modifications of mIoU.
+ The paper Re-evaluation of existing semantic segmentation methods on various datasets is interesting.

Some improvements were proposed:
+ Potential limitations are not discussed enough.
+ The work would benefit from focusing on to the most relevant metrics. 

The low score reviewer did not think the paper is suitable for the Dataset and Benchmark track, but authors seems to convince him. I believe that even if the new metrics are only simple adaptations of the old mIoU, the fact that the paper addresses the class/size imbalances is very important. In fact, using simple mIoU metrics one can play with the results by including/removing classes etc.
This paper highlights this problem and tries to address it.

I therefore suggest accepting this paper if the authors revise according to the reviewers' suggestions.
Decision: Accept (Poster)