SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zifu Wang $^{1*}$  Maxim Berman $^{2}$  Amal Rannen-Triki $^{3}$  Philip H.S. Torr $^{4}$  Devis Tuia $^{5}$
Tinne Tuytelaars
Luc Van Gool<sup>1,6,7</sup>
Jiaqian  $\mathbf{Y}\mathbf{u}^{8}$
2 Google, Zürich, Switzerland
<sup>3</sup> Google DeepMind, London, United Kingdom
<sup>4</sup> University of Oxford, Oxford, United Kingdom
5 EPFL, Lausanne, Switzerland
$^{6}$  ETH Zürich, Zürich, Switzerland
<sup>7</sup> INSAIT, Sofia, Bulgaria
Samsung Research, Beijing, China
Semantic segmentation datasets often exhibit two types of imbalance: class imbalance, where some classes appear more frequently than others and size imbalance, where some objects occupy more pixels than others. This causes traditional evaluation metrics to be biased towards majority classes (e.g. overall pixel-wise accuracy) and large objects (e.g. mean pixel-wise accuracy and per-dataset mean intersection over union). To address these shortcomings, we propose the use of fine-grained mIoUs along with corresponding worst-case metrics, thereby offering a more holistic evaluation of segmentation techniques. These fine-grained metrics offer less bias towards large objects, richer statistical information, and valuable insights into model and dataset auditing. Furthermore, we undertake an extensive benchmark study, where we train and evaluate 15 modern neural networks with the proposed metrics on 12 diverse natural and aerial segmentation datasets. Our benchmark study highlights the necessity of not basing evaluations on a single metric and confirms that fine-grained mIoUs reduce the bias towards large objects. Moreover, we identify the crucial role played by architecture designs and loss functions, which lead to best practices in optimizing fine-grained metrics. The code is available at https://github.com/zifuwanggg/JDTLosses.
Every metric reflects a certain property of the result and choosing the right one is important to emphasize those that we care about. However, this choice may not be easy, as many metrics come with certain biases [34]. For semantic segmentation, the overall pixel-wise accuracy (Acc) is believed unsuitable due to its bias towards majority classes, especially given that semantic segmentation datasets typically have long-tailed class distributions [18]. The mean pixel-wise accuracy (mAcc) is introduced to counter this issue by averaging pixel-wise accuracy across all classes, and it eventually became the official evaluation metric in PASCAL VOC 2007 [16], which was one of the earliest challenges that showcased a segmentation leaderboard. However, mAcc does not account for false positives, leading to over-segmentation of the results. Consequently, in PASCAL VOC 2008 [17],
(b)
Figure 1: An illustration of size imbalance. The class Car is depicted in blue and the regions are highlighted within white dashed circles. (a) A car whose side mirror takes up around 2,000 pixels (from Cityscapes). (b) A car that takes up around 1,000 pixels (from Cityscapes). (c) A car whose front wheel takes up around 600,000 pixels (from Mapillary Vistas). (d) A car that takes up around 1,000 pixels (from Mapillary Vistas).
the official metric was updated to the per-dataset mean intersection over union  $(\mathrm{mIoU}^{\mathrm{D}})$ . Since then, PASCAL VOC [18] has consistently used  $\mathrm{mIoU}^{\mathrm{D}}$ , and this metric has been adopted by a multitude of segmentation datasets [38, 9, 39, 59, 5, 11, 12, 15, 43].
In  $\mathrm{mIoU}^{\mathrm{D}}$ , true positives, false positives and false negatives are accumulated across all pixels over the whole dataset and therefore this metric suffers several notable shortcomings [10, 9]:
- It is biased towards large objects, which can pose a significant issue given that most semantic segmentation datasets inherently have a considerable size imbalance (see Figure 1). This bias is particularly concerning in safety-aware applications with critical small objects such as autonomous driving [4, 9, 39, 11, 43] and medical imaging [37, 22, 49, 3].
- It fails to capture valuable statistical information about the performance of methods on individual images or instances, which impedes a comprehensive comparison.
- It is challenging to link the results from user studies to those obtained with a per-dataset metric, since humans generally assess the quality of segmentation outputs at the image level.
In this paper, we advocate fine-grained mIoUs for semantic segmentation. In line with previous studies [10, 55, 28, 31], we calculate image-level metrics. When instance-level labels are available [9, 39, 59, 5], we propose to compute approximated instance-level scores. These fine-grained metrics diminish the bias toward large objects. Besides, they yield a wealth of statistical information, which not only allows for a more robust and comprehensive comparison, but also leads to the design of corresponding worst-case metrics, further proving their importance in safety-critical applications. Furthermore, we show that these fine-grained metrics facilitate detailed model and dataset auditing.
Despite the clear advantages of fine-grained mIoUs over  $\mathrm{mIoU}^{\mathrm{D}}$ , these metrics are seldom considered in recent studies within the segmentation community. We bridge this gap with a large-scale benchmark, where we train and evaluate 15 modern neural networks on 12 natural and aerial segmentation datasets, spanning a variety of scenarios including (i) street scenes [4, 9, 39, 11, 43], (ii) "thing" and "stuff" [18, 38, 59, 5], (iii) aerial scenes [12]. Our benchmark study emphasizes the peril of depending solely on a single metric and verifies that fine-grained mIoUs mitigate the bias towards large objects. Moreover, we conduct an in-depth analysis of neural network architectures and loss functions, leading to best practices for optimizing these metrics.
In sum, we believe no evaluation metric is perfect. Thus, we advocate a comprehensive evaluation of segmentation methods and encourage practitioners to report fine-grained metrics and corresponding worst-case metrics, as a complement to mIoU $^{\text{D}}$ . We believe this is particularly crucial in safety-critical applications, where the bias towards large objects can precipitate catastrophic outcomes.
2 Fine-grained mIoUs
In this section, we review the traditional per-dataset mean intersection over union, followed by our proposed fine-grained image-level and instance-level metrics.
$\mathbf{mIoU}^{\mathbf{D}}$ . Intersection over union (IoU) is defined as
$$
\mathrm {I o U} = \frac {\mathrm {T P}}{\mathrm {T P} + \mathrm {F P} + \mathrm {F N}}, \tag {1}
$$
where TP, FP and FN represent true positives, false positives and false negatives, respectively. Per-dataset IoU computes these numbers by accumulating all pixels over the whole dataset. In particular, for a class  $c$ , it is defined as
$$
\mathrm {I o U} _ {c} ^ {\mathrm {D}} = \frac {\sum_ {i = 1} ^ {I} \mathrm {T P} _ {i , c}}{\sum_ {i = 1} ^ {I} \left(\mathrm {T P} _ {i , c} + \mathrm {F P} _ {i , c} + \mathrm {F N} _ {i , c}\right)}, \tag {2}
$$
where  $I$  is the number of images in the dataset;  $\mathrm{TP}_{i,c}$ ,  $\mathrm{FP}_{i,c}$  and  $\mathrm{FN}_{i,c}$  are the number of true-positive, false-positive and false-negative pixels for class  $c$  in image  $i$ , respectively. The per-dataset mean IoU is then calculated as
$$
\mathrm {m I o U} ^ {\mathrm {D}} = \frac {1}{C} \sum_ {c = 1} ^ {C} \mathrm {I o U} _ {c} ^ {\mathrm {D}}, \tag {3}
$$
where  $C$  denotes the number of classes for the dataset.
Compared to pixel-wise accuracy (Acc) and mean pixel-wise accuracy (mAcc), mIoU<sup>D</sup> aims to mitigate class imbalance and to take FP into account. It is widely used in the evaluation of semantic segmentation models, but it has several shortcomings. Most notably, it is biased towards large objects in the dataset, due to dataset-level accumulations of TP, FP and FN. To address this issue, we can resort to fine-grained mIoUs at per-image  $(\mathrm{mIoU}^{\mathrm{I}}$ , mIoU<sup>C</sup>) and per-instance  $(\mathrm{mIoU}^{\mathrm{K}})$  levels, as detailed below.
$\mathbf{mIoU}^{\mathbf{l}}$ . Following [10, 55, 28, 31], we compute per-image scores. In particular, for each image  $i$  and class  $c$ , a per-image-per-class score is calculated as
$$
\mathrm {I o U} _ {i, c} = \frac {\mathrm {T P} _ {i , c}}{\mathrm {T P} _ {i , c} + \mathrm {F P} _ {i , c} + \mathrm {F N} _ {i , c}}. \tag {4}
$$
Then the per-image IoU for image  $i$  is defined as
$$
\mathrm {I o U} _ {i} ^ {\mathrm {I}} = \frac {\sum_ {c = 1} ^ {C} \mathbb {1} \left\{\mathrm {I o U} _ {i , c} \neq \mathrm {N U L L} \right\} \mathrm {I o U} _ {i , c}}{\sum_ {c = 1} ^ {C} \mathbb {1} \left\{\mathrm {I o U} _ {i , c} \neq \mathrm {N U L L} \right\}} \tag {5}
$$
where we only sum over  $\mathrm{IoU}_{i,c}$  that is not NULL (discuss below). Averaging these per-image scores yields
$$
\mathrm {m I o U} ^ {\mathrm {I}} = \frac {1}{I} \sum_ {i = 1} ^ {I} \mathrm {I o U} _ {i} ^ {\mathrm {I}}. \tag {6}
$$
Using image-level metrics presents a challenge: not all classes appear in every image, leading to NULL values and potential ambiguities. In multi-class segmentation, we denote the per-image-per-class value as NULL if the class is missing from the ground truth, regardless of it appearing in the prediction or not. In contrast, for binary segmentation, the value is set to 0 instead of NULL when the prediction includes the foreground class, but the ground truth does not. More detail of the definition is in Appendix D.
$\mathbf{mIoU}^{\mathrm{C}}$ . As the left panel of Figure 2 shows, due to the presence of NULL values, averaging these per-image-per-class scores by class first and then by image  $(\mathrm{mIoU}^{\mathrm{I}})$  is different from averaging them by image first and then by class. A drawback of  $\mathrm{mIoU}^{\mathrm{I}}$  is that it is biased towards classes that appear in more images<sup>2</sup>. To address this bias, we first average these per-image-per-class scores by image and define the per-image IoU for class  $c$  as
$$
\mathrm {I o U} _ {c} ^ {\mathrm {C}} = \frac {\sum_ {i = 1} ^ {I} \mathbb {1} \left\{\mathrm {I o U} _ {i , c} \neq \mathrm {N U L L} \right\} \mathrm {I o U} _ {i , c}}{\sum_ {i = 1} ^ {I} \mathbb {1} \left\{\mathrm {I o U} _ {i , c} \neq \mathrm {N U L L} \right\}}. \tag {7}
$$
Similar to  $\mathrm{mIoU^{D}}$ , we then average these per-class values to obtain
$$
\mathrm {m I o U} ^ {\mathrm {C}} = \frac {1}{C} \sum_ {c = 1} ^ {C} \mathrm {I o U} _ {c} ^ {\mathrm {C}}. \tag {8}
$$
2This is different from Acc that is biased towards classes that take up more pixels.
I1	I2	I3	I4	I5
C1	0.85	0.94	0.56	0.73	0.68
C2	0.42	NULL	0.59	0.88	NULL
C3	NULL	0.79	NULL	NULL	NULL
Figure 2: Left: mIoU<sup>I</sup> first averages per-image-per-class scores by class and then by image. mIoU<sup>C</sup> first averages them by image and then by class. mIoU<sup>I</sup> is biased towards classes that appear in more images, e.g.  $C_1$ . Right: Given two ground-truth instances 1 (red) and 2 (blue), and the prediction (green), we can compute TP<sub>1</sub>, TP<sub>2</sub>, FN<sub>1</sub>, FN<sub>2</sub>. We propose to approximate FP<sub>1</sub> and FP<sub>2</sub> by distributing FP proportional to the size of each instance.
$\mathbf{mIoU}^{\mathbf{K}}$ . IoU is known for its scale-variance. However, this property only makes sense at an instance level. When instance-level annotations are available, for example, for "thing" classes in panoptic segmentation [27] ("thing" classes are with characteristic shapes and identifiable instances such as a truck; see a more detailed discussion of "thing" and "stuff" in Appendix B.1), we can compute a even more fine-grained metric.
Specifically, for an image  $i$  and a class  $c$ , we have a binary mask  $H \times W$  from the prediction of a semantic segmentation network and an instance-level annotation  $H \times W \times K_{i,c}$  from panoptic segmentation, where  $K_{i,c}$  is the number instances of class  $c$  that appear in image  $i$ . The key insight, as illustrated in the right panel of Figure 2, is to realize that TP and FN for each instance can be computed exactly, and we are only left with image-level FP. We propose to approximate instance-level FP by distributing image-level FP proportional to the size of each instance, with other variants discussed in Appendix E. Therefore, for class  $c$  in image  $i$ , we compute the per-instance IoU as
$$
\mathrm {I o U} _ {i, c, k} = \frac {\mathrm {T P} _ {i , c , k}}{\mathrm {T P} _ {i , c , k} + \mathrm {F N} _ {i , c , k} + \frac {S _ {i , c , k}}{\sum_ {k = 1} ^ {K _ {i , c}} S _ {i , c , k}} \mathrm {F P} _ {i , c}}, \tag {9}
$$
such that  $\mathrm{FP}_{i,c}$  is the total number of false-positive pixels of class  $c$  in image  $i$ , and  $S_{i,c,k} = \mathrm{TP}_{i,c,k} + \mathrm{FN}_{i,c,k}$  is the size of instance  $k$ . Having these per-instance IoUs, we can again calculate a per-class score as
$$
\mathrm {I o U} _ {c} ^ {\mathrm {K}} = \frac {\sum_ {i = 1} ^ {I} \sum_ {k = 1} ^ {K _ {i , c}} \mathrm {I o U} _ {i , c , k}}{\sum_ {i = 1} ^ {I} K _ {i , c}}, \tag {10}
$$
With a slightly abuse of notation, we have
$$
\mathrm {m I o U} ^ {\mathrm {K}} = \frac {1}{C} \sum_ {c = 1} ^ {C} \mathrm {I o U} _ {c} ^ {\mathrm {K}}, \tag {11}
$$
such that for classes with instance-level labels, we calculate  $\mathrm{IoU}_c^{\mathrm{K}}$  according to Eq. (10), and for classes without instance-level labels, such as "stuff" classes (classes that are amorphous and do not have identifiable instances such as sky, see Appendix B.1), we compute it as in Eq. (7).
To conclude, similar to  $\mathrm{mIoU}^{\mathrm{D}}$ , we average the scores on a per-class basis for  $\mathrm{mIoU}^{\mathrm{C}}$  and  $\mathrm{mIoU}^{\mathrm{K}}$ , therefore mitigating class imbalance. Additionally,  $\mathrm{mIoU}^{\mathrm{I}}$  and  $\mathrm{mIoU}^{\mathrm{C}}$  reduce size imbalance from dataset-level to image-level, and  $\mathrm{mIoU}^{\mathrm{K}}$  further computes the score at an instance level. On the other hand, scores of  $\mathrm{mIoU}^{\mathrm{I}}$  are finally averaged on a per-image basis; therefore it will be biased towards classes that appear in more images. However,  $\mathrm{mIoU}^{\mathrm{I}}$  leads to a single averaged score per image (while  $\mathrm{mIoU}^{\mathrm{C}}$  has  $I$  scores per class), making per-image analysis (e.g. image-level histograms and worst-case images) feasible as we will present in section 5.1.
3 Worst-case Metrics
It is essential for safety-critical applications to evaluate a model's worst-case performance. Adopting fine-grained mIoUs allows us to compute worst-case metrics, where only images/instances that a model obtains the lowest scores are considered. In this section, we focus on  $\mathrm{mIoU}^{\mathrm{C}}$  as an example. However, the same idea can be applied to both  $\mathrm{mIoU}^1$  and  $\mathrm{mIoU}^{\mathrm{K}}$ .
For each class  $c$ , we first sort  $\mathrm{IoU}_{i,c}$  by images such that  $\mathrm{IoU}_{1,c} \leq \ldots \leq \mathrm{IoU}_{I_{c},c}$ , where  $I_{c} = \sum_{i=1}^{I} \mathbb{1}\{\mathrm{IoU}_{i,c} \neq \mathrm{NULL}\}$ . We then only compute the average of those scores that fall below the  $q$ -th quantile:
$$
\mathrm {I o U} _ {c} ^ {\mathrm {C} q} = \frac {1}{\max \left(1 , \left\lfloor I _ {c} \times q \% \right\rfloor\right)} \sum_ {i = 1} ^ {\max \left(1, \left\lfloor I _ {c} \times q \% \right\rfloor\right)} \mathrm {I o U} _ {i, c}. \tag{12}
$$
For the ease of comparison, we want to derive a single metric that considers different  $q\%$ . Thus, we partition the range into 10 quantile thresholds  $\{10,20,\dots ,90,100\}$ . We then average the scores corresponding to these thresholds:
$$
\mathrm {I o U} _ {c} ^ {\mathrm {C} ^ {\bar {q}}} = \frac {1}{1 0} \sum_ {q \in \{1 0, \dots , 1 0 0 \}} \mathrm {I o U} _ {c} ^ {\mathrm {C} ^ {q}}. \tag {13}
$$
Consequently, the scores with lower values will be given more weights to the final metric. Additionally, we consider  $\mathrm{IoU}_c^{\mathrm{C}^1}$  and  $\mathrm{IoU}_c^{\mathrm{C}^5}$  to evaluate a model's performance on extremely hard cases. Having these per-class scores, we can average them to obtain the mean metrics:  $\mathrm{mIoU}^{\mathrm{C}^{\bar{q}}}$ ,  $\mathrm{mIoU}^{\mathrm{C}^5}$  and  $\mathrm{mIoU}^{\mathrm{C}^1}$ .
4 Advantages of Fine-grained mIoUs
In summary, we advocate these fine-grained mIoUs as a complement of  $\mathrm{mIoU}^{\mathrm{D}}$  because they present several notable benefits:
- Reduced bias towards large objects. Unlike  $\mathrm{mIoU}^{\mathrm{D}}$ , fine-grained metrics compute TP, FP and FN at an image or an instance level, thus decreasing the bias towards large objects. To further illustrate this on real datasets, we can compute the ratio of the size of the largest object to that of the smallest object for each class, evaluated at both dataset  $(r_c^{\mathrm{D}})$  and image levels  $(r_c^{\mathrm{I}})$ :
$$
r _ {c} ^ {\mathrm {D}} = \frac {\operatorname* {m a x} _ {i , k} S _ {i , c , k}}{\operatorname* {m i n} _ {i , k} S _ {i , c , k}}, \quad r _ {c} ^ {\mathrm {I}} = \max  _ {i} \frac {\operatorname* {m a x} _ {k} S _ {i , c , k}}{\operatorname* {m i n} _ {k} S _ {i , c , k}}. \tag {14}
$$
These ratios can serve as an indicator of size imbalance at either the dataset or image level. We average  $\log \frac{r_{\mathrm{c}}^{\mathrm{D}}}{r_{\mathrm{c}}^{\mathrm{I}}}$  for "thing" and "stuff" classes separately, and note that  $r_{c}^{\mathrm{I}} = 1$  for all "stuff" classes. In Figure 5 (Appendix B.1), we present the numbers for Cityscapes, Mapillary Vistas, ADE20K, and COCO-Stuff. The size imbalance is considerably reduced at the image level, especially for "stuff" classes. As a result, fine-grained mIoUs can be less biased towards large objects.
- A wealth of statistical information. For example, fine-grained metrics enable to compute worst-case metrics at image and instance levels as discussed in the previous section. We can also plot a histogram for  $\mathrm{mIoU}^{\mathrm{I}}$ , from which we can extract various statistical information and identify worst-case images. These worst-case images can be instrumental in examining the specific scenarios in which a model underperforms, i.e. model auditing. Additionally, employing fine-grained scores facilitates statistical significance testing. Altogether, this yields a more robust and comprehensive comparison between different methods.
- Dataset auditing. Images that consistently yield a low image-level score across various models can be examined. These low scores could potentially be attributed to mislabeling. Furthermore, the presence of discrepancies between image- and object-level labels can result in an abnormal  $\mathrm{mIoU}^{\mathrm{K}}$  values (NaN) because the denominator will become zero. We present the identified mislabels in Appendix B.3.
We provide a large-scale benchmark study, where we train 15 modern neural networks from scratch and evaluate them on 12 datasets that cover various scenes. More details of 15 models and 12 datasets considered in our benchmark study can be found in Appendix A and Appendix B, respectively. The total amount of compute time takes around 1 NVIDIA A100 year.
In our benchmark, we emphasize the importance of training models from scratch (training details are in Appendix C) instead of relying on publicly available pretrained checkpoints. This choice is motivated by several factors:
- Analysis. By training models from scratch, we aim to analyze how specific training strategies can lead to high results on fine-grained metrics. This provides insights into the training process and enables a better understanding of the factors affecting performance.
- Completeness. While there are pretrained models available for popular datasets like Cityscapes and ADE20K, there is a scarcity of checkpoints for other datasets. Training models from scratch ensures a more comprehensive evaluation across different datasets, providing a more complete perspective on model performance.
- Fairness. Publicly available pretrained checkpoints are often trained with different hyperparameters and data preprocessing procedures. Training models from scratch ensures fairness in comparing different architectures and training strategies, as they all start from the same initialization and undergo the same training process.
- Performance. We leverage recent training techniques [33, 50], which have been shown to improve results. By training models from scratch with these techniques, we aim to achieve better performance compared to other publicly available checkpoints. We provide a comparison of our results with those of MMSegmentation [8] in Table 1 and Table 2.
- Statistical significance. Publicly available pretrained checkpoints often stem from a single run, which may lack statistical significance. By training models from scratch, we can perform multiple runs and obtain statistically significant results. This ensures a more reliable evaluation and avoids potential misinterpretations.
Table 1: Comparing the performance of DeepLabV3+-ResNet101 with MMSegmentation. All results are mIoU $^{\text{D}}$ . Red: the best in a column.
Dataset	Cityscapes	ADE20K	VOC	Context
MMSegmentation	80.97	45.47	78.62	53.20
Ours	80.91 ± 0.17	46.32 ± 0.33	81.07 ± 0.28	56.49 ± 0.10
Table 2: Comparing the performance on ADE20K with MMSegmentation. All results are mIoU $^{\text{D}}$ . Red: the best in a column.
Model	DL3+-R101	DL3+MB2	UPN-R101	UPN-Conv	Seg-MiTB4	PSP-R101
MMSegmentation	45.47	34.02	43.82	48.71	48.46	44.39
Ours	46.32 ± 0.33	36.85 ± 0.34	45.89 ± 0.11	51.08 ± 0.42	50.23 ± 0.26	45.69 ± 0.33
Complete results, including tabular entries, image-level histograms and worst-case images are in Appendix G. Besides, we rank 15 models by their averaged ranking across 10 datasets, excluding Nighttime Driving and Dark Zurich. As depicted in Figure 3, we contrast the rank of  $\mathrm{mIoU}^{\mathrm{C}}$  with (a)  $\mathrm{mIoU}^{\mathrm{D}}$ , (b)  $\mathrm{mIoU}^{\mathrm{I}}$ , (c)  $\mathrm{mIoU}^{\mathrm{C}^{\bar{q}}}$ , (d)  $\mathrm{mIoU}^{\mathrm{C}^5}$ . Furthermore, we count the number of times each worst-case image appears for each model, and present the count of three most common worst-case images for each dataset in Table 13 (Appendix G). The key findings are summarized below:
No model achieves the best result across all metrics and datasets. UPerNet-ConvNeXt usually obtains the highest score for most of the metrics, possibly due to the ImageNet-22K pretraining of ConvNeXt, while other backbones are pretrained on ImageNet-1K. However, UPerNet-MiTB4 and SegFormer-MiTB4 outperform UPerNet-ConvNeXt on many worst-case metrics. Since different metrics focus on a certain property of the result, it is important to have a comprehensive comparison using various metrics.
(d)
Figure 3: Contrasting the rank of  $\mathrm{mIoU^C}$  with (a)  $\mathrm{mIoU^D}$ , (b)  $\mathrm{mIoU^I}$ , (c)  $\mathrm{mIoU^{C\bar{q}}}$ , (d)  $\mathrm{mIoU^{C^5}}$ . Models below the green dashed line have a higher rank of  $\mathrm{mIoU^C}$  than that of the counterpart.
Models do not overfit to a particular metric or a specific range of image-level values. As shown in Figure 3, we witness only minor local rank discrepancies when comparing  $\mathrm{mIoU}^{\mathrm{C}}$  with  $\mathrm{mIoU}^{\mathrm{D}}$ , and notably, no rank differences arise when comparing  $\mathrm{mIoU}^{\mathrm{C}}$  with  $\mathrm{mIoU}^{\mathrm{I}}$ . In general, we observe a monotonic trend. This proves that models are not tailored to favor a particular metric. The rank differences between  $\mathrm{mIoU}^{\mathrm{C}}$  and  $\mathrm{mIoU}^{\mathrm{C}^q}$  are also minimal. This implies that the advancements on  $\mathrm{mIoU}^{\mathrm{C}}$  are global and models do not overfit to a specific range of image-level values. However, we identify a larger rank drop of UNet-ResNet101 in  $\mathrm{mIoU}^{\mathrm{C}^5}$ , indicating that UNet may suffer from more challenging examples.