SlowGuess/NeurIPS_2023_OCR huggingface dataset:
DeepLabV3 [6] attaches the Atrous Spatial Pyramid Pooling (ASPP) module to the output of the feature extractor. ASPP applies atrous (dilated) convolutions at a variety of dilation rates in parallel, allowing the model to handle objects of diverse sizes in the image.
DeepLabV3+ [7] builds upon DeepLabV3 and retains the use of the ASPP module. Its primary enhancement lies in incorporating an encoder-decoder structure where the decoder upsamples the low-level features from the encoder and combines them with high-level features to generate detailed segmentations.
PSPNet [58] adopts the Pyramid Pooling Module (PPM) to capture different levels of details in the scene. The idea behind this module is to apply pooling operations at varying scales and concatenate the output alongside the original feature map.
UPerNet [53] applies an encoder-decoder structure. The encoder takes inspiration from the Feature Pyramid Network (FPN) [29], creating a pyramid of features extracted at multiple scales. The decoder merges multi-level feature maps and adopts PPM at the lowest FPN level to bring useful global prior representations.
SegFormer [54] features a lightweight decoder that solely consists of MLP layers for upsampling and fusing multi-level features. It avoids computationally intensive components, and the key for this simplified design is the large receptive field provided by the hierarchical transformer encoder.
A.2 Backbones
ResNet [21] is a type of CNN that uses shortcut connections, or "skip connections". These shortcut connections enable the network to learn an identity function, ensuring that the output of the block is at least as good as its input, and hence the term "residual". This identity function makes it easier for the network to learn and allows gradients to flow directly through the network during backpropagation, which can alleviate the problem of vanishing gradients in deep networks.
ConvNeXt [32] is a pure convolutional model, inspired by the design of vision transformers [13]. It consists of several macro designs (stage ratios, patchified stem, etc), micro-designs (replacing ReLU with GELU [23], substituting BN [26] with LN [1], etc), modern training techniques (AdamW [33], label smoothing [45], etc) and other designs (large kernel sizes, inverted bottlenecks, etc) to modernize ResNets.
EfficientNet [46] is a family of CNN architectures innovatively designed for model scaling. Efficient-Net's central insight is recognizing the interactive nature of the network width, depth, and resolution. Therefore, they need to be scaled together to maintain model efficiency. The key is to adopt a neural architecture search [60] approach to find the best scaling combinations.
MobileNetV2 [44] is designed to be a lightweight, efficient network architecture optimized for speed and size, making it particularly well-suited for mobile and embedded vision applications. It adopts depth-wise separable convolutions from MobileNetV1 [24] to reduce computational costs. Additionally, it leverages linear bottleneck layers to prevent nonlinearities from destroying too much information and inverted residuals blocks to facilitate gradient propagation across multiple layers.
MiT [54] is a type of vision transformer [13] specifically optimized for semantic segmentation tasks. One key aspect of MiT is its hierarchical structure. The model comprises several stages, forming a multi-scale structure that is effective for semantic segmentation. Another noteworthy feature of MiT is its relatively lightweight nature compared to some other vision Transformers, making it more practical for real-world deployment.
MobileViTV2 [36] is a lightweight and low-latency hybrid network for mobile vision tasks. It follows the macro-architecture of MobileViT [35], which adopt a hybrid design of MobileNetV2 blocks [44] and self-attention operations [48]. Its main innovation is a separable self-attention that reduces the computational complexity of self-attention from  $O(n^{2})$  to  $O(n)$ .
Our benchmark include 12 datasets that cover various domains including: (i) street scenes [4, 9, 39, 11, 43], (ii) "thing" and "stuff" [18, 38, 59, 5], (iii) aerial scenes [12]. The number of images in the dataset ranges from 50 (e.g. Nighttime Driving and Dark Zurich) to more than a hundred thousand (e.g. COCO-Stuff); the number of classes varies from binary (e.g. DeepGlobe Road and DeepGlobe Building) to over a hundred (e.g. ADE20K and COCO-Stuff). Table 6 presents an overview of 12 selected datasets. Following is a brief introduction to each.
Cityscapes [9] is a large-scale dataset that focuses on semantic understanding of urban street scenes. It contains a diverse set of stereo video sequences recorded in street scenes from 50 different cities, with high quality pixel-level annotations of 5,000 frames (2,975 for training, 500 for validation and
Table 6: An overview of 12 selected datasets.
Dataset	Scene	#Train	#Val	#Classes	Coverage (%)
Cityscapes [9]	Urban	2,975	500	19	63.22 ± 18.88
Nighttime Driving [11]	Urban	-	50	19	34.42 ± 24.50
Dark Zurich [43]	Urban	-	50	19	57.99 ± 16.28
Mapillary Vistas [39]	Urban	18,000	2,000	65	30.18 ± 22.23
CamVid [4]	Urban	469	232	11	89.53 ± 9.18
ADE20K [59]	"Thing" & "stuff"	20,210	2,000	150	5.63 ± 50.62
COCO-Stuff [5]	"Thing" & "stuff"	118,287	5,000	171	4.92 ± 54.21
PASCAL VOC [18]	"Thing"-only	10,582	1,449	21	11.59 ± 28.54
PASCAL Context [38]	"Thing" & "stuff"	4,996	5,104	60	9.60 ± 41.77
DeepGlobe Land [12]	Aerial	643	160	6	63.33 ± 27.09
DeepGlobe Road [12]	Aerial	4,981	1,245	2	100 ± 0
DeepGlobe Building [12, 15]	Aerial	8,476	2,117	2	88.97 ± 23.30
1,525 for testing). In addition, there are 20,000 coarsely annotated frames also included in the dataset. In this study, we utilize the finely annotated training and validation images exclusively.
Nighttime Driving [11] and Dark Zurich [43] are datasets that provide images of various road scenes captured during the nighttime. They are designed to facilitate research in autonomous driving, and specifically addresses the challenges associated with low-light and nighttime conditions. They incorporate fine pixel-level Cityscapes [9] labels.
Mapillary Vistas [39] is a large-scale street-level imagery dataset used primarily for semantic segmentation tasks. It is one of the most diverse publicly available datasets of street-level imagery, collected from numerous cities around the world under a variety of weather and lighting conditions. While images are mainly taken from mobile devices, in general the dataset spans a wide range of different camera types, including head- or car-mounted ones. This diversity in viewpoints is useful for training more robust machine learning models that can generalize well to novel scenarios.
CamVid [4] is one of the pioneering datasets used for semantic segmentation. The dataset comprises of footage captured from the perspective of a driving automobile, providing a vision of the road scene ahead. It has been collected over different times of the day, offering an excellent selection of varying lighting conditions and appearances.
ADE20K [59] is a widely used dataset for scene parsing. One unique feature of the ADE20K dataset is its high diversity in object classes. The dataset has annotations for over 150 object categories, and includes a diverse set of scenes from both indoor and outdoor environments. This diversity makes the dataset challenging and well-suited for developing and benchmarking algorithms for semantic segmentation.
COCO-Stuff [5] is an extension of the original COCO [30] dataset which is renowned for its diverse set of high-quality, real-world images containing complex everyday scenes of common objects in their natural context. It augments the COCO dataset by adding pixel-level annotations for "stuff" classes. Thus, it contains the same images as the original COCO dataset and is well-suited for semantic segmentation tasks.
PASCAL VOC [18] is composed of images collected from the Flickr photo-sharing web-site. The labels include a diverse set of classes, including various types of vehicles, animals, and indoor objects. The original dataset contains 1,464 images for training, 1,449 images for validation, and a further 1,456 images for testing. Extra annotations are provided in [20], leading to a total of 10,582 training images.
PASCAL Context [38] is an extension of the popular PASCAL VOC dataset [18] where only object classes of interest were annotated, leaving a large portion of pixels marked as "background" or "unlabeled". In the PASCAL Context dataset, the goal is to provide a comprehensive understanding of the scene, thus all pixels in an image are assigned a label, giving the "context" of the objects.
DeepGlobe [12] contains high-resolution satellite images from around the globe, covering various geographical locations, features, and phenomena. It offers three distinct satellite image understanding datasets: land cover classification, road extraction, and building detection. With this rich assortment of large-scale satellite imagery, DeepGlobe aids in propelling advancements in geospatial analysis.
B.1 "Thing" and "Stuff"
The differences between "thing" and "stuff" encompass various aspects [5]:
Figure 5: Results of averaged log  $\frac{r_c^{\mathrm{D}}}{r_c^{\mathrm{I}}}$  for "thing" and "stuff" classes on Cityscapes, Mapillary Vistas, ADE20K and COCO-Stuff.
- Shape: "thing" has characteristic shapes (e.g. person, car, phone), whereas "stuff" is amorphous (e.g. sky, road, water).
- Size: "thing" occurs in characteristic sizes with little variance, whereas "stuff" is highly variable in size.
- Parts: "thing" possess identifiable parts, whereas "stuff" does not. For example, a fragment of sky is still considered sky, but a wheel alone does not constitute a car.
- Instances: "stuff" is typically uncountable and has no clearly defined instances.
- Texture: "stuff" is typically highly textured.
The characterization of "thing" and "stuff" is usually a decision made by the dataset creator. However, many datasets do not define which classes are "thing" or "stuff" (e.g. Cityscapes, Mapillary Vistas, ADE20K, etc). In this work, for datasets that provide instance-level annotations (Cityscapes, Mapillary Vistas, ADE20K, COCO-Stuff), we treat classes that have instance-level labels as "thing" classes, and as "stuff" classes otherwise.
B.2 Dataset Coverage
For each dataset, we calculate the proportion of distinct classes present in each image to the total number of classes within that dataset. We define the coverage of a dataset as the mean and the normalized standard deviation over all images. A dataset exhibiting a high mean and a low normalized standard deviation is considered to have high coverage, implying low variations in contexts.
Urban-scene datasets, such as Cityscapes, Mapillary Vistas, and CamVid, generally demonstrate high coverage. In contrast, "thing" and "stuff" datasets like ADE20K, COCO-Stuff, and PASCAL Context typically exhibit lower coverage as they incorporate a wide range of indoor and outdoor scenes. Datasets with low coverage often showcase more spread-out image-level histograms (see Appendix G) with fat tails, indicating a diverse range of image-level IoU values.
B.3 Dataset Auditing
Images that consistently yield a low image-level score across different models can be inspected. These low scores could potentially be attributed to mislabeling within the images. For example, in COCO-Stuff, instances in "000000064574" and "000000053529" are erroneously labeled as "stuff" (as shown in Figures 31 - 33). This mislabeling leads to most models registering an image-level IoU of 0.
Additionally, the presence of inconsistencies between image- and object-level labels can lead to NaN values during the computation of  $\mathrm{mIoU^K}$ , because the denominator in Eq. (9) will become zero. This provides a way to identify mislabels within the dataset. As demonstrated in Table 7, we find these mislabels exist in Cityscapes and ADE20K. Specifically, only a single mislabeled image is in the validation set (highlighted in red), and only a single mislabeled object is "stuff" (highlighted in green).
Table 7: Identified mislabels in Cityscapes and ADE20K. ID: object ID. Reason: 1 means that the object is present in the object-level label but absent in the image-level label; 2 means that the object is included in the image-level label but missing from the object-level label. Red: the image is in the validation set. Green: the object is "stuff".
Dataset	File	Class	ID	Reason
Cityscapes	strasbourg_000000_029577	Terrain	22	2
Cityscapes	hanover_000000_053604	Rider	25000	2
Cityscapes	aachen_000022_000019	Car	26000	2
ADE20K	ADE_val_00000899	Door	3407624	2
ADE20K	ADE_train_00000939	Door	3407624	1
ADE20K	ADE_train_00019938	Door	3407624	1
ADE20K	ADE_train_00005928	Seat	14745351	1
ADE20K	ADE_train_00018960	Seat	14745351	1
ADE20K	ADE_train_00018960	Seat	15859465	2
ADE20K	ADE_train_00015355	Seat	14745351	1
ADE20K	ADE_train_00001176	Seat	14745351	1
ADE20K	ADE_train_00010454	Seat	14745351	1
ADE20K	ADE_train_00008027	Seat	14745351	1
ADE20K	ADE_train_00016771	Seat	13361409	2
ADE20K	ADE_train_00013611	Seat	14745351	1
ADE20K	ADE_train_00013611	Seat	16116992	2
ADE20K	ADE_train_00002139	Seat	14745351	1
ADE20K	ADE_train_00005924	Seat	15204121	2
ADE20K	ADE_train_00016746	Seat	14745351	1
ADE20K	ADE_train_00008022	Seat	14745351	1
ADE20K	ADE_train_00008022	Seat	14410752	2
ADE20K	ADE_train_00019174	Seat	13034752	2
ADE20K	ADE_train_00019174	Seat	16383749	2
ADE20K	ADE_train_00019174	Seat	14673932	2
ADE20K	ADE_train_00019174	Seat	13696785	2
ADE20K	ADE_train_00019177	Seat	12909092	2
ADE20K	ADE_train_00013522	Seat	14745351	1
ADE20K	ADE_train_00001354	Booth	13369599	1
ADE20K	ADE_train_00001387	Booth	13369599	1
ADE20K	ADE_train_00001387	Booth	11469055	2
ADE20K	ADE_train_00001392	Booth	13369599	1
ADE20K	ADE_train_00001392	Booth	11796735	2
ADE20K	ADE_train_00018451	Booth	13369599	1
ADE20K	ADE_train_00015050	Booth	13369599	1
ADE20K	ADE_train_00001573	Booth	13369599	1
ADE20K	ADE_train_00001573	Booth	11469055	2
ADE20K	ADE_train_00001573	Booth	13769215	2
ADE20K	ADE_train_00001573	Booth	13893861	2
ADE20K	ADE_train_00001573	Booth	11993599	2
ADE20K	ADE_train_00001573	Booth	15275519	2
ADE20K	ADE_train_00012787	Booth	13369599	1
ADE20K	ADE_train_00012787	Booth	14943216	2
ADE20K	ADE_train_00012787	Booth	11999999	2
ADE20K	ADE_train_00012787	Booth	12715263	2
ADE20K	ADE_train_00012713	Booth	13369599	1
ADE20K	ADE_train_00014369	Booth	13369599	1
ADE20K	ADE_train_00005268	Booth	13369599	1
ADE20K	ADE_train_00001386	Booth	13369599	1
ADE20K	ADE_train_00016930	Booth	13369599	2
Training details. We use AdamW [33] with a weight decay of 0.01. The learning rate starts from 1e-6 and linearly warms up during the first  $1\%$  iterations to the initial learning rate which is 6e-5 for ResNet101/50, ConvNeXt, MiTB4/B2, and 6e-4 for ResNet34/18, EfficientNetB0, MobileNetV2, MobileViTV2. The learning rate is then decayed in a "poly" policy with an exponent of 1. The number of training iterations and the crop size for each dataset is summarized in Table 8. All models are trained with a combination of the cross-entropy loss and the Jaccard metric loss (JML) [50], balancing with weights of 0.25 and 0.75, respectively. JML optimizes for  $\mathrm{mIoU}^{\mathrm{I}}$  and  $\mathrm{mIoU}^{\mathrm{C}}$  with equal weights, as discussed in Section 5.2.2. Data augmentations including (i) random scaling in the range of [0.5, 2], and (ii) random horizontal flipping with a probability of 0.5.
All results are based on three independent runs and are presented in the format of mean±standard deviation. We do not apply any inference-time augmentation such as multi-scale inference and flipping.
Dataset splits. For Nighttime Driving and Dark Zurich, we evaluate models trained on Cityscapes on their test sets, following MMSegmentation [8]. For CamVid, we merge "train" and "val" into the training set, and "test" as the test set. For DeepGlobe, since it does not provide an official data split, we select the first  $80\%$  (sorted by indexes) of training images as the training set and the rest as the test set. For all other datasets, we adhere to the official training/validation splits.
In general, our implementations (models, dataset preprocessing, training recipes) closely follow MMSegmentation [8], except that we adopt a smaller number of training iterations to save the computational costs and utilize recent training techniques [50, 33] for superior performance. As demonstrated in Table 1 and Table 2, we either match or exceed the results of MMSegmentation with less training iterations (see Table 8). Also note that our training hyper-parameters are not optimized for mIoU $^{\text{D}}$  (see Table 4).
Table 8: The number of training and warmup iterations and the crop size for each dataset. MMSegmentation: the number of training iterations in MMSegmentation [8].
Dataset	MMSegmentation	#Iterations	Warmup	Crop Size
Cityscapes [9]	80,000	40,000	400	512 × 1024
Nighttime Driving [11]	-	-	-	512 × 1024
Dark Zurich [43]	-	-	-	512 × 1024
Mapillary Vistas [39]	-	160,000	1,600	512 × 1024
CamVid [4]	-	20,000	200	512 × 512
ADE20K [59]	160,000	80,000	800	512 × 512
COCO-Stuff [5]	160,000	80,000	800	512 × 512
PASCAL VOC [18]	40,000	40,000	400	512 × 512
PASCAL Context [38]	80,000	40,000	400	512 × 512
DeepGlobe Land [12]	-	20,000	200	512 × 512
DeepGlobe Road [12]	-	40,000	400	512 × 512
DeepGlobe Building [12, 15]	-	40,000	400	512 × 512
D NULL in Fine-grained mIoUs
For each image, based on the presence or absence of a class in both the prediction and the ground truth, we can identify four cases:
1. The class is present in both the prediction and the ground truth.
2. The class is absent in both the prediction and the ground truth (true negatives).
3. The class is absent in the prediction but present in the ground truth (false negatives).
4. The class is present in the prediction but is absent in the ground truth (false positives).
In multi-class segmentation, for cases 1 and 3, we compute the per-image-per-class score as usual. In particular, the score for case 3 is 0. For cases 2 and 4, we define the score as NULL. In binary segmentation, if it is viewed as 2-class segmentation, then we can refer to the definition above. Otherwise, if it is considered as foreground-background segmentation, the definition for cases 1 and 3 remains consistent with the multi-class setting. For case 2, we define the score as 1 and for case 4, we define the score as 0. A summary is shown in Table 9.
Notably, in the multi-class setting, our definition differs from Csurka et al. [10]. In their definition, the score is 0 for case 4. Our modification aims at preventing the metric from overly reacting to minor false positives, e.g. a few pixels are mispredicted as a car absent from the ground truth. In safety-critical settings, false negatives should have higher importance, e.g. overlooking a car present in the ground truth. Overemphasizing such minor false positives might inadvertently diminish the significance of false negatives. A concerning scenario could be when the ground truth contains only a small number of classes, yet each unrelated class has a few pixels erroneously present in the prediction. It is pivotal to note that in our definition, these mispredicted pixels still incur penalties—they are false negatives with respect to the corresponding ground-truth classes.
Besides, we do not want to discriminate between false positives solely based on whether the predicted class is present in the ground truth or not. Generally speaking, mispredicting Road as either Car or Truck should incur a similar penalty. However, according to the definition of [10], if Car is present in the ground truth and Truck is not, mispredicting Road as Car may only result in a mild penalty, whereas mispredicting Road as Truck will incur a full penalty.
To further illustrate the differences between the definition of [10] and ours, consider an example image composed of just 4 pixels, with a total of 6 classes in the dataset, as presented in Table 10. We compare the two ways of computing the score, as detailed in Table 11. When calculating the score as prescribed by [10], false positives that appear in the prediction but not in the ground truth incur a considerable penalty. Consequently, the mean IoU value for this image is only 0.25 under their definition, but is 0.5 under ours.
For a more empirical comparison, in Table 12, we present the two ways for calculating  $\mathrm{mIoU^C}$  using DeepLabV3+-ResNet101. It is evident that, when the score is derived following [10], there is a more aggressive penalty, resulting in a generally lower value.
In the end, although we present our definition with the belief that it offers advantages over [10] in certain scenarios, we understand that each metric will have its pros and cons. It is essential to note that our aim is not to assert that our metrics are the best or only way, but to highlight this distinction. Ultimately, end users could select the version most aligned with their specific requirements.
Table 9: The definition of  $\mathrm{IoU}_{i,c}$  in multi-class and binary segmentation.
Case	Multi-class	Binary
1	IoU i, c	IoU i, c
2	NULL	1
3	0	0
4	NULL	0
Table 10: Predicted and ground-truth classes of an example image composed of just 4 pixels, with a total number of 6 classes in the dataset.
Pred	0	2	1	3
GT	0	0	1	1
Table 11: The two ways of computing  $\mathrm{IoU}_{i,c}$  for the example image.
[10]	0.5	0.5	0	0	NULL	NULL
Ours	0.5	0.5	NULL	NULL	NULL	NULL
Table 12: The two ways of computing  $\mathrm{mIoU^C}$  using DeepLabV3+-ResNet101.
Definition	Cityscapes	Mapillary	ADE20K	COCO-Stuff	VOC	Context	Land
[10]	56.14 ± 0.43	23.95 ± 0.19	20.23 ± 0.05	20.98 ± 0.09	62.30 ± 0.58	25.92 ± 0.07	40.14 ± 1.39
Ours	70.18 ± 0.16	46.23 ± 0.12	46.52 ± 0.30	41.98 ± 0.04	79.07 ± 0.18	55.89 ± 0.07	53.76 ± 0.40
E Other Per-instance mIoUs
Recall that we have instance-level  $\mathrm{TP}_{i,c,k}$ ,  $\mathrm{FN}_{i,c,k}$  and image-level  $\mathrm{FP}_{i,c}$ . We can solve a minimization problem:
$$
\min  \sum_ {k} \frac {\mathrm {T P} _ {i , c , k}}{\mathrm {T P} _ {i , c , k} + \mathrm {F N} _ {i , c , k} + \mathrm {F P} _ {i , c , k}} \tag {15}
$$
$$
\text {s . t .} \sum_ {k} \mathrm {F P} _ {i, c, k} = \mathrm {F P} _ {i, c}. \tag {16}
$$
Alternatively, we can replace the minimization with maximization. Denote the resulting instance-level values as  $\mathrm{IoU}_{\min c,i,k}$  and  $\mathrm{IoU}_{\max c,i,k}$ , respectively. They provide a lower/upper bound on the value of  $\mathrm{IoU}_{i,c,k}$ :
$$
\operatorname {I o U} _ {\min  c, i, k} \leq \operatorname {I o U} _ {i, c, k} \leq \operatorname {I o U} _ {\max  c, i, k}. \tag {17}
$$
Compared to distributing image-level FP according to the size of each instances (as we defined in Section 2),  $\mathrm{IoU}_{\min c,i,k}$  and  $\mathrm{IoU}_{\max c,i,k}$  may yield results misaligned with our intuition. For example, consider a case with two instances where  $\mathrm{TP}_1 = 1$ ,  $\mathrm{TP}_2 = 10$ ,  $\mathrm{FN}_1 = \mathrm{FN}_2 = 0$ ,  $\mathrm{FP}_1 + \mathrm{FP}_2 = 2$ . Then we have
$$
\frac {1}{1 + \mathrm {F P} _ {1}} + \frac {1 0}{1 0 + \mathrm {F P} _ {2}}, \tag {18}
$$
which is minimized when  $\mathrm{FP}_1 = 2, \mathrm{FP}_2 = 0$ . Thus,  $\mathrm{IoU}_{\min c,i,k}$  will distribute FP to the smaller instance when there is an instance imbalance.
Consider another example with two instances such that  $\mathrm{TP}_1 = \mathrm{TP}_2 = 10$ ,  $\mathrm{FN}_1 = \mathrm{FN}_2 = 0$ ,  $\mathrm{FP}_1 + \mathrm{FP}_2 = 10$ . Then we have
$$
\frac {1 0}{1 0 + \mathrm {F P} _ {1}} + \frac {1 0}{1 0 + \mathrm {F P} _ {2}}, \tag {19}
$$
which is maximized when  $\mathrm{FP}_1 = 10, \mathrm{FP}_2 = 0$  or  $\mathrm{FP}_1 = 0, \mathrm{FP}_2 = 10$ . Therefore,  $\mathrm{IoU}_{\max c,i,k}$  will distribute FP to one instance when instances are of the equal size.
F The Jaccard Metric Loss
IoU is defined over sets that are discrete and therefore cannot be directly optimized by neural networks. Given two sets  $x, y \in \{0,1\}^p$ , the Jaccard metric loss (JML) [50] rewrites IoU as a function of set difference:
$$
\mathrm {I o U} = \frac {| x \cap y |}{| x \cup y |} = \frac {| x | + | y | - | x \Delta y |}{| x | + | y | + | x \Delta y |}, \tag {20}
$$