SlowGuess/NeurIPS_2023_OCR huggingface dataset:
What a PRESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentation
Benedikt Blumenstiel*  
Karlsruhe Institute of Technology  
IBM Research Europe  
benedikt.blumenstiel@kit.edu
Hilde Kühne  
University of Bonn  
MIT-IBM Watson AI Lab  
hildegard.kuehne@ibm.com
Johannes Jakubik*  
Karlsruhe Institute of Technology  
IBM Research Europe  
johannes.jakubik@kit.edu
Michael Vössing  
Karlsruhe Institute of Technology  
IBM Germany  
michael.voessing@kit.edu
While semantic segmentation has seen tremendous improvements in the past, there are still significant labeling efforts necessary and the problem of limited generalization to classes that have not been present during training. To address this problem, zero-shot semantic segmentation makes use of large self-supervised vision-language models, allowing zero-shot transfer to unseen classes. In this work, we build a benchmark for Multi-domain Evaluation of Semantic Segmentation (MESS), which allows a holistic analysis of performance across a wide range of domain-specific datasets such as medicine, engineering, earth monitoring, biology, and agriculture. To do this, we reviewed 120 datasets, developed a taxonomy, and classified the datasets according to the developed taxonomy. We select a representative subset consisting of 22 datasets and propose it as the POSS benchmark. We evaluate eight recently published models on the proposed POSS benchmark and analyze characteristics for the performance of zero-shot transfer models. The toolkit is available at https://github.com/blumenstiel/MESS.
Zero-shot semantic segmentation utilizes self-supervised models such as CLIP to minimize labeling requirements during training and to improve model generalization. Recent models are already able to include classes during inference that were not present during training. For this reason, zero-shot semantic segmentation is becoming increasingly relevant for real-world scenarios. In particular, the performance on domain-specific datasets such as earth monitoring datasets, as visualized in Figure 1, becomes more and more relevant. Current standard benchmarks tend to focus on in-domain tasks but do not capture performance comparisons across domains. This is problematic because it limits insight into the applicability of zero-shot semantic segmentation to new domains. It also makes it difficult to assess whether architectures might be suitable for datasets that pose additional challenges (e.g., different sensor types or specialized vocabulary). To better understand the behavior of zero-shot semantic segmentation models on a wider range of more complex, domain-specific datasets, we propose a holistic Multi-domain Evaluation of Semantic Segmentation (MESS). To this end, we have examined 120 datasets and classified them within a developed taxonomy. We leverage our benchmark to evaluate eight recently published models for zero-shot semantic segmentation including
Figure 1: CAT-Seg-L [7] predictions for a range of domain-specific datasets. The model achieves promising predictions on everyday and satellite images, while it faces difficulties in segmenting small segments such as blood vessels, and distinguishing similar classes such as bird species.
the state-of-the-art models on 22 datasets from the fields of medical sciences, earth monitoring, agriculture and biology, engineering as well as a general domain including datasets on, e.g., driving scenes, maritime scenes, paintings, and body parts. Our evaluation focuses on zero-shot text-to-mask models—also known as open-vocabulary semantic segmentation (OVSS)—and later also compares their performance with zero-shot point-to-mask and box-to-mask approaches of SAM [20]. Using the proposed benchmark, we identify and analyze several characteristics that influence the performance of OVSS models, i.a., showing that the semantic, textual similarity of classes as well as the underlying sensor type, significantly affect the performance of current models.
Our experiments reveal various challenges for the application of zero-shot semantic segmentation on domain-specific datasets, e.g., we found that the selection of class labels can significantly affect the quality of predictions. We also observe that the models are sensitive to the semantics of the textual prompts, e.g., general terminology leads to better performance than domain-specific terminology. Overall, we hope that our benchmark will support accelerating zero-shot semantic segmentation and improve the real-world applicability of semantic segmentation in general.
We summarize the contributions of this work as follows: (1) We develop a taxonomy based on a quantitative and qualitative analysis of a broad variety of semantic segmentation datasets. (2) We propose a new benchmark for multi-domain semantic segmentation. (3) We evaluate eight zero-shot models on theMESS benchmark with an in-depth analysis of the task characteristics.
2.1 Zero-shot semantic segmentation
Large-scale self-supervised pre-training has revolutionized the field of computer vision over the last couple of years. One stream of work focuses on vision-language pre-training such as in recent foundation model architectures like CLIP [37], ALIGN [19], and Florence [54]. These models are trained on image-text pairs and encode both visual and text semantics in a shared embedding space. This approach particularly enables so-called open-vocabulary image classification by computing the similarity between the embeddings of the image and the embeddings of natural language describing the classes in the image. The text describing the images can be any arbitrary textual sequence and might describe classes on the images that have been unseen during training. This is in contrast to recent segmentation models, like Segment-Anything (SAM, see [20]), which are trained only on image data and therefore do not include a text encoder to encode semantic concepts. Hence, segmentation models like SAM do not facilitate open-vocabulary out of the box and need to be adapted to support the processing of textual information (e.g., by using additional models that generate text embeddings or models that provide bounding boxes as input such as Grounding DINO [27]).
Early approaches in OVSS have been built upon standard zero-shot semantic segmentation, such as ZS3Net [5], using simple word2vec text encoders. Subsequent two-stage approaches made use of mask proposals based on MaskFormer [6] in stage 1 followed by predictions of each mask by
CLIP [9, 10, 25, 51]. Recently, one-stage frameworks like SAN [50] generate masks in a side adapter network during the CLIP inference. Therefore, CLIP does not classify many mask proposals but only the image ones, resulting in a faster inference. Other mask-based models like GroupViT [49] and ViewCo [40] are grouping pixels into larger segments which are then classified.
Decoder-focused approaches such as DenseCLIP [39] and LSeg [23] encode the image with CLIP and obtain the pixel-level patch embeddings. Because the pre-training is focused on the class embedding, the approaches append additional decoders to refine the patch embeddings. For this refinement, CAT-Seg [7] utilizes multiple stages of cost aggregation to generate the final segmentation mask. PACL [34] aligns patch embeddings and class embeddings during training and, as a result, does not require segmentation-specific training data or additional modules. Zero-shot semantic segmentation models have been combined with other tasks as well. OpenSeeD [55] implements open-vocabulary for object detection and segmentation. SEEM [58] processes text prompts and additional inputs like visual prompts similar to SAM. Apart from differences in the architecture, the models vary in the training process—particularly in fine-tuning CLIP's vision encoder.
2.2 Evaluation and benchmarking of zero-shot semantic segmentation
Zero-shot semantic segmentation models are typically evaluated on datasets consisting exclusively of everyday images, such as ADE20K [56], Pascal Context [33], and Pascal VOC [12]. These datasets are the de facto standard for evaluating these models (see [7, 14, 25, 50, 51, 57]). Few studies have considered additional datasets. Notably, Zou et al. [57] proposed a Segmentation in the Wild (SegInW) benchmark with 25 datasets. However, the majority of the datasets in SegInW still consist of everyday images with only two exceptions: brain tumor segmentation and a bird's eye view in stables. To the best of our knowledge, zero-shot semantic segmentation and OVSS have not been evaluated on other datasets. Outside of zero-shot semantic segmentation and OVSS, semantic segmentation is usually evaluated based on collections of datasets, like MSeg [22]. These datasets generally only include everyday images, indoor scenes, and driving datasets and lack domain-specific datasets. SAM has been evaluated on 23 instance segmentation datasets in a point-to-mask setting [20]. This collection of datasets is the most extensive for segmentation tasks but still misses domains, such as engineering and earth monitoring. Other works evaluate specifically domain dataset collections such as medical tasks [32] or satellite data [21].
Table 1: Multi-domain benchmark for zero-shot semantic segmentation models consisting of 22 downstream tasks, a total of 448 classes, and 25,079 images.
Dataset	Domain	Sensor type	Segment size	Number of classes	Class similarity	Vocabulary	Number of images	Task
BDD100K [53]	General	Visible spectrum	Medium	19 (Medium)	Low	Generic	1,000	Driving
Dark Zurich [41]	Visible spectrum	Medium	20 (Medium)	Low	Generic	50	Driving
MHP v1 [24]	Visible spectrum	Small	19 (Medium)	High	Task-spec.	980	Body parts
FoodSeg103 [47]	Visible spectrum	Medium	104 (Many)	High	Generic	2,135	Ingredients
ATLANTIS [11]	Visible spectrum	Small	56 (Many)	Low	Generic	1,295	Maritime
DRAM [8]	Visible spectrum	Medium	12 (Medium)	Low	Generic	718	Paintings
iSAID [46]	Earth Monitoring	Visible spectrum	Small	16 (Medium)	Low	Generic	4,055	Objects
ISPRS Potsdam [4]	Multispectral	Small	6 (Few)	Low	Generic	504	Land use
WorldFloods [31]	Multispectral	Medium	3 (Binary)	Low	Generic	160	Floods
FloodNet [38]	Visible spectrum	Medium	10 (Few)	Low	Task-spec.	5,571	Floods
UAVid [29]	Visible spectrum	Small	8 (Few)	High	Task-spec.	840	Objects
Kvasir-Inst. [18]	Medical Sciences	Visible spectrum	Medium	2 (Binary)	Low	Generic	118	Endoscopy
CHASE DB1 [13]	Microscopic	Small	2 (Binary)	Low	Domain-spec.	20	Retina scan
CryoNuSeg [30]	Microscopic	Small	2 (Binary)	Low	Domain-spec.	30	WSI
PAXRay-4 [42]	Electromagnetic	Large	4x2 (Binary)	Low	Domain-spec.	180	X-Ray
Corrosion CS [3]	Engineering	Visible spectrum	Medium	4 (Few)	High	Task-spec.	44	Corrosion
DeepCrack [28]	Visible spectrum	Small	2 (Binary)	Low	Generic	237	Cracks
ZeroWaste-f [2]	Visible spectrum	Medium	5 (Few)	High	Generic	929	Conveyor
PST900 [43]	Electromagnetic	Small	5 (Few)	Low	Generic	288	Thermal
SUIM [17]	Agriculture and Biology	Visible spectrum	Medium	8 (Few)	Low	Generic	110	Underwater
CUB-200 [45]	Visible spectrum	Medium	201 (Many)	High	Domain-spec.	5,794	Bird species
CWFID [15]	Visible spectrum	Small	3 (Few)	High	Generic	21	Crops
3 PRESS benchmark
Following the HELM benchmark [26] proposed for the evaluation of large language models, we develop a taxonomy with task characteristics for semantic segmentation and retrieve a set of more than 500 datasets that we review as part of the benchmark creation. For the development of the taxonomy, we use a method proposed by Nickerson et al. [35]. We start the development of the taxonomy by specifying the so-called meta-characteristic of the taxonomy (i.e., our goal): identify visual and language characteristics of downstream tasks influencing the performance of zero-shot semantic segmentation models. We then initialize the taxonomy in a conceptual-to-empirical cycle based on a review of other benchmarks and literature. Next, we refine the taxonomy in multiple empirical-to-conceptual iterations. We reviewed semantic segmentation datasets on Papers with Code, Kaggle, and additional test datasets used by recent segmentation models. We repeatedly reduced the dimensions of the taxonomy to the most meaningful ones for the meta-characteristic. We then conducted a statistical analysis of potential taxonomy dimensions to identify and remove similar or overlapping dimensions (see supplementary material). We identified multiple dimensions that highly correlate with each other like color map and sensor type, segment size and segments per image, as well as viewpoint and domain. Based on this analysis, we discarded color map, resolution, segments per image, and viewpoint. The final taxonomy matches all ending conditions [35]. While the proposed taxonomy identifies the most important dimensions and characteristics validated based on 120 classified datasets, there may be additional dimensions that influence the performance of zero-shot semantic segmentation models in specific cases. Overall, we observe that certain characteristics are more likely to co-occur. E.g., binary datasets typically imply a low class similarity, whereas task-specific vocabulary is often associated with a high similarity between the task-specific classes. We account for this imbalance in the distribution of the characteristics and reflect it in our benchmark.
Following the taxonomy development, we selected a representative set of datasets so that theMESS benchmark is informative, reproducible, and manageable. Specifically, we filtered the 120 classified datasets based on four exclusion criteria: each dataset has an official and annotated validation or test set, high annotation quality, moderate disk usage, and sufficient image size. Next, we selected a subset that consists of complementing use cases to avoid duplication and covers all characteristics of the taxonomy. We present the 22 selected datasets and their characteristics within the taxonomy's dimensions in Table 1. These datasets cover a variety of applications, resulting in a holistic evaluation of domain-specific applications. We publish this newMESS benchmark at https://blumenstiel.github.iomess-benchmark and invite others to suggest additional datasets and refine classes for future versions.
During dataset selection, we have not identified any ethical issues with these datasets based on the information provided by the data sources. Our use follows the terms and conditions set by the data providers, and we list the corresponding licenses in the supplementary material. However, we acknowledge the importance of considering the societal impact of our work. FMs, such as CLIP, are pre-trained on vast corpora of data that may contain biases. We refer to Agarwal et al. [1] for a detailed analysis of biases in CLIP. While the majority ofMESS datasets are less prone to such biases, some may include data specific to gender or geographic regions. We believe that assessing model performance across a range of datasets can help to identify and mitigate the impact of biases.
In this section, we provide a brief definition of the zero-shot semantic segmentation task, describe the metrics, and outline implementation details.
4.1 Task
Let  $I$  denote an image with a set of candidate classes  $\mathcal{C} = \{C_1, C_2, \dots, C_N\}$ , where each candidate class  $C_i$  is described in natural language. Zero-shot semantic segmentation models then assign a class  $C_i$  to each pixel of  $I$ . The number of candidate classes  $N$  can vary during inference (e.g., different downstream tasks) and, additionally, the model may not have seen the candidate classes during training. This is in contrast to traditional semantic segmentation, where the set of classes is fixed during training and inference [7]. Each dataset represents a set of images with the same label set, and in our evaluations, none of the models is trained on the datasets from the benchmark
or the same set of candidate classes. However, it is reasonable to assume the evaluated classes have been present in the pre-training of the underlying vision-language models (like CLIP). All evaluated models have been trained on images with three channels (i.e., RGB). To account for datasets with varying numbers of input channels we mapped them to RGB (i.e., inputs with a single channel are mapped to RGB, for multispectral inputs we selected a subset of three channels).
Following common practice, we evaluate all models using the mean of class-wise intersection over union (mIoU) [6, 7, 25, 50, 51]. We split very large images from the earth monitoring datasets into smaller patches of  $1024 \times 1024$  pixels. Further, we use an IRRG color map for multispectral datasets (ISPRS Potsdam and WorldFloods) and select the thermal data in PST900. All other datasets include images with one or three channels. Across our implementation, we use PyTorch [36] and Detectron2 [48] for implementing the data loaders. For the convenience of users and contributors to our benchmark, we additionally provide wrappers for torchvision and MMSeg to process datasets in the Detectron2 dataset catalog. We did not train any models but used the publicly available weights and model configurations. The evaluation was conducted on an NVIDIA V100S.
4.3 Models
We utilize ourMESS benchmark to evaluate a range of recent models for zero-shot semantic segmentation including the state-of-the-art, selecting models based on the reported performance and the availability of official code and weights. OVSeg [25], SAN [50], and CAT-Seg [7] represent the state-of-the-art across different approaches in the architecture for zero-shot semantic segmentation (i.e., two-stage mask-based, one-stage mask-based, and pixel-based). We additionally consider ZSSeg [51] and ZegFormer [9] which are frequently consulted as baseline models e.g. by [7, 25, 50]. The previously listed models are based on CLIP and use COCO Stuff to train the additional segmentation modules. Additionally, OVSeg uses COCO Captions for fine-tuning. X-Decoder [57] and OpenSeeD [55] are part of our evaluation since these approaches do not make use of CLIP but are based on UniCL [52] (i.e., their public versions). X-Decoder and OpenSeeD are trained on multiple datasets which we detail in the supplementary material.
To account for recent developments in the field, we additionally include SAM [20] in our evaluations. Standard SAM can only process visual prompts and does not facilitate text-to-mask settings. Therefore, we validated other ways to make use of SAM. We implement Grounded-SAM [16] using the predicted bounding boxes from Grounding DINO [27] as input for SAM and thereby enabling an open-vocabulary setting (i.e., text-to-mask). This serves as a baseline to better understand the potential of SAM-based text-to-mask models. The overall evaluation time per model on theMESS benchmark varies in our experiments between 1 hour for SAN-B and 14.5 hours for OVSeg-L.
In the following, we provide a holistic overview of the performance of multiple zero-shot semantic segmentation models based on our MESS benchmark. We conduct a range of in-detail analyses of model performances across the dimensions of our taxonomy including sensor types, the class similarity, and the vocabulary—additional experiments are included in the supplementary material.
5.1 Multi-domain zero-shot semantic segmentation
We provide a quantitative comparison across models and all datasets summarized by their domain in Table 2 and per dataset results in Fig. 2 and 3. We add a random prediction as a lower bound by calculating the expected mIoU value with uniformly distributed predictions over all classes. In addition, we report fully supervised results based on the current SOTA from supervised semantic segmentation (see supplementary material). Overall, CAT-Seg-L achieves a strong performance across domains with an average mIoU of  $38.14\%$ , followed by its base and huge version. CAT-Seg is followed by SAN-L with a performance of  $30.06\%$ . Notably, the performance of zero-shot CAT-Seg-L in the general domain is only 8.69pp (average mIoU) below the performance of supervised SOTA approaches. In comparison, CAT-Seg-L reaches on average  $50.36\%$  of the supervised performance in earth monitoring and  $54.18\%$  on medical sciences. The performance gap compared to supervised
Table 2: mIoU results averaged by the dataset domain. Best-performing models are highlighted in bold, and the second-best are underlined. Random represents the randomly expected mIoU with uniformly distributed predictions. The best supervised models are separately selected for each dataset (see supplementary material for the supervised models and results).
Model	Parameters	Inference (s/iter)	General	Earth Monit.	Medical Sciences	Engineer.	Agri. and Biology	Mean
Random (LB)	1.17	7.11	29.51	11.71	6.14	10.27
Best supervised (UB)	48.62	79.12	89.49	67.66	81.94	70.99
ZSSeg-B [51]	211M	0.49	19.98	17.98	41.82	14.0	22.32	22.73
ZegFormer-B [9]	210M	0.18	13.57	17.25	17.47	17.92	25.78	17.57
X-Decoder-T [57]	164M	0.1	22.01	18.92	23.28	15.31	18.17	19.8
SAN-B [50]	158M	0.04	29.35	30.64	29.85	23.58	15.07	26.74
OpenSeeD-T [55]	116M	0.08	22.49	25.11	44.44	16.5	10.35	24.33
CAT-Seg-B [7]	181M	0.17	34.96	34.57	41.65	26.26	29.32	33.74
OVSeg-L [25]	531M	1.64	29.54	29.04	31.9	14.16	28.64	26.94
SAN-L [50]	437M	0.14	36.18	38.83	30.27	16.95	20.41	30.06
CAT-Seg-L [7]	490M	0.33	39.93	39.85	48.49	26.04	34.06	38.14
CAT-Seg-H [7]	1049M	0.5	37.98	37.74	34.65	29.04	37.76	35.66
models is even larger for the two other domains. Looking at the dataset-specific performance in Fig. 2 and 3, we observe that the performance varies between datasets and models. While SAN-L is the best-performing model on CUB-200 and DRAM, it has significantly lower performance on CWFID or CHASE DB1 compared to CAT-Seg-L. The model achieves scores between  $50\%$  and over  $100\%$  of the performance of supervised state-of-the-art in the general domain. Within the other domains, CAT-Seg-L has a performance gap of more than 25pp for most of the datasets.
The inference time varies between the models and, in particular, between different model architectures with some models requiring more than ten times higher computational effort indicated by higher inference times. In general, we observe the highest inference times for two-stage mask-based approaches, such as ZSSeg and OVSeg, which are between five to twelve times higher than other mask-based approaches (X-Decoder, OpenSeeD, and SAN). The point-based CAT-Seg uses a sliding window approach which requires five passes and therefore results in higher inference times than SAN. Overall, SAN represents the fastest model in our experiments.
Figure 2: mIoU results for large models on a log scale. The datasets are grouped by their domain and sorted by supervised performance.
Figure 3: mIoU results of large models in relative comparison to the supervised mIoU on a log scale. 100 is equal to the supervised mIoU.
5.2 Sensor type evaluation