SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Meng Wei  $^{1,2}$  Xiaoyu Yue  $^{3}$  Wenwei Zhang  $^{1}$  Shu Kong  $^{4,5}$
Xihui Liu  $^{2}$  Jiangmiao Pang  $^{1*}$
$^{1}$ Shanghai AI Laboratory  $^{2}$ The University of Hong Kong
$^{3}$ The University of Sydney  $^{4}$ University of Macau  $^{5}$ Texas A&M University
mengwei.kelly@connect.hku.hk yuexiaoyu002@gmail.com skong@um.edu.mo
xihuiliu@eee.hku.hk {zhangwenwei, pangjiangmiao}@pjlab.org.cn
Figure 1: An illustration of the Generalized Zero-Shot Setting (red arrow), Cross-Dataset Setting (blue arrow) and Few-Shot Setting with examples from Pascal-Part-116 and ADE20K-234.
Segmenting and recognizing diverse object parts is a crucial ability in applications spanning various computer vision and robotic tasks. While significant progress has been made in object-level Open-Vocabulary Semantic Segmentation (OVSS), i.e., segmenting objects with arbitrary text, the corresponding part-level research poses additional challenges. Firstly, part segmentation inherently involves intricate boundaries, while limited annotated data compounds the challenge. Secondly, part segmentation introduces an open granularity challenge due to the diverse and often ambiguous definitions of parts in the open world. Furthermore, the large-scale vision and language models, which play a key role in the open vocabulary setting, struggle to recognize parts as effectively as objects. To comprehensively investigate and tackle these challenges, we propose an Open-Vocabulary Part Segmentation (OV-PARTS) benchmark. OV-PARTS includes refined versions of two publicly available datasets: Pascal-Part-116 and ADE20K-Part-234. And it covers three specific tasks: Generalized Zero-Shot Part Segmentation, Cross-Dataset Part Segmentation, and Few-Shot Part Segmentation, providing insights into analogical reasoning, open granularity and few-shot adapting abilities of models. Moreover, we analyze and adapt two prevailing paradigms of existing object-level OVSS methods for OV-PARTS. Extensive experimental analysis is conducted to inspire future research in leveraging foundational models for OV-PARTS. The code and dataset are available at https://github.com/OpenRobotLab/OV_PARTS.
The ability to identify and reason about object parts is crucial for a wide range of human activities. For instance, when preparing a meal, we rely on specific parts of utensils such as the blade of a knife for slicing and the handle of a spatula for stirring. Hence, developing a vision system capable of part-level object segmentation is crucial and offers substantial benefits across applications in vision and robotics such as image editing [14, 19], object manipulation [27] etc. Despite dedicated efforts in annotating fine-grained parts by previous works [4, 34], the complex nature and diverse granularity of object parts make it hard to create a comprehensive closed category set. Recently, the research on Open Vocabulary Semantic Segmentation (OVSS) [22, 9, 7, 13, 33, 32] extends the image-text alignment ability of large-scale Vision-Language Models (VLM) like CLIP [26] to pixel-level prediction, which shows remarkable performance in open vocabulary object segmentation.
Despite the satisfying performance on object-level OVSS, part-level OVSS raises additional challenges. Parts exhibit complex structures, often with more intricate boundaries and appearance variations than objects. However, the available labeled data for training part segmentation models is significantly limited compared to that of objects. Moreover, in the open-vocabulary setting, there are further challenges to overcome. Firstly, while objects are typically well-defined entities with strict boundaries, the granularity of parts can be flexible, posing the extra open granularity challenge that rarely occurs in object-level OVSS. Secondly, the widely used large-scale VLM are mainly pretrained on natural image-text pairs which are inherently biased to object words. Hence, their ability to recognize object parts can be less proficient. This disparity is reflected in the less discriminative class activation maps of CLIP [26] for parts as opposed to objects, as illustrated in Figure 2 (a).
Considering these challenges, we propose to break down the complex part-level OVSS problem into specific subtasks. One key observation is, common objects often exhibit shared characteristics in terms of parts. For example, furniture like chair, table and sofa often have shared parts such as legs, seats, and backrests. Humans can categorize a new object based on its part descriptions related to a known object. Similarly, we can leverage this analogical reasoning ability to improve data efficiency in part-level OVSS. Hence, we propose a Generalized Zero-Shot Part Segmentation setting, focusing on assessing the transferability of part segmentation from seen objects to related unseen objects. Additionally, we extend the related Cross-Dataset Segmentation setting, commonly used in object-level OVSS, to part-level OVSS. This setting further emphasizes the open granularity challenge, due to the varying part vocabularies and granularity levels across datasets, as shown in Figure 1. Finally, concerning the weaker transferability of large-scale foundation models for the part, we further expect a Few-Shot Part Segmentation setting to enable fast adaptation of the foundation models to part-level OVSS. The entire benchmark, named Open-Vocabulary Part Segmentation (OVPARTS), is further supplemented with carefully cleaned and reorganized versions of Pascal-Part [4] and ADE20K-Part [34], namely Pascal-Part-116 and ADE20K-Part-234.
Additionally, we design strong baselines for OV-PARTS based on two paradigms (i.e., two-stage and one-stage) of existing object-level OVSS methods. The first paradigm [9, 33, 32] designs a two-stage approach that decouples the segmentation and open vocabulary classification abilities. Applying this paradigm to OV-PARTS involves training a class-agnostic part proposal model followed by using CLIP for part region classification. However, it turns out that treating object part as independent classes like objects lead to suboptimal performance. Indeed, even humans struggle to distinguish between "cow's leg" and "sheep's leg" if only the region of their legs is shown. To mitigate this problem, we propose two improvements: (1) An Object Mask Prompt strategy which introduces the object-awareness to the first part proposal stage. (2) A Compositional Prompt Tuning strategy which not only enhances object awareness in the second stage but also shifts CLIP's attention from objects to object parts. This two-stage paradigm offers the advantage of combining class-agnostic part parsing models [15, 25] and various finetune methods to adapt foundation models for classification [38, 37, 2]. However, using a class-agnostic part proposal model has limitations. It is trained on pre-defined parts, which is not applicable to the open granularity scenario. Moreover, the mask proposal model is inclined to overfit the training data, leading to reduced zero-shot generalization ability.
The other line of works [7, 22] follows a one-stage paradigm that trains a unified open-vocabulary segmentation model based on CLIP, eliminating the missing object context problem. As for the open granularity ability, we experimentally find that CATSeg [7] and CLIPSeg [22], which are pretrained on COCO-Stuff[1] and PhraseCut [31] respectively, can already achieve the first-level granular generalization from object to some simple parts as shown in Figure 2 (b). Motivated by this,
Figure 2: (a) Class activation maps of CLIP. The object prompt "Sheep" can activate its area while the part prompt "Head" fails. When using "Sheep's Head" as prompt, the activation is still biased to "Sheep". (b) Without finetuning on part datasets, CATSeg outputs rough part masks of "Bird" and "Sheep" (Left) and CLIPSeg produces finer granular part masks of "Bird" and "Dog" (Right.).
we design some one-stage baselines by finetuning specific modules of CLIPSeg and also investigate parameter-efficient finetuning strategies [12] for OV-PARTS. However, the segmentation ability of one-stage baselines is weaker than the two-stage baselines, mainly limited by the frozen CLIP visual encoder which is pretrained without sufficient segmentation data.
In summary, the two-stage and one-stage baselines exhibit complementary strengths and weaknesses. But there are no currently clear solutions about how to achieve both high-quality segmentation and strong open vocabulary and granularity generalization ability in OV-PARTS. Hence, there remains much to uncover in addressing OV-PARTS, particularly in unlocking the full potential of large foundation models in vision, language and multi-modality, which can be the value of the benchmark.
Open Vocabulary Semantic Segmentation. Open vocabulary semantic segmentation has recently achieved significant progress with the help of large-scale Vision-Language Models (VLM) like CLIP [26], which excel at recognizing objects in images. One line of research [9, 33, 32] combines powerful segmentation models like MaskFormer [5] with CLIP in a two-stage way. In the first stage, MaskFormer [5] produces class-agnostic object mask proposals. In the second stage, CLIP [26] classifies the image regions of these proposals. ODISE [32] further used a pretrained diffusion model to enhance the proposal stage. Another one-stage approach focuses on extending CLIP [26] to pixel-level prediction [7, 22]. They mainly train a pixel decoder on top of the CLIP image encoder. CLIPSeg [22] adds a lightweight transformer-based pixel decoder with a FiLM[10] module to fuse the multi-modality features. CATSeg [7] designs a spatial and class aggregation network with multi-modality guidance features for effective open vocabulary pixel classification.
Part Segmentation. Fine-grained part segmentation has been actively studied in the literature [17, 8, 39, 24, 23, 29, 25, 28]. Most of these methods [17, 8, 39, 24, 23] adopt a supervised closed-set setting. Tang et al. [29] designs a supervised language-driven segmentation model which allows interactive whole-to-part segmentation. Recently, Pan et al. [25] proposes an open-world part segmentation setting that only focuses on the class-agnostic part mask generation which is not language-driven. The concurrent work Sun et al. [28] tackles the open vocabulary part segmentation task with a cross-dataset setting. However, the big performance gap compared to open vocabulary object segmentation hasn't been well studied. Also, their proposed method emphasizes building semantic correspondences based on visual features. But the role of language in the open vocabulary setting as well as the potential of vision language models has not been adequately discussed. Moreover, some existing open-vocabulary object detection models [16, 18, 20] can also detect parts with bounding boxes. Because they are trained on crowd-sourced datasets in which the texts can also include part words. However, pixel-level understanding is more conforming to the nature of part which is ambiguous, multi-granular and has intricate boundaries. For example, with bounding boxes, it'll be hard to identify a little bird's parts accurately.
3 Dataset and Benchmark Details
In this section, we first introduce the two proposed datasets Pascal-Part-116 and ADE20K-Part-234 in section 3.1. Then we elaborate on the three task settings designed for OV-PARTS in section 3.2. The exhaustive list of object part classes, the specific data splits and the distribution of part scales and numbers of Pascal-Part-116 and ADE20K-Part-234 are left to the supplementary material.
Pascal-Part-116. Pascal-Part dataset [4] is an extension of the PASCAL VOC 2010 dataset [11], further annotating objects' part masks. Some categories such as "cow", are annotated with a comprehensive list of parts, while others like "chair", "boat", and "dining table" only provide silhouette annotations. Moreover, the part definition includes directional terms like "left," "right," "front," "back," "upper," and "lower", such as "cow's left front lower leg". However, in an open-vocabulary setting, it's unnecessary to discern between the semantics of labels such as "cow's left front lower leg" and "cow's right back lower leg". They can not only create a bottleneck in part segmentation but also cause overfitting which hinders effective language-driven generalization. Hence, we have manually merged some of the over-segmentation parts to create a more practical version. Our revised Pascal-Part dataset [4] includes a total of 116 object part classes across 17 object classes, which is the most extensive set among various versions of Pascal-Part dataset[30, 21, 3, 24].
ADE20K-Part-234. The ADE20K dataset [34] provides open-ended annotations of 847 objects and  $1000+$  parts, following the WordNet hierarchy. It covers a broad range of scenes, including indoor spaces such as "bedrooms", and outdoor spaces like "streetscapes". However, the part annotations in ADE20K are extremely sparse and incomplete (less than  $15\%$  object instances have part annotations), which poses significant challenges for both training models and evaluating their performance. Despite attempts to reorganize the dataset [23, 29], either the revised versions have not been publicly released or they still contain considerable noise. To get a clean version, we started with the widely used SceneParse150 [35] subset and only keep the objects which have more than one frequently annotated part (over 100 occurrences) and then filter the rare parts (less than 10 occurrences). Moreover, we manually merge some duplicated parts such as "chair arm" and "chair armrest", "table stretcher" and "table h-stretcher" as well as the over-segmentation parts. The resulting subset consists of 44 objects and 234 parts, providing a cleaner dataset for improved analysis and evaluation.
3.2 Benchmark Tasks
There are two primary challenges : (1) The available pixel-level part data is limited. (2) Pretrained features from large-scale VLM exhibits weaker transferability to parts. To evaluate the OV-PARTS models comprehensively, we have designed three task settings: Generalized Zero-Shot Part Segmentation, Cross-Dataset Part Segmentation, and Few-Shot Part Segmentation.
Generalized Zero-Shot Part Segmentation. Considering the limited ability of VLMs to recognize parts, this task aims to assess the model's analogical reasoning ability, which is designed by selecting novel objects that possess related parts to the base objects, rather than being completely irrelevant.
Data Split. To split the object classes in each dataset, we group the object classes into higher-level categories (e.g. Animals, Vehicles) based on their shared attributes. Within each hyper-category, we split the objects into base and novel classes (74/42 for Pascal-Part-116 and 176/58 for ADE20K-Part-234). The unseen objects in the training set are set to the background. In this way, a novel object part class may be novel at the object level (e.g., "dog's head" is a novel class while "cat's head" is a base class) or both at the object level and the part level (e.g., "bird's beak"). The complete base and novel class set can be found in the supplementary material.
Evaluation Protocol. Following previous OVSS methods [33, 9], we first calculate the mean class-wise Intersection over Union (mIoU) on both base and novel classes. Then, to provide a balanced assessment of the model's performance across both base and novel classes in this setting, we use harmonic mean IoU (hIoU) as the primary metric.
Cross-Dataset Part Segmentation. In object-level OVSS, the cross-dataset setting evaluates the model's ability to handle variations in data distribution and novel object vocabulary. While in OVPARTS, considering diverse part definitions, the model further needs to generalize between different annotation granularity levels in addition to different vocabularies. For example, the part set of "car" is
(c)
Figure 3: The overall frameworks of the representative two-stage (ZSseg+) and one-stage (CLIPSeg) baselines. (a) is the modified proposal generation stage of ZSseg. (b) shows the design of Compositional Prompt Tuning based on CoOp. (c) is the CLIPSeg model architecture which also indicates the modules that can be finetuned. CLIP is fixed in both frameworks.
[“wheel”, “headlight”, “license plate”, “mirror”, “door”, “window”, “side”, “front”, “back”, “roof”] in Pascal-Part-116 while is [“bumper”, “door”, “headlight”, “hood”, “license plate”, “logo”, “mirror”, “wheel”, “window”, “wiper”] in ADE20K-Part-234 which has a finer granularity.
Data Split. Overall, ADE20K-234 covers a more diverse set of objects than Pascal-Part-116. We train models on the full training set of ADE20K-234 and evaluate them on Pascal-Part-116's testing set. Evaluation Protocol. We reported mIoU on both the source dataset (ADE20K-Part-234) and target dataset (Pascal-Part-116).
Few-Shot Part Segmentation. Indeed, there's still a big performance gap between OV-PARTS models compared to the object-level OVSS models, mainly due to the inherent task difficulty and data limitation. Inspired by recent advances in foundation models, we design this few-shot part segmentation setting mainly to explore the strategies to adapt large-scale foundation models effectively for OV-PARTS and investigate the feasibility of utilizing object-level data to achieve parameter-efficient training and data efficiency.
Data Split. We sample 16 images for each object class. Since there may be multiple objects in an image or each object may not have exhaustive part annotations, the exact number of sampled shots for each object part class may be slightly over or below 16.
Evaluation Protocol. We use mIoU as the main metric.
Existing object-level Open Vocabulary Semantic Segmentation (OVSS) methods can be categorized into the two-stage and one-stage paradigms. These paradigms differ in the way to apply the CLIP model to the segmentation process. In this section, we will give a brief overview of the framework for each paradigm. We then introduce our modifications with insights into the baselines.
4.1 Two-Stage Baseline
For the two-stage framework (Figure 3 (a)), we select ZSseg [33] as the baseline, which includes a proposal generation stage and a zero-shot prediction stage. The proposal generation stage mainly includes a MaskFormer [5]. Given an input image  $\mathbf{I}$ , a backbone network extracts the image features  $\mathbf{Z}$ . Then a pixel decoder is used to get the mask features  $\mathbf{F}$  and a query-based transformer decoder is used to get the query features  $\mathbf{S}$  with learnable queries  $\mathbf{Q}$ . Finally,  $\mathbf{S}$  is fed into a mask predictor to get the class-agnostic masks  $\mathbf{M}$  by computing the dot product of  $S$  and  $F$  and a fixed classifier
(initialized from CLIP text embeddings) to compute the classification score  $C_P$ :
$$
S = \text {T r a n s f o r m e r D e c o d e r} (Q; Z) \tag {1}
$$
$$
M = \operatorname {M a s k P r e d i c t o r} (S; F), C _ {P} = \text {C l a s s i f i e r} (S) \tag {2}
$$
In the zero-shot prediction stage, the mask proposals  $\mathbf{M}$  are used to crop the original image  $\mathbf{I}$ . The resulting masked image is then fed into the CLIP model for zero-shot class prediction. The final classification scores are obtained through an ensemble of both stages:
$$
C _ {C} = \operatorname {C L I P} (I; M), C = \operatorname {E n s e m b l e} \left(C _ {P}; C _ {C}\right) \tag {3}
$$
Indeed, directly transferring the zsseg method from object-level data to part-level data has limitations and can lead to suboptimal performance. Unlike objects, which exist independently, parts are always contextually dependent on their respective objects. Classifying part regions without considering their object context is challenging. To address this, we propose two approaches to incorporate object context into both stages:
Object Mask Prompt. In the first stage, we propose to incorporate object masks as a mask prompt which can be obtained in a more cost-effective manner. This decoupling of object masks from the part segmentation brings object awareness to the model and helps resolve the difficulty. Furthermore, it enables interactive segmentation applications such as image editing.
In detail, as shown in Figure 3 (a), we modify the transformer decoder by replacing the cross-attention layers with mask-guided cross-attention layers [6]. This ensures that the cross-attention mechanism is computed only within the regions corresponding to objects, resulting in more accurate and context-aware part proposals. Equation (1) thus becomes:
$$
S = \text {M a s k e d T r a n s f o r m e r D e c o d e r} (Q; Z; M _ {o}) \tag {4}
$$
where  $M_{o}$  indicates the object mask. We further find that the proposals of parts are very noisy compared to objects. Hence, we design a mask-denoising technique during testing. Instead of directly using  $\mathbf{M}$  for the second stage, we aggregate  $\mathbf{M} \in \mathbb{R}^{q \times h \times w}$  and  $C_{P} \in \mathbb{R}^{q \times c}$  to get a refined set of part mask proposals:
$$
C _ {C} = \operatorname {C L I P} (I; \operatorname {B i n a r y} \left(\sum_ {q} \left(C _ {P} \times M\right) \in \mathbb {R} ^ {c \times h \times w}\right)) \tag {5}
$$
Here,  $q, c, h$  and  $w$  represent the number of learnable queries, the number of classes, the image height and the image width respectively. In the Binary operation, we process the semantic mask into  $c$  binary masks.
Compositional Prompt Tuning. In the second stage, to ensure that CLIP is not confused when presented with only partial regions of objects, we adapt CLIP's vision encoder following MaskCLIP [36] to obtain dense image embedding instead of the global image embedding. Then we use mask pooling to get CLIP visual features for both the object mask and the part mask. Since we observed that objects tend to capture most of CLIP's attention, overshadowing the parts, we design a compositional prompt tuning method based on CoOp [38]. We add object tokens and part tokens for prompt tuning with a learnable fusion weight. The modified zero-shot prediction stage is shown in Figure 3 (b). We find that this prompt tuning strategy is effective with minimal data and training cost.
4.2 One-Stage Baseline
In the one-stage framework, we employ CLIPSeg [22] as our baseline. The overall framework of CLIPSeg is shown in Figure 3 (c). CLIPSeg adds a parameter-efficient three-layered transformer decoder to the original CLIP model for segmentation. It integrates visual features from the final layer of the visual encoder and text features of all object part prompts from the text encoder through the FiLM module, forming cross-modal input token embeddings for the decoder. Furthermore, the features extracted from the 3rd, 6th, and 9th layers of CLIP's visual encoder are projected and added to the intermediate features of the corresponding decoder layers. It's worth noting that, the visual features extracted from the frozen CLIP visual encoder first pass through the added visual adapter, which consists of a two-layered MLP, before reaching the decoder. CLIPSeg is originally designed to generate binary segmentation maps conditioned on either a visual prompt or a text prompt. So we replace the original binary segmentation head with a multi-class one and modify its loss for multi-class segmentation using only text prompts.
By directly applying CLIP to the pixel-level prediction, we find that CLIPSeg, trained on Phrase-Cut [31] with object phrases as text input, shows amazing zero-shot generalization to parts. From this finding and considering the data sparsity issue in part segmentation, our focus shifts to exploring parameter-efficient finetuning for the one-stage baseline. Finetuning the entire CLIP model has proven to be harmful to its generalization ability [36]. Hence we conduct experiments to finetune different combinations of the specific modules of CLIPSeg, including the text embedding layer in CLIP, the pixel decoder and the extra light-weight CLIP-Adapter modules [12] (Vision-Adapter and Text-Adaper are simple MLPs) as shown in Figure 3 (c).