OV-PARTS: Towards Open-Vocabulary Part Segmentation
Authors: Meng Wei
Xiaoyu Yue
Wenwei Zhang
Shu Kong
Xihui Liu
Jiangmiao Pang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Open vocabulary
semantic segmentation
part segmentation
foundation model

Abstract: Segmenting and recognizing diverse object parts is a crucial ability in applications spanning various computer vision and robotic tasks. While significant progress has been made in object-level Open-Vocabulary Semantic Segmentation (OVSS), i.e., segmenting objects with arbitrary text, the corresponding part-level research poses additional challenges. Firstly, part segmentation inherently involves intricate boundaries, while limited annotated data compounds the challenge. Secondly, part segmentation introduces an open granularity challenge due to the diverse and often ambiguous definitions of parts in the open world. Furthermore, the large-scale vision and language models, which play a key role in the open vocabulary setting, struggle to recognize parts as effectively as objects. To comprehensively investigate and tackle these challenges, we propose an Open-Vocabulary Part Segmentation (OV-PARTS) benchmark. OV-PARTS includes refined versions of two publicly available datasets: Pascal-Part-116 and ADE20K-Part-234. And it covers three specific tasks: Generalized Zero-Shot Part Segmentation, Cross-Dataset Part Segmentation, and Few-Shot Part Segmentation, providing insights into analogical reasoning, open granularity and few-shot adapting abilities of models. Moreover, we analyze and adapt two prevailing paradigms of existing object-level OVSS methods for OV-PARTS. Extensive experimental analysis is conducted to inspire future research in leveraging foundational models for OV-PARTS. The code and dataset are available at https://github.com/kellyiss/OV_PARTS.

OV-PARTS: Towards Open-Vocabulan Part Segmentation
Meng Wei 1.2
Xiaoyu Yue
Wenwei Zhang
Shu Kong 4.5
Xihui Liu
Jiangmiao Pang
SSanghai Al Laboratory
"The University of Hong Kong
The University of Sydney *University of Macau TTexas A&M University
mengwei. kellyyconnect. hku.hk hk yuexiaoyu0028gmail.cc com skongöum edu xihuiliu@eee hku hk
(zhangwenwei, panggiangmiao Sp org.
Trainin
Test
Zero Shot
Zero Shat
SEBN
SERN
PASCAL
VLM
Cross Dataeet
Adapter
JASFNN
Finetuning
Few-Shot
Figure 1: An illustration of the Generalized Zero-Shot Setting (red arrow), Cross-Dataset Setting (blue arrow) and Few-Shot Setting with examples from Pascal-Part-I1 and ADE20K-234.
Abstract
Segmenting and recognizing diverse object parts is crucial ability in applications spanning various computer vision and robotic tasks. While significant progress has been made in object-level Open- Vocabulary Semantic Segmentation (OVSS), i.e. segmenting objects with arbitrary text, the corresponding part-level research poses additional challenges. Firstly, part segmentation inherently involves intricate boundaries, while limited annotated data compounds the challenge. Secondly, part segmentation introduces an open granularity challenge due to the diverse and often ambiguous definitions of parts the open world. Furthermore, the large-scale vision and language models, which play a key role in the open vocabulary setting, struggle to recognize parts as effectively as objects. To comprehensively investigate and tackle these challenges, propose Open- Vocabulary Part Segmentation (OV-PARTS) benchmark. OV-PARTS includes refined versions of two publicly available datasets: Pascal-Part- 116 and ADE20K-Part-25 And it covers three specific tasks: Generalized Zero-Shot Part Segmentation, Fross-Datase Part Segmentation, and Few-Shot Part Segmentation. providing insights into analogical reasoning, open granularity and few-shot adapting abilities models. Moreover, we analyze and adapt prevailing paradigms of existing object-level OVSS methods for OV-PARTS. Extensive experimental analysis conducted to inspire future research in leveraging foundational models for OV-PARTS. The code and PARTS
dataset are available at wt/https:/// http///gtttt...//// cttpb.cccmmpeeeeeee
*Corresponding Author.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Introduction
The ability to identify and reason about object parts is crucial for a wide range of human activities. For instance, when preparing a meal, we rely specifie parts of utensils such the blade of knife for slicing and the handle of a spatula for stirring. Hence, developing a vision system capable of part-level object segmentation is crucial and offers substantial benefits across applications in vision and robotics such as image editing 14, 19, object manipulation 277 etc. Despite dedicated efforts in annotating fine-grained parts by previous works 4 (34, the complex nature and diverse granularity of object parts make it hard to create a comprehensive closed category set. Recently,
the research on Open Vocabulary Semantic Segmentation (OVSS) [22 9 7
[3]
33 32 extends
the image-text alignment ability of large-scale Dision-Language Models (VLM) like CLIP (26 to
pixel-Ievel prediction, which shows remarkable performance in open vocabulary object segmentation.
Despite the satisfying performance on object-level OVSS, part-level OVSS raises additional chal- lenges. Parts exhibit complex structures, often with more intricate boundaries and appearance variations than objects. However, the available labeled data for training part segmentation models is significantly limited compared to that of objects. Moreover, in the open-vocabulary setting, there further challenges to overcome. Firstly, while objects are typically well-defined entities with strict boundaries, the granularity of parts can be flexible, posing the extra open granularity challenge that rarely occurs in object-level OVSS. Secondly, the widely used large-scale VLM are mainly pretrained on natural image-text pairs which are inherently biased to object words. Hence, their ability to recognize object parts can be less proficient. This disparity is reflected in the less discriminative class activation maps of CLIP (66 for parts as opposed to objects, as illustrated in Figure 2 (a).
Considering these challenges, we propose to break down the complex part-level OVSS problem into specific subtasks. One key observation is, common objects often exhibit shared characteristics in terms of parts. For example, furniture like chair, table and sofa often have shared parts such as legs, seats, and backrests. Humans can categorize new object based its part descriptions related to a known object. Similarly, we can leverage this analogical reasoning ability to improve data efficiency in part-level OVSS. Hence, we propose Generalized Zero-Shof Part Segmentation setting, focusing on assessing the transfcrability of part segmentation from seen objects to related unseen objects. Additionally, we extend the related Cross-Datased Segmentation setting, commonly used in object-level OVSS, to part-level OVSS. This setting further emphasizes the open granularity challenge, due the varying part vocabularies and granularity levels across datasets, as shown in Figure 1 Finally, concerning the weaker transferability of large-scale foundation models for the part, we further expect a Sew-Shot//e Part Segmentation setting to enable fast adaptation of the foundation models to part-level OVSS. The entire benchmark, named Open- Vocabulary Part Segmentation (OV- PARTS), is further supplemented with carefully cleaned and reorganized versions of Pascal-Part [ and ADE20K-Parr [34] namely Pascal-Part- 16 and DD220--Part-244
Additionally, we design strong baselines for OV-PARTS based on two paradigms (i.e., two-stage and one-stage) of existing object-level OVSS methods. The first paradigm [3] 32] designs a two-stage approach that decouples the segmentation and open vocabulary classification abilities. Applying this paradigm to OV-PARTS involves training a class-agnostic part proposal model followed by using CLIP for part region classification. However, it turns out that treating object part as independent classes like objects lead to suboptimal performance. Indeed, even humans struggle to distinguish between "cow's leg" and "sheep's 's leg" if only the region of their legs is shown. To mitigate this problem, we propose two improvements: (1) An Object Mask Prompt strategy which introduces the object-awarenes. to the first part proposal stage. (2) A Compositional Prompt Tuning strategy which not only enhances object awareness in the second stage but also shifts CLIPP's attention from objects to object parts. This two-stage paradigm offers the advantage of combining class- -aasssiio part parsing models 15 25 and various finetune methods to adapt foundation models for classification [38 37 2 However, using a class-agnostic part proposal model has limitations. It is trained on pre-defined parts, which is not applicable to the open granularity scenario. Moreover, the mask proposal model is inclined to overfit the training data, leading to reduced zero-shot generalization ability.
The other line of works 7 22 follows one-stage paradigm that trains a unified opea-vocabulary segmentation model based on CLIP, eliminating the missing object context problem. As for the open granularity ability, we experimentally find that CATSeg 7 and CLIPSSeg (22), which are pretrained on COCO-Stuff[] and PhraseCut (3I] respectively, can already achieve the first-level granular generalization from object to some simple parts shown in Figure (b). Motivated by this,

---

"Sheep's head'
"Head"
(b)
Figure 2: (a) Class activation maps of CLIP. The object prompt "Sheep" can activate its area while the part prompt "Head" fails. When using "Sheep's 's Head" as prompt, the activation still biased to "Sheep". (b) Without finetuning part datasets, CATSeg outputs rough part masks of "Bird" and "Sheep" (Left) and CLIPSeg produces finer granular part masks of "Bird" and "Dog" (Right.).
we design some one-stage baselines by finetuning specific modules of CL.PSeg and also investigate parameter-efficint finetuning strategies [12] for OV-PARTS. However, the segmentation ability of one-stage baselines is weaker than the two-stage baselines, mainly limited by the frozen CLIP visual encoder which is pretrained without sufficient segmentation data.
In summary, the two-stage and one-stage baselines exhibit complementary strengths and weaknesses. But there are no currently clear solutions about how to achieve both high- quah--uultty segmentation and strong open vocabulary and granularity generalization ability in OV-PARTS. Hence, there remains much to uncover addressing OV-PARTS, particularly in unlocking the full potential of large foundation models in vision, language and multi-modality, which can be the value of the benchmark.
2 Related Work
Open Vocabulary Semantic Segmentation. Open vocabulary semantic segmentation has recently achieved significant progress with the help of large-scale Vision-Language Models (VLM) like
CLIP 266, which excel recognizing objects in images. One line of research 9 33 32
combines
powerful segmentation models like MaskFormer (5] with CLIP in a two-stage way. In the first stage, MaskFormer (5) produces class-agnostic object mask proposals. In the second stage, CLIP (26) classifies the image regions of these proposals. ODISE [32 further used pretrained diffusion model enhance the proposal stage. Another one-stage approach focuses on extending CLIP (26) to pixel-level prediction 77 (22). They mainly train a pixel decoder on top of the CLIP image encoder. CLIPSeg (22) adds a lightweight transformer-based pixel decoder with a FiLM module to fuse the multi-modality features. CATSeg 77] designs a spatial and class aggregation network with multi-modality guidance features for effective open vocabulary pixel classification.
Part Segmentation. Fine-graa tene-grained part segmentation has been actively studied the literature 17 g
39 24
23
29
23 adopt a supervised closed-set setting. 29 designs a supervised language-driven segmentation model which allows interactive
25 28). Most of these methods [17 g 39
24
Tang et al. whole-to-part segmentation. Recently, Pan al.
25 proposes open-world part segmentation setting that only focuses on the class-agnostic part mask generation which is not language-driven The concurrent work Sun et al. (28) tackles the open vocabulary part segmentation task with cross- dataset setting. However, the big performance gap compared to open vocabulary object segmentation hasn't been well studied. Also, their proposed method emphasizes building semantic correspondences based on visual features. But the role of language in the open vocabulary setting well as the potential of vision language models has not been adequately discussed. Moreover, some existing open- vecabulcyyulary object detection models [16] I8 20 can also detect parts with bounding boxes. Because they are trained on crowd-sourced datasets in which the texts can also include part words. However, pixel-level understanding is more conforming to the nature of part which is ambiguous, multi-granular and has intricate boundaries. For example, with bounding boxes, it'll be hard to identify a little bird's 's parts accurately.
3

---

3 Dataset and Benchmark Details
In this section, we first introduce the two proposed datasets Pascal-Part-116 and DODE2OK-Part-22 in section [.] Then we elaborate on the three task settings designed for OV-PARTS in section 3.2 The exhaustive list of object part classes, the specific data splits and the distribution of part scales and numbers of Pascal-Parre 16 and ADE20K-PPat Part-234 are left the supplementary material.
3.1 Datasets
Pascal-Part-116. Pascal-Part dataset [4] is an extension of the PASCAL VOC 2010 dataset [T] further annotating objects' part masks. Some categories such as "cow", are annotated with a comprehensive list of parts, while others like "chair", "boat", and "dining table" only provide silhouette annotations. Moreover, the part definition includes directional terms like "left," "right," "front," "back," "upper," and "lower", such as "cow's left front lower leg". However, in an open- vocabulary setting, it's unnecessary to discern between the semantics of labels such as "cow's left front lower leg" and "cow's 's right back lower leg". They can not only create bottleneck in part segmentation but also cause overfitting which hinders effective language-driven generalization. Hence, we have manually merged some of the over-segmentation parts to create more practical version. Our revised Pascal-Part dataset 4 includes a total of 116 object part classes across 17 object classes, which is the most extensive set among various versions of Pascal-Part dataset (30) 21 3, 24).
ADE20K-Part-234 The ADE20K dataset [34] provides open-ended annotations of 847 objects and 1000+ parts, following the WordNet hierarchy. It covers a broad range of scenes, including indoor spaces such as "bedrooms", and outdoor spaces like "streetscapes". However, the part annotations in ADE20K are extremely sparse and incomplete (less than 15% object instances have part annotations), which poses sigmificant challenges for both training models and evaluating their performance. Despite attempts to reorganize the dataset [3 29), either the revised versions have not been publicly released or they still contain considerable noise. To get clean version, we started with the widely used SceneParse 50 35 subset and only keep the objects which have more than one frequently annotated part (over 100 occurrences) and then filter the rare parts (less than 10 occurrences). Moreover, we manually merge some duplicated parts such as "chair arm" and "chair armrest", "table stretcher" and "table h-stretcher" as well as the over-segmentation parts. The resulting subset consists of 44 objects and 234 parts, providing cleaner dataset for improved analysis and evaluation.
3.2 Benchmark Tasks
There are two primary challenges: (1) The available pixel-level part data limited. (2) Pretrained features from large-scale VLM exhibits weaker transferability to parts. To evaluate the OV-PARTS models comprehensively, we have designed three task settings: Generalized Zero-Shot Part Segmen- tation, Cross- Part Segmentation, and Few-Shot Part Segmentation.
Generalized Zero-Shot Part Segmentation. Considering the limited ability of VLMs to recognize parts, this task aims to assess the model's 's analogical reasoning ability, which is designed by selecting novel objects that possess related parts to the base objects, rather than being completely irrelevant. Data Splir. To split the object classes in each dataset, we group the object classes into higher-level categories (e.g. Animals, Vehicles) based on their shared attributes. Within each hyper-category. we split the objects into base and novel classes (74/42 for Pascal-Part-11 16 and 176/58 for ADE20K-Part 234). The unseen objects in the training set are set to the background. In this way, a novel object part class may be novel at the object level (e.g.. "dog's 's head" is a novel class while "cat's 's head" is base class) or both at the object level and the part level (e.g., "bird's beak"). The complete base and novel class set can be found in the supplementary material.
Evaluation Protocol. Following previous OVSS methods B3 9], we first calculate the mean class- wise Intersection over Union (mloU) both base and novel classes. Then, to provide balanced assessment of the model's 's performance across both base and novel classes in this setting, we use harmonic mean IoU (hloU) as the primary metric.
object-level OVSS, the cross-dataset setting evaluates the model's 's ability to handle variations in data distribution and novel object vocabulary. While in OV- PARTS, considering diverse part definitions, the model further needs to generalize between different annotation granularity levels in addition to different vocabularies. For example, the part set of "car" is
Datsset Part Segmentation.

---

Proposal Generation
Pixel Decoder
Backbone
Mask Predictor
Zero-Shot
Masked Transformer Decoder
Prediction
Classifier
M
Image Encoder
Image
Adapter
Vision
Encoder
FILM
Pixel Decoder
Weighted Sum
Text Adapter
[OBJECT]
[PART]
Encoder
Text Prompt
Text Einooder
Learnable Text Prompt
(b)
(c)
Figure 3: The overall frameworks the representative two-stage (ZSseg+) and one-stage (CLIPSeg) baselines. (a) is the modified proposal generation stage of ZSseg. (b) shows the design of Composi- tional Prompt Tuning based on CoOp. (c) is the CLIPSeg model architecture which also indicates the modules that can be finetuned. CLIP is fixed in both frameworks.
["wheel", "headlight", "license plate", "mirror", "door", "window", "side", "front", "back", "roof") in Pascal-Part-1 16 while is ["bumper", "door", "headlight", "hood", "license plate", "logo", "mirror", "wheel", "window", "wiper"] in ADE20K- -ar20k-- which has a finer granularity. Data Split. Overall, ADE20K-253 covers a more diverse set of objects than Passal-Part-116. We train models on the full training set of ADE20K-234 and evaluate them on Pascal-Part-I 16's testing set. Evaluation Protocol. We reported mloU on both the source dataset (ADE20K-Part-234) and target dataset (Pascal-Part-116)
Few-Shot Part Segmentation. Indeed, there's still big performance gap between OV-PARTS models compared to the object-level OVSS models, mainly due to the inherent task difficulty and data limitation. Inspired by recent advances in foundation models, design this few-shot part segmentation setting mainly to explore the strategies to adapt large-scale foundation models effectively for OV-PARTS and investigate the feasibility of utilizing object-level data achieve parameter- efficient training and data efficiency.
Data Split. We sample 16 images for each object class. Since there may be multiple objects in an image or each object may not have exhaustive part annotations, the exact number of sampled shots for each object part class may be slightly over below 16. Evaluation Protocol. We use mloU as the main metric.
Methodology
Existing object-level Open Vocabulary Semantie Segmentation (OVSS) methods can be categorized into the two-stage and one-stage paradigms. These paradigms differ in the way to apply the CLIP model to the segmentation process. In this section, will give brief overview of the framework for each paradigm. We then introduce our modifications with insights into the baselines.
4.1 Two-Stage Baseline
For the two-stage framework (Figure 3 (a)). we select ZSseg [33 as the baseline, which includes proposal generation stage and zero-shot prediction stage. The proposal generation stage mainly includes MaskFormer (5). Given an input image I, a backbone network extracts the image features Z. Then pixel decoder is used to get the mask features F and query-based transformer decoder used get the query features with learnable queries Q. Finally, S is fed into mask predictor to get the class- class-agnostic masks M by computing the dot product of S and F and fixed classifier
(a)).
5

---

(initialized from CLIP text embeddings) to compute the classification score Op:
S frrnnfforreeeeedderree Z)
(1)
M = MasM Predictor(S; (S; F); Cp = Classifieris
(2)
In the zero-shot prediction stage, the mask proposals M are used to crop the original image I. The resulting masked image is then fed into the CLIP model for zero-shot class prediction. The final classification scores are obtained through ensemble of both stages:
Ce = CLIP(I C = Ensemble(Cp;CCa Cc)
(3)
Indeed, directly transferring the zsseg method from object-level data to part-level data has limitations and can lead suboptimal performance. Unlike objects, which exist independently, parts are always contextually dependent on their respective objects. Classifying part regions without considering their object context is challenging. To address this, we propose two approaches to incorporate object context into both stages:
Object Mask Prompt. In the first stage, we propose incorporate object masks as a mask prompt which can be obtained in a more cost-effective manner. This decoupling of object masks from the part segmentation brings object awareness to the model and helps resolve the difficulty. Furthermore, it enables interactive segmentation applications such as image editing.
In detail, as shown in Figure (a), we modify the transformer decoder by replacing the cross- attention layers with mask-guided cross-attention layers (6. This ensures that the cross-attention mechanism is computed only within the regions corresponding objects, resulting in more accurate and context-aware part proposals. Equation ( thus becomes:
S MaskedT rans ormer Decoder(Q: Z; M.)
where Mo indicates the object mask. We further find that the proposals of parts are very noisy compared to objects. Hence, we design mask-denoising technique during testing. Instead of directly using M for the second stage, we aggregate M E Roxhxx and Cp E Raxe to get a refined set of part mask proposals:
CCIPLI; Binary(E, (Cp M) E Rox)x"))
(5)
Here, q, c, h and w represent the number of learnable queries, the number of classes, the image height and the image width respectively. In the Binary operation, we process the semantic mask into c binary masks.
Compositional Prompt Tuning. In the second stage, to ensure that CLIP is not confused when pre- sented with only partial regions of objects, we adapt CLIP's vision encoder following MaskCLIP (36) obtain dense image embedding instead of the global image embedding. Then we use mask pooling to get CLIP visual features for both the object mask and the part mask. Since we observed that objects tend to capture most of CLIP's attention, overshadowing the parts, we design a compositional prompt tuning method based on CoOp [8 We add object tokens and part tokens for prompt tuning with a learnable fusion weight. The modified zero-shot prediction stage shown in (b). We find that this prompt tuning strategy is effective with minimal data and training cost.
4.2 One-Stage Baseline
In the one-stage framework, employ CLIPSeg 22) our baseline. The overall framework of CLiPSeg is shown in Figure 3 (c). CLiiPeeg adds a parameter-efficien three-layered transformer decoder to the original CLIP model for segmentation. layer of the visual encoder and text features of all object part prompts from the text encoder through the FiLM module, forming cross-modal input token embeddings for the decoder. Furthermore, the features extracted from the 3rd, 6th, and 9th layers of CLIP's visual encoder are projected and added to the intermediate features of the corresponding decoder layers. It's worth noting that, the visual features extracted from the frozen CLIP visual encoder first pass through the added visual adapter, which consists two-layered MLP, before reaching the decoder. CLIPSeg originally designed to generate binary segmentation maps conditioned on either a visual prompt or a text prompt. So we replace the original binary segmentation head with a multi-class one and modify its loss for multi-class segmentation using only text prompts.
integrates visual features from the final

---

By directly applying CLIP to the pixel-level prediction, we find that CLIPSeg, trained on Phrase- Cut [3T] with object phrases as text imput, shows amazing zero-shot generalization to parts. From this finding and considering the data sparsity issue in part segmentation, our focus shifts to exploring parameter-efficien finetuning for the one-stage baseline. Finetuning the entire CLIP model has proven to be harmful to its generalization ability [36]. Hence we conduct experiments to finetune different combinations of the specific modules CLIPSeg. including the text embedding layer in CLIP, the pixel decoder and the extra light-weight CLIP-Adapter modules [[2] (Vision-Adapp and Texi-Adaper are simple MLPs) as shown in Figure 3 (c).
5 Experiments
5.1 Experimental Setup
Evaluation Setup for Three Task Settings. Since the state-of-the-arr object-level OVSS models are powerful, we design two evaluation settings to investigate the primary bottlenecks: (1) Oracle-Ohj Setting: The ground-truth mask and class of objects are assumed be known, (2) Pred-Obj Setting: The ground-truth mask and class of objects are not provided. Implementation Details. The details of baselines and training are left to the supplementary material.
5.2 Results in Generalized Zero-Shot Part Segmentation
Settings. We reported results of both two-stage and one-stage baselines on Pascal-Part-I and ADE20K-Part-2 datasets as shown in Table [ and Table respectively. Firstly, to provide comprehensive comparison, we reported the results of the supervised baseline: MaskFormer (5] with ResNet-50 backbone on the seen and unseen classes. Then for the two-stage baselines, we reported results the original ZSseg [33] and our modified ZSseg (ZSseg+) with different finetuning methods for the second stage: Compositional Prompt Tuning based on CoOp (38] and CoCoOp 37], abbreviated to CPT-CoOp and CPT-CoCoOPp For the one-stage baselines, we examined two state-of- the-art methods: CAT-Seg 7 and CLIP-Seg (22). We used the pretrained model of CATSeg trained on COCO-Stufi and CLIPSeg trained on PhraseCut. Then we reported their results without any finetuning, as well as the results when adopting various finetuning strategies. For the Pred-Obj Setting evaluation, we use ZSseg's 's pretrained object model on COCO-Stuff in the two-stage baselines while don't t adopt separate object models in one-stage baselines.
Two-Stage Baselines. We evaluated the effects of the Object Mask Prompt, Mask Denoise, and CPTCoOp in ZSseg+ on Pascal-Parts 16 as shown in Table We gradually add each approach to ZSseg to measure individual effect under the Oracle-Obj Setting. The results indicate that the
Table 1: Zero-shot performance of the two-stage and one-stage baselines on Pascal-Part-11
Model
Backbone
Finetuning
Oracle-Obj
Pred-Obj
Seen
Unseen Harmonic
Unseen Harmonic
Super-Su Baseline
MaskFormer ResNet-50
55.28
52.14
53.07
47.82
Two-Stage Baselines
ZSseg
ResNet-50
X
49.35
12.57
20.04
40.80
12.07
18.63
ZSseg+
ResNet-50
CPTCoOp
55.33
19.17
28.48
54.23
17.10
26.00
ZSseg+
5e
CPTCoCoOp
54.43
19.04
28.21
53.31
16.08
24.71
7.Sseg+
ResNet-101c
CPTCoOp
57.88
21.93
31.81
56.87
20.29
29.91
One-Stage Baselines
CATSeg
ResNet-101 101
14.89
10.29
12.17
13.65
7.73
9.87
ViT-B/16
CATSeg
ResNet-101
B+D
43.97
26.11
32.76
41.65
26.08
32.07
VIT-B/16
CLIPSeg CLIPSSe 22
VIT-B/16
X
22.33
19.73
20.95
14.32
10.52
12.13
ViT-B/16
VA+L+14
48.68
27.37
35.04
44.57
27.79
34.24

---

Table 2: Zero-shot performance of the two-stage and stage baselines on ADE20K-PPart-2
Model
Backbone
Finetuning
Oracle-Obj
Pred-Obj
Seen
Unseen Harmonic
Seen
Unseen Harmonic
Fully-Supervised Baseline
MaskFormer [3] ResNet-50
46.25
47.86
35.52
16.56
Two-Stage Baselines
ZSseg+
Re
CPTCoOp
43.19
27.84
33.85
21.30
5.60
8.87
ZSseg+
ResNet-50
CPTCoCoOp
39.67
25.15
30.78
19.52
2.98
5.17
ZSseg+
ResNet-100
CPTCoOp
43,41
25.70
32.28
21.42
3.33
5.76
One-Stage Baselines
CATSeg 7
ResNet-101
X
11.49
8.56
9.81
6.30
3.79
4.73
& ViT-B/16
CATSeg y
ResNet-101
B+D
31.40
25.77
28.31
20.23
8.27
11.74
& ViT-B/16
CLIPSeg 22
VIT-B/16
X
15.27
18.01
16.53
5.00
3.36
4.02
CLiPSeg 22
ViT-B/16
VA+L+F4D
38.96
29.65
33.67
24.8
6.24
9.98
Table 3: Ablations on proposed strategies in ZSseg+ by adding each module to ZSseg on Pascal-Part-116.
Cross- Performance. Models are trained on the source dataset ADE20K-Part-22 and tested on the tar- get dataset Pascal-Part-I
Table 4:
Oracle-Obj Unseen Harmonic
Model
Soen
Source Oracle Pred
Target Oracle Pred
Model
ZSseg 3
49.35
12.57
20.04
+Obj Mask Prompt
48.00
13.89
21.54
CATSeg
27.95 17.22
16.00
14.72
+Mask Denoise
48.00
16.84
24.93
VA+L+F
35.01
21.74
16.18
11.70
+CPTCoOp
55.33
19.17
28.48
VA+L+F+D
37.76
21.87
19.69
13.88
Object Mask Prompt improves the performance on unseen classes (+1.32%), and adding Mask Denoise further improves it (+2.95%). However, there is a slight performance drop (-1.35%) on the seen classes. Morcover, finetuning the CLIP embeddings with CPTCoOp leads to significant performance gains both the (+7.33%) and unseen (+2.33%) classes. Notably, the finetuning with CPTCoOp requires only 500 iterations and less than 128 samples per class.
One-Stage baselines. Both CATSeg and CLIPSeg employ frozen CLIP visual encoder to extract image features. But CATSeg further uses an extra learnable visual backbone to guide the pixel decoder. In Table and Table [ we can see that the CATSeg and CLIPSeg models, pretrained on object datasets, already show impressive results. Notably, CLIPSeg even outperforms ZSseg+ with ResNet-50 backbone the unseen classes. The comparison between CLIPSeg and CATSeg shows that OVSS models trained with phrases of objects exhibit stronger generalization ability to parts than sole objects. Hence, explore alternative finetuning strategies rather than training from scratch. We finetune three different components: language embedding layer in text encoder (L), FiLM (F) and Decoder (D) of CLIPSeg and two added lightweight modules: CLIP-Adaphe the visual encoder (VA) and text encoder (TA). As shown in Table 5 we draw three key conclusions: (1) Multi-modalit finetuning is better than single-modale
[12 to
(VA+L+FiLM surpasses VA+FiLM and L+FiLM).; (2) Finetuning language embedding performs better than the text adapter (VA+L+F surpasses VA+TA+F); (3) Parameter-efffcient finetuning can effectively transfer the knowledge from large-scale foundation models and object-level datasets to part parsing. We can see that VA+L+F even outperforms finetuning the entire pixel decoder on the unseen classes.
The qualitative results, which present a comparison among ZSseg+, CATSeg and CLIPSeg on the unseen class "Bird" of Pascal-Part-I 16, as well as additional qualitative results Pascal-Part- 116 and ADE20K-Part2255 are available in Section of the supplementary material.
5.3 Results in Few-Shot Part Segmentation
In the few-shot setting, the two-stage baselines have inherent limitations. Firstly, training the MaskFormer from scratch is prone to overfitting with limited data. Secondly, the alass-agnostic proposal generation is unable to generalize from object to part, making it hard to make use of the
8

---

Table 5: Performance on finetuning various modules CLIPSeg both the zero-shot setting and
feo-shot setting on Pascal-Part-Ia and ADE20K -ar00KK-Part---. Part-234.
Model
Pascal-Part-116
ADE20K-Part-234
Setting
Zero-Shot Setting (Oracle)
Few-Shot Setting
Few-Shot Setting
Finetuning
Seen
Unseen
Harmonic
Oracle
Pred
Oracle
Pred
X
22.33
19.73
20.95
21.58
15.38
44.65
26.03
32.89
29.86
27.16
24.01
12.96
F
31.34
21.45
25.46
L+F
38.11
21.14
27.19
27.61
25.90
26.04
14.55
TA+F
33.13
21.70
26.23
VA+F
45.13
22.55
30.07
29.99
26.84
23.74
12.72
VA+TA+F
44.83
24.16
31.39
VA+L+F
47.04
27.69
34.85
31.34
27.67
28.67
17.32
VA+L+F+D
48.68
27.37
35.04
33.13
30.70
29.36
16.12
object-level data. We reported the results of finetuning various modules in CLIPSeg shown in T5 We can draw similar conclusions regarding the different finetuning strategies as observed in the zero-shot setting results. Besides, we observe inconsistent performance trends on Pascal-Part- and ADE20K- PPrtt-234: (1) Visual modality finetuning is superior on Pascal-Part- 16 while is inferior DDE20K-Part2234 (L+F VA+F). (2) Finetuning decoder (D) even underperforms the single language modality (I++F) on ADE20K- Partt-22, whereas the opposite trend is observed on Pascal-Part- (3) Finetuming VA+L+F+D on ADE20K-Part-2
brings 0.69% gains VA+L+F under the Oracle-Obj Setting but causes 1 1.2% performance drops (1.2%) under the Pred-Obj Setting, which is different from the Pascal-Part-116. The cause of the inconsistencies is that, ADE20K-Part-255 is more challenging than Pascal-Part- 1 16 thus the transferability gap from object to part is larger. Finetuning more parameters may cause worse performance and biased to parts compared to objects. Qualitative results are left to Section C of the supplementary material.
5.4 Results in Cross-Dataset Part Segmentation
We reported three baselines as shown in Table 4 CATSeg with backbone and decoder finetuned, CLIPSeg with Visual Adapter, Language Embedding and FiLM (VA+L+F) finetuned and further the decoder VA+L+F+DD finetuned. Under the Oracle-Obj Setting, CLIPSeg performs better than CATSeg on both source and target datasets. But CATSeg less biased the part as performs the best on the target dataset under the Pred-Obj Setting. Hence CATSeg is less prone to overfitting than CLIPSeg with even more learnable parameters. We also explore the cross-datasct part segmentation from Pascal-Parts
P24 for analyzing the potential failures cases in either the part boundaries delincation or the language misunderstanding. The qualitative results are left to Section of the supplementary material.
to
Conclusion
In conclusion, open-vocabulary part segmentation has unique challenges: the limited availability of labeled data, the complexity of part structures and the open granularity challenge. To inspire future research, we proposed the OV-PARTS benchmark with newly cleaned Pascal-Part-I and ADE20K- Part-234 datasets. And introduced three benchmark tasks: Generalized Zero-Shot Segmentation, Cross-Dataset Segmentation, and Few-Shot Segmentation. which assesses the analogical reasoning, open granularity, and few-shot adapting abilities of OV-PARTS models. Furthermore, improved two paradigms from existing object-level OVSS methods as the baselines. Through comprehensive experimental analysis, provided insights into the strengths and limitations of current approaches, highlighting the potential of leveraging large foundation models for OV-PARTS.
Ackmowledgements. This work is supported by Shanghai Artificial Intelligence Laboratory, HKU Startup Fund, HKU Seed Fund for Basic Research, and HKU Seed Fund Translational and Applicd Research. Shu Kong supported by R2G2003-000444-0

---

References
[1] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
[2] Guangyi Chen, Weiran Yao, Xiangchen Song. Xinyue Li. Yongming Rao, and Kun Zhang. Prompt learning with optimal transport for vision-language models. arXiv preprint arXi:22200.0/253, 2022.
[3] Liang-Chich Chich Chen, Yi Yang, Jiang Wang, Wei Xu, and Alan L Yuille. Attention scale: Scale-aware semantic image segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Xianjie Chen, Roozbeh Mottaghi, Xiaohai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1971-1978, 2014
[5] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34:17864-1
2021.
[6] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1290-1299) 2022.
[7] Seokju Cho, Heeseong Shin, Sunghwan Hong, Seungjun An, Seungjun Lee, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open- vecabuoayy semantic segmentation. arXiv preprint arXiv:2303. 1799, 2023.
[8] Daan de Geus, Panagiotis Meletis, Chenyang Lu, Xiaoxiao Wen, and Gijs Dubbelman. Part aware panoptic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition. pages 5485-5494 2021.
[9] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11583- 11592, 2022.
[10] Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature-wisg wise transformations. Distill, 2018.
[11] M Everingham, I. Van Gool, C. Williams, Winn, and A Zisserman. The pascal visual ob- ject classes challenge 2010, voc 2010 (2010). Results http://www. pascal-network. org/chal-
html, 2010.
[12] Peng Gao, Shijie Geng, Renrui Zhang. Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better languae models with feature adapters. arXiv preprint arXiv:2110.04544 2021.
[13] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Open-vocabulary image segmentation. arXiv preprint arv:: 21.. 2021.
[14] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang. Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: bext-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007-6017, 2023.
[15] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.0m 02643, 2023.
[16] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[17] Xiangtai Li, Shilin Xu, Yiho Yang, Guangliang Cheng, Yunhai Tong, and Dacheng Tao. Panoptic- partformer: Learning unified model for panoptic part segmentation. Computer Vision-ECCY 2022: 17th European Conference, Aviv, Isruel, October 23-27, 2022, Proceedings, Part XXVII, pages 729-747. Springer, 2022.
[18] Chuang Lin, Peize Sun. Yi Jiang. Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jian- fei Cai. Learning object-language alignments for open-vocabulary object detection. arXiv preprint arXiv:2211.14843, 2022.
10

---

[19] Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler. Editgan: High-precision semantic image editing. Advances in Neural Information Processing Systems, 34:16331- 16345, 2021.
[20] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li. Hao Zhang, Jie Yang. Chunyuan Li. Jianwei Yang. Hang Su, Jun Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, arXiv preprint arXiv:2303.
2023.
[21] Wenhao Lu, Xiaochen Lian, and Alan Yuille. Parsing semantic parts of cars using graphical models and segment appearance consistency. arXiv preprint arXiv: 2375, 2014.
[22] Timo Lüddecke and Alexander Ecker. Image segmentation using and image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7086-7096, June 2022.
[23] Umberto Michieli and Pietro Zanuttigh. Edge-aware graph matching network for part-based semantic segmentation. International Journal of Computer Vision, 2022.
[24] Umberto Michieli, Edoardo Borsato, Luca Rossi, and Pietro Zanuttigh. Gmnet: Graph matching network for large scale part semantic segmentation in the wild. In Computer Vision- ECCV 2020: 16th European Conference, 2020.
[25] Tai- Yu Pan, Qing Liu, Wei-Lun Chao, and Brian Price. Towards open-worts segmentation of parts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15392- 15401, 2023.
[26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[27] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipula- tion. In Conference on Robot Learning, pages 894-906. PMLR, 2022.
[28] Peize Sun, Shoufa Chen, Chenchen Zhu, Fanyi Xiao, Ping Luo, Saining Xie, and Zhicheng Yan. Going denser with open-vocabulary part segmentation. arXiv preprint a7Xiv:2305.11173, 2023.
[29] Chufeng Tang, Lingxi Xie. Xiaopeng Zhang, Xiaolin Hu, and Qi Tian. Visual recognition by request. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[30] Jianyu Wang and Alan L Yuille. Semantic part segmentation using compositional model combining shape and appearance. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015.
[31] Chenyun Wu, Zhe Lin, Scott Cohen, Trung Bui, and Subhransu Maji. Phrasecut: Language- based image segmentation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10216 10225, 2020.
[32] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models. arXiv preprint 2rX3v:2303.04800 2023.
[33] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao. Han Hu, and Xiang Bai. simple baseline for zero-shot semantic segmentation with pre-trained vision-language model. arXiv preprint 112iv:2112.14757. 2021.
[34] Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017.
[35] Bolci Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.
[36] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference Computer Vision (ECCV). 2022.
[37] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-1688 16825, 2022.

---

[38] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130())2337-23355 2022.
[39] Tianfei Zhou, Wenguan Wang. Si Liu, Yi Yang, and Luc Van Gool. Differentiable multi-granularii human representation learning for instance-aware human semantic parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1622-1631, 2021.
12

---

Table A.1: Model complexity analysis on Miscal-Part-llle ZSseg+ 1/2st first/second stage.
Model
Backbone
Image Size
Trainable Params (M)
FLOPs (G)
Z 1st
ResNet- 101c
512 x 704
61.1
103.9
7 + 2st
ViT-B/16
384 x 384
0.004
58.9
CATSeg
ResNet-101 101 & ViT-B/16
384 x 384
30.9
139.0
CLIPSeg
ViT-B/16
352 x 352
1.3
97.9
A Implementation Details
Two-Stage Baselines. Except for the Object Mask Prompt and Compositional Prompt Tuning designs, we follow the default architecture in the original ZSseg paper. The number of part queries set to 50, All the two-stage baselines are trained with AdamW optimizer with the initial learning rate of le-4 and weight decay of le-4. A poly learning rate policy with a power of 0.9 is adopted. The total batch size is 16 and the total training iteration 20k. For the training of Compositional Prompt Tuning, SGD optimizer with the initial learning rate of 2c-2 and weight decay of 5e-4 is used. And we adopt warm-up cosine learning rate policy with 100 warm-up iterations. The total batch size is 32 and the total training iteration is 3k. We sample 128 training samples for each object part class. The length of the learnable object and part prompt tokens are 4. The object tokens are initialized from the text embedding of the template "a photo of". The initial value of the learnable fusion weight is 0.5.
One-Stage Baselines. We adopt the original architecture of both CATSeg and CLIPSeg as described their respective papers. For finetuning CATSeg, we utilize their pretrained model with a ResNet- 101 backbone. However, while CATSeg achieves the best performance by finetuning the attention layers of CLIP's visual encoder in open vocabulary object segmentation, we observe poor performance with the same finetuning strategy in OV-PARTS. In our experiments, we only finetune the backbone with a backbone multiplier 0.1 and the swin transformer based decoder. We employ an Adam W optimizer with initial learning rate of 2e-4, weight decay of le-4, and cosine learning rate policy. The total batch size is 8, and the training iterations amount to 40k. For CLIPSeg, we utilize the same optimizer settings and learning rate policy as CATSeg. The training iterations set 20k for the zero-shot/croos-aatass part segmentation setting and 3k for the few-shot part segmentation setting.
Model Complexity. We analyze the computational complexity of these two types of baselines and summarize the number of trainable parameters and FLOPs in Table [A.] The complexity is evaluated on Pascal-Part-116 It's evident that the one-stage CLIPSeg. which solely refines a lightweight transformer decoder and employs parameter-efffiient modules, showcases the fewest trainable parameters and the lowest FLOOPs In contrast, the two-stage 7Sseg+ approach, involving the training of a complete maskformer with larger resolution, leads the highest count of trainable parameters and FLOPs.
B Benchmark Datasets Details
B.1 Pascal-Part-116
Pascal-Part-1116 contains 8431 training images and 850 testing images. Compared to the original version of Pascal-Part, we discard the directional indicator for some part classes and merge them to avoid overly intricate part definitions. The category vocabulary and merging details are follows:
aeroplane [body, stern, left/right wing, tail, engine, wheel]
bicycle [front/back wheel, saddle, handlebar, chainwheel, headlight]
bird [left/right wing, tail, head, left/right eye, beak, torso, neck, left/right leg, left/right foot]
bottle [body, cap]
bus [wheel, headlight, front, left/right side, back, roof, left/right mirror, front/back license plate, door, window]
13

---

tvmonitor
train
sheep
pottedplant
person
motorbike
horse
dog
cow
cat
car
bus
bottle
bird
bicycle
aeroplane
500
1000
1500
2000
Figure A.1: The number of object masks that have corresponding part masks in Pascal-Part 11.
tvmonitor
train
sheep
pottedplant
person
motorbike
horse
dog
cow
cat
bus
bottle
bird
bicycle
aeroplane
2500
5000
7500 10000 12500 15000 17500 20000
Figure B.2: The number of part masks for each object class in Pascal-Part-11 16. Each horizontal bar is color-coded to represent specific part class belonging to the object. The colors of the bars ordered from left to right based on the part sequence in the list of objects with parts.
14

---

car [wheel, headlight, front, left/right side, back, roof, left/right mirror, front/back license plate, door, window]
cat [tail, head, left/right eye, torso, neck, t /left-back/right-bact leg, nose, aet.:taa//rggl...ttoe/feeaaaaaaa paw, left/right ear]
cow [tail, head, left/right eye, torso, neck, Let--tooo---pppp//fff----o------
/right-back-lower leg, left/right ear, muzzle, left/right horn]
dog [tail, head, left/right eye, torso, neck, lett-front/righttrrort /eft-back/righttbbad. leg, nose, aet--foaw//ghhtttnmmmlffeaaaaass paw, left/right ear, muzzle]
horse [tail, head, left/right eye, torso, neck, hett-tthott-ppppr/alt----o----- rright-front-upeerrriittt http:///rttgtttttttooooo//etttat htwprr//tt.ccccppr/raafaaaaaa//r /right-back-lozen leg, left/right ear, muzzle, Let--ffonm/rg@ttftro///ee---e /right-back hoof]
motorbike [front/back wheel, saddle, handlebar, headlight]
person [head, left/right eye, torso, neck, leg, foot, nose, left/right ear, left/right eyebrow, mouth, hair, left/right lower arm, left/right upper arm, left/right hand]
pottedplant [pot, plant]
sheep [tail, head, left/right eye, torso, neck, hett-foan---perr/ett---foo------
/right-back-lorer leg, left/right ear, muzzle, left/right horn]
train [headlight, head, front, left/right side, back, roof, coach] tvmonitor [screen]
The unseen objects are colored blue and the removed terms are colored purple
B.2 ADE20K-Part-234
The original subset of SceneParse comprises 20,210 training images and 2,000 validation images. After filtering out less frequent parts, the subset is reduced to 7,347 training images and 1,016 validation images. ADE20K, most object parts have sparse mask annotations, and only subset of object instances have part annotations. Hence, ADE20K-Part object mask annotations along with their part mask annotations. To maximize the use of labeled data and ensure authentic evaluations, different data splits are designed for the three task settings. (I) Generalized Zero-Shot Part Segmentation: Models are trained on the seen object instances from the 7,347 training images. Testing is performed on both unseen object instances from the same 7,347 training images and all object instances from the 1,016 validation images. (2) Few-Shot Part Segmentation: For each object class, 16 training images are sampled following the approach in Liscal-Part-1l we adapt the validation set from the generalized zero-shot part segmentation setting by removing the images that occur in the sampled 16-shot training set. (3)
provides the instance-level
Part Segmentation: The original data split (7347/1016 training/validation images) used since we mainly test on the Pascal-Part- 116 dataset. The annotated objects with their parts are listed as follows:
person [arm, back, foot, gaze, hand, head, leg, neck, torso] door [door frame, handle, knob, panel] clock [face, frame]
toilet [bowl, cistern, lid]
cabinet [door, drawer, front, shelf, side, skirt, top]
sink [bowl, faucet, pedestal, tap, top]
lamp [arm, base, canopy, column, cord, Eighlight,,light source, shade, tube] sconce [arm, backplate, highlight, light source, shade] chair [apron, arm, back, base, leg, seat, seat cushion, skirt, stretcher] chest of drawers [apron, door, drawer, front, leg]
15

---

3500
3000
2500
2000
1500
1000
500
airplane
beg
bookcase
bus
cabinet
car
chaif
chandelier
chest of drawers
coffee bbble
computer
cooking stove
dess
dishwashe
door
kitchen Iaand
ight
microw
minibiie
octoman
oven
podi table
reingeatoo
Sconce
she
sink
table
television receive
tollet
traffic light
truck
van
wardrobe
washer
swivel
Figure B.3: The statistics for the number of object masks with part masks on ADE20K-Part-254
chandelier [arm, bulb, canopy, chain, cord, highlight, light source, shade] bed [footboard, headboard, leg, side rail] table [apron, drawer, leg, shelf, top, wheel]
armchair [apron, arm, back, back pillow, leg, seat, seat base, seat cushion] ottoman [back, leg, seat]
shelf [door, drawer, front, shelf]
swivel chair [back, base, seat, wheel]
fan [blade, canopy, tube]
coffee table [leg, top]
stool [leg, seat]
sofa [arm, back, back pillow, leg, seat base, seat cushion, skirt] computer [computer case, keyboard, monitor, mouse]
desk (apron, door, drawer, leg, shelf, top]
wardrobe [door, drawer, front, leg, mirror, top] car [bumper, door, headlight, hood, license plate, logo, mirror, wheel, window, wiper]
bus [bumper, door, headlight, license plate, logo, mirror, wheel, window, wiper]
oven [button panel, door, drawer, top]
cooking stove [burner, button panel, door, drawer, oven, stove] microwave [button panel, door, front, side, top, window] refrigerator [button panel, door, drawer, side] kitchen island [door, drawer, front, side, top] dishwasher [button panel, handle, skirt] bookcase [door, drawer, front, side]
television receiver [base, buttons, frame, keys, screen, speaker] glass [base, bowl, opening, stem]
pool table [bed, leg, pocket]
van [bumper, door, headlight, license plate, logo, mirror, taillight, wheel, window, wiper]
airplane [door, fuselage, landing gear, propeller, stabilizer, turbine engine, wing] truck [bumper, door, headlight, license plate, logo, mirror, wheel, windshield]
minibike [license plate, mirror, seat, wheel]
16

---

10000
8000
6000
4000
2000
airplane
arrii articcaair
bed
bench
bookcase
bus
cabinet
car
chair
chandelier
chest of drawers
coffee table b
comput
cooking Stove
desk
dishwas waaseer
door
fan
glass
Isand
lamp
light
microwa
Iitomma
pool tabie
refnigerato
Sconce
shell
snk
S000
swivel chale
recens
Solles
traffic light
truck
van
wardrobe
washer
television rec
kitchen
Figure B.4: The number of part masks for each object class in M-DE20K-Part-222 Each horizontal
bar is color-coded to represent specific part class belonging the object. The colors of each bar are ordered from bottom to top according to the part sequence in the list of objects with parts.
Part'ss Taad
s Bardd wing
bre's head
PIRE VEN TES
DT''S 10
MEDE
Bra
Ledt's
Figure B.5: Qualitaive results on ZSsegt, CATSeg and CLIPSeg concerning the challenging unseen "bird" class in Pascal-Part-11 Part-116 16, as shown in the first row. The second row shows the corresponding ground truth. We can observe that CATSeg and CLIPSeg can generalize the more novel parts: "Bird's Beak" and "Bird's Wing"
washer [button panel, door, front, side]
bench [arm, back, leg, seat] traffic light [housing, pole] light [aperture, canopy, diffusor, highlight, light source, shade]
B.3 Data Statistics Analysis.
We report the statistics for the number of object masks that have part annotations in Pascal-Part- 16 (see Figure I. and ADE20K-Part-22 (see Figure B.3). The total number of part masks for each object and the proportion of each part are shown in Figure B.2 (Pascal-Part-1 16) and Figure B.4 (ADF:20-PPrt-2244
Figure B.2 the color sequence from left right corresponds the part word sequence as listed in Section B.I In Figure B.4. the color sequence from bottom to up corresponds to
17

---

Denerh NM
PATMON
Figure B.6: Qualitative results on CATSeg's multi-granular generalization ability. From the left the middle image, the model generalizes from "head" to the more fine-grained "beard". From the middle the right image, the model generalizes from ["hair", "eyebrow", "eye"] to the coarse-graine poarse-grained "head" and also from "neck" to "torso"
Figure B.7: More qualitative results of generalized zero-shot part segmentation on Pascal-Part Part-116 in the first row. The ground truth is in the second row. The seen classes are "cat" and "horse" while the unseen classes are "dog" and "sheep".
the part word sequence as listed in Section B.2 Additionally, report the scale distribution for the part masks each object as shown Figure
C Qualitative Results
The qualitative results on the comparison among ZSsegt, CATSeg and CLIPSeg for the challenging case "bird" are shown in Figure B.5 Figure shoss the multi-granula generalization ability of the one-stage baselines. The adopted model CATSeg. The visualization sample from the "person" class in Pascal-Pal Saal-Part-1le 116. We give more qualitative results Pascal-Part-11 and ADE20K-Parte on the three proposed task settings. The adopted model CLIPSeg with finetuning (VA+L+F+D). The visualization results for the Generalized Zero-Shot Part Segmentation Pascal-Part-11 and
[0.0.01]
10,0.11]
57.00
62.6%
4 4.6%
4,4%
(0.07.1]
(0 (0.05,0007]
18.5%
7.6%
22.5%
(0.03.0.05)
(0.05.0.07]
[0.01.0.03]
00.03.0.55
[0.01.9.03)
Figure B.8: The scale ratio (number pixels in the object part mask out of pixels in an image.) distribution of all part masks of Pascal-Parte 116 (Left) and ADE20K-Pann (Right).
18

---

ADE20K-Part-222 The first and second rows show the generalize from the seen classes [chair, armchair, sofa] the unseen classes [swivel chair, ottoman, stool]. The third row shows the generalize from the seen classes [lamp, chandelier] to the unseen class [fan]. Notably, "fan's 's blade" is novel at the object and part level.
Figure B.9: Qualitative results of generalized zero-shot part segmentation
Figure 10 Qualitative results of few-shot part segmentation on Pascal-Part 116. We display the segmentation map of four classes: "bird", "aeroplane", "car" and "bicycle".
19

---

Figure B.11: Qualitative results of few-shot part segmentation ADE20KKPPatt223 We display the segmentation map of four classes: "lamp", "sink", "toilet" and "cooking stove".
Figure 1.12: Qualitative results of cross-dataset part segmentation 116 provides more fine- mne-grained part annotations for the "person" category, such as "hair" and "upper arm". The model trained ADE20K --22t-Part-234 demonstrates the ability recognize "hair" but struggles to generalize from "arm" to "upper arm" and "lower arm" accurately. Moreover, the model exhibits potential in generalizing parts of the "airplane" category. Although ADE20K-Part-2 annotates the parts as "door", "fuselage", "landing gear", "propeller", "stabilizer", "turbine engine", and "wing". the model can generalize them to Pascal-Part- 16's parts, including "body", "stern", "wing", "tail", "engine", and "wheel", despite the differences in vocabulary and granularity. Notably, does not contain related classes "bird", "sheep", and "potted plant", but the model demonstrates a certain level of generalization ability to segmenting these categories.
Auscal-Part-i 116. Pascal-Part-
nler
Figure 13: Qualitative results of cross-dataset part segmentation on ADE20K-Part For the categories "car" and "bus", the part annotations in Pascal-Part- 16 more coarse- granse.. When tested on ADE20K-Part-2
the model trained on Pascal-Pas Partt-art-ll 116 can predict novel parts like "logo", "wiper", "hood", and "bumper". However, the segments and part labels don't align accurately, For example, the model still segments the "bus's 's roof", which is annotated in Pascal-Parts 16, but wrongly assigns it to "bus's 's bumper" ADE20K-Part- 234. This showcases the challenge of generalizing across different granular part definitions. For the novel object "swivel chair", the model adeptly delineates part boundaries even without relevant objects Pascal-Part-I 16. But the category errors are still present. In the case of the "person" category, the model only segments the "upper arm", which demonstrates the difficulty generalizing from "upper/lower arm" "arm".
20

---

ADE20K -art0KK-- are shown in Figure B.7 and Figure B.9 respectively. We report the qualitative results for the Few-Shot Part Segmentation on Pascal-Part- in Iin on ADE20K-Part 234 in Figure B.11 And the results for the Cross-Datase Part Segmentation on Pascal-Part- 16 are shown in Figure B.12 Furthermore, we present the qualitative results for models trained on Pascal-Part-1 16 and then tested on ADE20K-Part-234 are shown in Figure B.13
D Future Works and Negative Societal Impacts
Although part-level OVSS indeed presents more challenges compared to object-level OVSS, the OV-PARTS benchmark datasets have lower quality than existing object-level OVSS benchmark datasets. The original version of Pascal-Part and ADE20K-Part are annotated without considering the open vocabulary scenario especially the analogical reasoning ability and open granularity ability that we care about in a part-level OVSS model. The benchmark datasets need to be continuously expanded and improved encompass be potential negative societal impacts associated with the OV-PART benchmark. The deployment of fine-graet fine-grained part segmentation models various real-world applications may lead to unintended consequences. We must ensure that the predictions be reliable and accurate in critical applications, such medical diagnosis or autonomous vehicles. Also, there possibility of misuse of part segmentation technology for malicious purposes, such as creating deepfake images or spreading misinformation. Ensuring security measures and appropriate regulations to prevent such misuse vital in the development and deployment process.
diverse and complex object-part annotations. There may
21


Reviews:
[{"review_id": "ZX3sN4tLdy", "reviewer": "Reviewer_NFDP", "summary": "", "questions": "", "limitations": "- Limitations of the current models were mentioned in the texts. However, the limitations of the benchmark were not discussed. Is there any aspect of open-vocabulary part segmentation that the proposed tasks did not cover? What can potentially be further improved on the data end in the future?\n- Potential negative societal impact was not discussed in either the main text or the supplement.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Explored open-vocabulary part segmentation, a significant task in the research community.\n- Thorough analysis of failure modes in state-of-the-art (SOTA) models, providing insights for future performance improvements.\n- The datasets are the cleaned up versions of existing noisy datasets that improves evaluation quality\n- Interesting finding that zero-shot performance of the OVSS models is already good on OV-PARTS\n- The experiments and observations in the paper help explain the substantial performance gap in performance between OVSS task and OV-PARTS.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "i8d0bZE8Cr", "reviewer": "Reviewer_r29M", "summary": "", "questions": "", "limitations": "This paper provides insights into the strengths and limitations of current approaches, highlighting the potential of leveraging large foundation models for OV-PARTS. \nWhy select ZSseg and CLIPSeg as baseline may require to be explained, and more diverse explorations on other state-of-the-art methods are expected.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "To address three challenges of open-vocabulary part segmentation, this paper proposes a benchmark containing three specific tasks and reorganized two datasets. The motivation for this paper is well justified, and implemented. The overall organization of the paper flows well. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "RctdmY2czM", "reviewer": "Reviewer_cZ7P", "summary": "", "questions": "", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. An important benchmark for studying open vocabulary part segmentation\n2. The benchmark can be used for a couple of tasks\n3. Experimental analysis is provided", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "TdBcX3G8Am", "reviewer": "Reviewer_osrm", "summary": "", "questions": "", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The writing in the manuscript is clear and understandable.\n2. The experiments are set up in a comprehensive manner, offering a complete benchmark for comparison.\n3. Some improvement strategies suggested by the authors appear to be effective based on the presented data.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "h7UD9ItuG6", "reviewer": "Reviewer_cNC2", "summary": "", "questions": "", "limitations": "see Opportunities For Improvement", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Object part datasets are rare, so adding a clean benchmark will be valuable to the community.\n- Baselines are strong, there are clear improvements to the existing methods on the new benchmarks.\n- Many things are tested and evaluated in different ways so the information density in the paper is high.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "7TBVUIpwGs", "reviewer": "Reviewer_VdPY", "summary": "", "questions": "", "limitations": "Please refer the above section.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "- As shown in Fig. 2, granularity gaps between the methods are important problems, but have been yet handled.  \n- The paper categories challenging open-vocabulary settings for part segmentation, including generalized zero-shot part segmentation, cross-dataset part segmentation, and few-shot part segmentation, which makes sense.\n- Not only the paper evaluates various methods, but also it analyzes the ablation of the previous method, as shown in Tab. 3 or 4.\n- The paper is well written.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "3AqaOtMETf", "reviewer": "Reviewer_v7gi", "summary": "", "questions": "", "limitations": "Not applicable.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The author has conducted a large number of comparative experiments on the proposed task, which will help researchers to understand the gap between various methods.\n2. The author proposes a benchmark for part segmentation and conducts systematic experiments, task design, and dataset construction, which will contribute to the development of the community.\n3. The writing and readability of the article are good.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper receives borderline reviews. But importantly, all 7 reviewers were leaning towards acceptance, and no major concerns with the dataset were raised post-rebuttal. Key elements for dataset and benchmarking papers, including formulation of the evaluation (r29M, cZ7P, v7gi, cNC2), the benchmarking and analysis of SOTA failures (NFDP, cZ7P, osrm, v7gi, VdPY, cNC2) and the insights provided by the work (NFDP, osrm, cNC2), are all presented properly in the work. Therefore, AC recommends for acceptance.
Decision: Accept (Poster)