Understanding Multi-Granularity for Open-Vocabulary Part Segmentation
Authors: Jiho Choi
Seonho Lee
Seungho Lee
Minhyun Lee
Hyunjung Shim
Venue: NeurIPS 2024 poster (2024)
Keywords: part segmentation
open-vocabulary
multi-granularity
Primary area: machine_vision
TL;DR: PartCLIPSeg is a novel framework that enhances open-vocabulary part segmentation by utilizing object-level contexts and attention control, significantly outperforming existing methods on major datasets.

Abstract: Open-vocabulary part segmentation (OVPS) is an emerging research area focused on segmenting fine-grained entities using diverse and previously unseen vocabularies.
Our study highlights the inherent complexities of part segmentation due to intricate boundaries and diverse granularity, reflecting the knowledge-based nature of part identification.
To address these challenges, we propose PartCLIPSeg, a novel framework utilizing generalized parts and object-level contexts to mitigate the lack of generalization in fine-grained parts.
PartCLIPSeg integrates competitive part relationships and attention control, alleviating ambiguous boundaries and underrepresented parts.
Experimental results demonstrate that PartCLIPSeg outperforms existing state-of-the-art OVPS methods, offering refined segmentation and an advanced understanding of part relationships within images.
Through extensive experiments, our model demonstrated a significant improvement over the state-of-the-art models on the Pascal-Part-116, ADE20K-Part-234, and PartImageNet datasets.

Understanding Multi-Granularity for Open-Vocabulary Part Segmentation
Jiho Choi'; Seonho Lee'r, Seungho Lee", Minhyun Lee", Hyunjung Shim't
GGaduute School of Artificial Intelligence, KAIST, Republic of Korea Schooo of Integrated Technology, Yonsei University, Republic of Korea (jihochoi, glanceyes, (aaeeshmmmkaisteeee. (seungholee, 1mh315)@yonsei.. kr
139's Cy
Bag KY1
tano
Hogg hocms
Joq' p30
(a) fhound-truth
(b) Object-level Pred. (c) Generalized Parts Pred.
(d) Final Pred.
(c) "head"
"torso"
(g) "car"
(h) "nose"
"paw"
Figure 1: Prediction results of our ParCLIPSSeg for unseen categories in the Pascal-Parte Parcal-Part-116 [7, 46] validation set. A "dog" unseen during training. The final prediction of PartCLIPSeg utilizes (b) object-level context and (c) generalized parts, incorporating disjoint activation among (e)-(i) parts, and enhancing activation for smaller parts (e.g.. (h) "nose").
Abstract
Open-vocabulary part segmentation (OVPS) is an emerging research area focused on segmenting fine -iaeggr entities using diverse and previously unseen vocabu- laries. Our study highlights the inherent complexities of part segmentation due intricate boundaries and diverse granularity, reflecting the knowledge-basee nature of part identification. To address these challenges, we propose PartCI.IPSe novel framework utilizing generalized parts and object-level contexts to mitigate the lack of generalization in fine-grained parts. PartCLIPSeg integrates competitive part relationships and attention control, alleviating ambiguous boundaries and underrepresented parts. Experimental results demonstrate that PartCLIPSeg outper- forms existing state-of-the-arr OVPS methods, offering refined segmentation and an advanced understanding of part relationships within images. Through extensive ex- periments, our model demonstrated a significant improvement over the state-of ohethe- models on the Pascal-Part Parccal-Paat---ee P16,-116, ADE20K-Part-22 and PartImageNet datasets. Our code is available at https: //gi thub. com/kai /aai aist-ttt- comilslttlmmmmmmmmmmmmmmmmm cvml haallppat-aaee cilipseg ipseg.
"Equal contribution 'Corresoonding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
The pursuit of understanding parts and muatulgranllarity in computer vision [7. 13, 21] mirrors the innate complexities of animal instincts. For example, a "cheetah" instinctively targets an "impala's neck" during a hunt, demonstrating its ability distinguish specific parts. This ability extends to applications such as robot commands [44], fine-grained controls on image editing [31], and more sophisticated image generation [45]. Part segmentation aims to mimic this ability by recognizing intricate details (e.g., parts) within objects, going beyond simple object-level segmentation to achieve detailed and diverse entity recognition.
Recognizing parts is more challenging than recognizing whole objects due to their complexity and diversity. Parts often have ambiguous boundaries not only defined by visual cues but also require broader spectrum of contextual information, reflecting their knowledge-basse nature. For example, the "head" of "dog" may include only the "face" or also the "neck" depending on the annotators' perspective [7, 21J.
To address difficulties in part segmentation, Open- Vocabulary Part Segmentation (OVPS) [40, 44, 46] has evolved by leveraging the knowledge of powerful Vision-Language Models (VLMs) like CLIP [38] ALIGN [24]. Especially, it aims to achieve adaptive recognition and processing of previously unseen categories with the aid of pre-trained VLMs, pushing the boundaries of vocabularies in traditional part segmentation. By utilizing Oracle supervision of base classes during training, recent studies in OVPS exploit part-level knowledge of base classes to generalize to novel classes. Recently, VLPart [40] uses DINO [5] features map correspondences between base and novel classes and creates pseudo labels for the novel categories. OV-PARTS [46] addresses the ambiguity of part boundaries by introducing object mask prompts and transferring knowledge of base class through a few-shot approach. These methods successfully extract knowledge from VLMs and extend it to novel classes, achieving significant performance improvements in open- vocabulary settings.
acroplane's 's stem
bird's 's head
bird torso
bird's tail
aeroplane's wing
bird's wing
acroplane's body
bird's 's log
acreplane's engine
hird's
acreplane's wheel
Ground Truth
(a) Lack of generalization
(b) Ambiguous boundaries
(c) Missing underrepresented part
Figure 2: Limitations of existing OVPS methods in predicting unseen categories. (a) Lack of generalization: Classification of a "dog's parts" involving categories like "cats" and "sheep", "dog's tail' misclassified as "sheep's 's ear". (VLPart [40]) (b) Ambiguous boundaries: Vague boundary output of "aeroplane's 's body". (c) Missing underrepresented parts: Neglecting parts such as "beak" and "leg". (CLIPSeg [32, 46]).
However, through empirical analysis of existing OVPS methods, we observed several common limitations in Figure 2. (Lack of generalization in (a)) Despite understanding part-level information, they often misidentify parts at the object level, e.g.. a "dog's s leg" as a "cat's 's's leg". Also, pert-level misclassification occurs as the knowledge of parts in the base class fails to generalize to novel class, e.g., "dog's tail" as a "sheep's 's ear". (Ambiguous boundaries of parts in (b)) They fail to maintain non-overlapping relationships between parts, frequently resulting in overlaps, e.g., an "airplane's wing" overlapping with its "body" or the presence of empty spaces where no part is predicted. (Missing underrepresented parts in (c)) They ignore small and less frequent parts, causing prediction bias based on part size.
To overcome these limitations, we propose novel framework called ParCLIISSe, which consists of three main components. First, we devise generalized parts with object-level contexts address the lack of generalization issue as the upper side of Figure 1. It explicitly obtains object-level and part-level pseudo-labels from VLMs and trains the OVPS model to satisfy both types of supervision. This guides the model to learn object boundaries while recognizing both part and object-level classes. Then, we suggest an attention control for minimizing the overlap between predicted parts, ensuring that parts are clearly separated as the lower side of Figure 1. In this way, we effectively leverage

---

internal part information to learn ambiguous part boundaries. Finally, we enhance the activation related certain parts by normalizing the activation scale of CLIP's self-attention information. prevents small and less frequent areas from being ignored in pseudo-labels. This strategy ensures that the smallest granularity levels are retained in the final prediction. Through these three modules, PartCI.PPSg effectively addresses the challenges of existing OVPS methods and achieves robust multi guanulltatllrrttyy segmentation. As a result, the proposed method achieves significant improvements in mloU for both unseen and the harmonic mean when compared to previous stateof-theeart methods on Pascal-Part- 16, ADE20K Part-224, and PartImageNet in both Pred-All and Oracle-Obj settings.
2 Related Work
Open- Voca-Vocay Semantic Segmentation. Open-vocabulary [19, 55] semantic segmentation (OVSS) goes beyond traditional semantic segmentation, which is restricted to predefined categories, by enabling predictions for unseen classes. Pioneering works focused on aligning predefined text embeddings with pixel-level visual features [4, 48, 56]. By leveraging large-scale Vision -anguuge
Models (VLMs) like CLIP [38] and ALIGN [24], OVSS enables zero-shot segmentation through rich multi-modal features learned from extensive image-text pairs. MaskCLIP [58] modified CLIPPs image encoder directly handle visual and text features for segmenting novel classes. Some works proposed two-stage strategy [15, 16, 18, 20, 29, 30, 51, 52]: first, models generate class-agnostic mask proposals [9, 10]; then, a pre-trained VLM predicts the category for each region. Some studies have introduced diffusion models to improve mask generation quality [51] or fine-tuned CLIP to enhance classification capabilities [20, 29]. Other studies have adopted a single-stage framework [11, 27, 32, 49, 54, 59J. They use pre-trained CLIP models to align pixel-level visual features with text features. CLIPSeg [32] adds a transformer- buaseed pixel decoder with a FiLM [17] module to fuse multi-modal features. ZegCLIP [59] enhances segmentation by incorporating Iearnable tokens. SAN [53] adopted a side adapter network for CLIP-aware end-to-end approach to predict proposal-wise classification. FC-CLIP [54] uses frozen convolutional CLIP to predict class-agnostic masks and classifies using mask -pooeed features [54]. CAT-Seg [11] and SED [49] generate pixel- level cost maps and refine them for segmentation.
Part Segmentation. Part segmentation aims to identify the individual parts of objects, a task that is more complex and costly due to the smaller and more diverse nature of parts compared to whole objects. To tackle this, various datasets like Pascal-Part [7], PartImageNett [21], ADE20K-Part [57], Cityscapes-
Ptyssapes-Panoopti-- Panoptic-PPaaa Part [13], and PACO [2] provide diverse and detailed part annotations. Earlier studies [7, 12, 22, 23, 43] used self-supervised constraints and contrastive settings for effective part-level entity segmentation. Recent studies extended this to open-vocabulary scenarios [35, 40, 46), opening new avenues for handling diverse parts. By leveraging class-agnostic detectors [35] and Vision-Language Models like CLIP [40, 46], part segmentation has extended its generalization ability to unseen parts. Our work builds upon and extends methodologies from these studies.
3 Methodology
As illustrated in Figure 2, we identified three primary challenges of open- vocennuaa part segmentation (OVPS): lack of generalization, overlapping parts, and missing underrepresented parts. Recognizing object-specific parts (such as "dog's torso") cannot be determined solely by looking at each part in isolation; it is imperative to consider both generalized part information and the overall context of the object. Furthermore, some parts may have overlapping meanings across different granularity labels (c.g., "eye", "face", and "head"). This implies that predictions should consider direct guidance for each part well as the relationships between different parts. These intricate spatial and functional dependencies between parts are crucial for achieving a holistic understanding and precise predictions in fine- fine-grained entity segmentation tasks.
Based on this motivation, we propose a novel OVPS method, PartCLIPSeg. This method leverages generalized part information combined with object-level context to tackle the lack of generalization problem (see Section 3.2). Also, we directly minimize the overlap among part predictions to improve the part boundaries (see Section 3.). Finally, we normalize the scale of attention activation from various parts for handling missing underrepresented parts (see Section 3.3.2). The overall architecture of our method is shown in 3.
3

---

"bus" "parson"
BBd
FILM
CLIP Text Encodar
Recon
"window" "side" "tarso"
PART
CBJ'
PARZ
FARE
PARO
CLIP Image Encoder
Transformr Decoder
Recon
FILM
Attention Control (Separate Enhaace))
Attn. Recon
Generalized Parts with Objjcctlovel Contexts
Trans
Conv
"bux"
"'doo""
Objech Contents
Goneralized
Object-speclffe
The overall architecture of PartCLIPSeg. The embeddings derived from the object category name and the part category name are conditioned using the FiLM operation. Each embedding. modified through attention control, is subsequently reconstructed to predict the final part results.
Figure 3:
3.1 Preliminary
OVPS aims to segment image into of object-specific part categories Clest obj-purt (e.g.. "dog's head," "car's front") the test set, where the image is I # and H and W are the height and width. During training, image-mask pairs Mx)) are used, consisting of images Is and corresponding ground-truth mask Mr which only contains the object-specific part categories Cttrii obj-ppart (e.g., "cat's head," "bus's front") in the Irain
Zero-Shot Part Segmentation. Open-vocabulary is a generalized zero-shot task, allowing the zero-shot segmentation protocol to evaluate zero-shot part segmentation performance. In this setting, train and test category names are divided into seen (base) and unseen (novel) sets, respectively, with
disjoint object-specific category names;
Cupseen pbj-part
abj-pas part = Ø
dataset and evaluated on another without fine-tuning. This means that the category names of the train and test sets come from different datasets, denoted Caain Ctest objttpatt pbrt-part" Considering the domain gap between the datasets, such differences in granularity, this setting is more challenging.
Cross- Doass-Dttsst Part Segmentation. In this setting, the model is trained
3.2 Generalized Parts with Object-level Contexts
To address the problem of lack of generalization, we propose leveraging generalized parts with object-level contexts. The concept of generalized parts involves identifying and utilizing common structural components that are shared across different object-level categories. For instance, many animals have parts like "head" "torso" which, although functionally and visually distinct, may share certain underlying characteristics. By introducing generalized parts from object- -specf-- parts, our PartCLIPSeg can efficiently recognize and segment these object-specific parts across diverse object classes, significantly enhancing the model's ability to generalize from seen to unseen categories.
Although generalized parts help distinguish the part-level categories, the visual information of part may not suffice for accurately classifying their object-level categories. For instance, predicting the "leg" part of an animal can be challenging to identify when solely examining the part as it may not clearly indicate to which animal it belongs. For this reason, there have been attempts to incorporate object-level guidance [33, 40, 46] part segmentation. However, object-level guidance without generalized part may lose contextual information and miss hierarchical relationships.
By integrating object contexts with generalized parts, PartCLIPSeg employs object-leve guidance that captures the holistic essence the object to which parts belong. This integration allows for a more precise understanding and classification of parts, improving the overall performance of OVPS.
Object and Part Embedding Generation. We modified the architecture of CLIPSeg [32, 46), which adopted CLIP [38] encoder-decoder architecture for semantic segmentation. However, it is worth

---

noting that our approach of utilizing generalized parts with object-level context is orthogonal to other previously proposed object-level segmentation methods [5, 26, 28, 39].
The proposed approach begins by parsing an object-specific part category name, Cubj-part E Cabj- part. into separate components: object category name (Cuti) and generalized part category name (Crun) ) e.g., "cat" and "torso". Then, the CLIP text encoder, CLIP7(-) (-). is used to transform these category names into their respective CLIP embeddings (e and epan)) It will condition the image features, e derived from the CLIP image encoder, C as:
CLIP (Ctohj pant), e² (L),
jobj I part]
where denotes frozen pre-trained models. By using Feature-wise Linear Modulation (FiLM) [36, 42], each category name embeddings respectively modulate the image features as:
e² + FiLM(e T (abj] pant))
eabj pan]
where @ is an element-wise sum. FiLM is an adaptive affine transformation widely used for multi- modal or conditional tasks. helps retrieve adequate conditioning for the image features. The modulated image features, efobj [ corresponding to each object and part category name, pass through decoder module. The decoder module will be discussed in detail in Section 3.3. They then proceed through transposed convolution model. Finally, the output mask of the object 8 and part & are evaluated with ground-truth mask of objects, 8", and parts, s". Oracle supervision for the object and parts mask simply computed from a combination of object-specific parts annotations: 8 M.
Object-specific Part Construction. We utilize previously computed generalized part embeddings (epar efun)) and object embeddings (e e/ay) to reconstruct object-specific part embeddings. This process involves separate operations modulated image features and category name embeddings.
Initially, we project the concatenated results of the object category name with the generalized part category name. This is to synthesize the embeddings for the target object-specific part category name. The approach ensures that the resultant embeddings are highly representative of parts and contextually relevant. The equivalent operation is applied to both object-level image features and part-level image features to generate object-specific image features as:
T obj- Proj(
obj T
T
The resulting object-specific part embeddings are further refined by a FiLM process. Combined with the respective object-specific image features, final modulated opject-specific part embeddings, eobj-part is computed as:
Cobj-part ppar
(Lij
These embeddings then processed through deconvolution layer to produce the final segmentation masks s E M. This step ensures that the embeddings are precisely aligned to enhance the definition and accuracy of the object-specifie part masks. It effectively bridges the gap between object and part-level categorical information with object-specifie parts information.
Object, Part, and Object-specific Part Mask Supervision. The mask supervision is provided for three distinct categories: object-specific parts, objects, and generalized parts. This multi-faceted supervision enables our model to effectively disentangle generalized parts from objects, thereby facilitating a more nuanced learning process for OVPS. This disentanglement is crucial for the model to accurately recognize and differentiate between various object categories and their corresponding parts. It enhances the model's 's ability to handle complex segmentation tasks with unseen object- specific parts. The overall mask guidance loss can be defined as follows:
|Cati-pon! 11
IC 111 object guidance
CCee
Grask
BCF(s;,8)) 8:)
+ +Xobs
Σ BCF(+?,87) 87) +> Aart
BCE 8
(5)
spject-specitic part
generalized part guidance
where |Cobj-panl + and |Caaj] + 1 are for uncategory (or background) prediction. The disentangled object and part generalization with object-specific parts guidance provides a clue to the lack of generalization problem.
5

---

Part Mank Attentionn Naps
Intersattion
Atlention Control
Minimize Intersection (Euep)
dogg
bebd
dog COlG
Maximize
Minimum
Aog'
faii
Figure 4: Example of attention control using separation and enhance losses. The proposed method manipulates attention maps to accurately identify and segment small parts.
3.3 Attention Control for Ambiguity and Omission
In this subsection, we address the previously mentioned challenges: (1) ambiguity in part boundaries and (2) omission of small or infrequenty appearing parts. The main reason for these challenges is the incomplete guidance from knowledge based, multi- gratti-gr characteristics of parts. To overcome these, we adopt unsupervised methods traditionally used in fine- Lne-grained recognition and part discovery studies [7, 12, 43]. Specifically, we utilize approaches for adjusting self-attention activation inspired by the recent diffusion methods [6, 25, 41
We assume that the distribution of self-attention activation maps for visual tokens belonging to the same object-specific part mask should exhibit inter-similarity characteristics [41], implying similar distributions. To this end, we first compute the average self-attention map AM. for each object- specific part mask Me, where с E Cobi-pan represents an object-specific part category. This is done by summing the self-attention activation maps from channels specifically corresponding to object Cobj and part Cpur. across all spatial tokens (h, w) within the mask, as follows:
AMe Me
Σ (h,w)EMe
(Aamlly, w; + Aegn [b, w, ;,:
(6)
Subsequently, the self-attention map AM, for the sppecc-specific part mask is refined through min- max normalization, followed by the application of Gaussian filter smooth the initial activation as in [6, 50]. Therefore, the dimensions of both the original and normalized self-attention maps for the object-specifie part masks follows: AM. Abora HxW
3.3.1 Minimizing Part Overlaps for Ambiguity
In the self-attention of the decoder layers, competition between object-specific parts helps define boundaries that cannot be sufficiently established by supervision alone. Using the previously obtained normalized attention map, our method generates parts with minimized intersections, inspired by [1, 25, 37, 47]. This approach effectively mitigates the ambiguity issue in part boundaries. Specifically, the normalized attention activation map A is first binarized based on an arbitrary threshold y as:
(M,Ch, w) = =1 (A (h.
(7)
where M. denotes binarized attention map for part mask Me. From now on, Cobj-part simply denoted as C. The separation loss Ceep. which indicates the degree of intersection between object- specific parts, is as follows:
1 (h, w) Ecsc BM (h, w) > 1 C
Laop
(8)
{(h, w) Eesc B. (h, w) 2 1
where separating activation mitigates the challenge of ambiguous boundaries between parts.
3.3.2 Enhancing Part Activation for Omission
To address the omission problem, we employ method inspired by attention controls in modern diffusion-based approaches [3, 6]. This approach enhances the activation within the self-attention

---

activation map to enhance underrepresented parts before normalization. Specifically, for each object- specific part mask, the maximum value within the attention map is identified. Subsequently, among all object-specific parts, the minimum activation of the part with the maximum value is enhanced as:
Lnhh = 1 - -min max AMelh, wl aEC (b,
thereby boosting its representational efficacy. this way, the enhancement loss Cenh provides sufficient guidance for small or infrequently occurring parts, effectively mitigating the omission problem.
The training objective for PartCLIPSeg integrates three key loss components as:
Cal1 = Luask + >aap + Aant Canh:
(10)
where (1) Cuasx for generalized parts with object-level context, (2) Leep for addressing ambiguous boundaries, (3) Lenh for handling missing underrepresented parts, and Xeep and Xeah are hyperparme- ters.
4 Experiments
4.1 Experimental Setups
Datasets. We evaluate our method on three part segmentation datasets: Pascal-Part- 11rrt-1166 [7, 46], ADE20K-Part-233 [46, 57], and PartImageNee [21]. Pascal-Part- 16 [7, 46] consists of 8,431 training images and 850 test images. It is modified version of PascalPart [7] by removing direction indicators for certain part classes and merging them to avoid overly complex part definitions. This dataset contains a total of 1 16 object part classes across 17 object categories. ADE20K-Part-234 [46, 57] consists of 7,347 training images and 1,016 validation images. provides instance-level object mask annotations along with their corresponding part mask annotations, including 44 objects and 234 parts. PartImageNet [21] contains 16k training images and 2.9k validation images, segmented into 158 object classes from ImageNet [ [4] and organizes them into 11 super-categories. For this study, we select 40 object classes that represent common categories to assess cross-dataset performance effectively. More details about the datasets can be found in the supplementary materials.
Evaluation Protocols. We use two evaluation protocols for the performance OVPS: (1) Pred-All setting, where the ground truth object-level mask and object class are not provided, and (2) Oracle- Obj setting. where the ground truth object-level mask and object class are known. particular, the Pred-Obj setting in OV-PARTS [46] uses predicted masks from the off-the-sher segmentation model. In contrast, our Pred-All setting is more challenging and practical setting because it does not rely on additional predicted masks or foundation models but solely uses the predicted object masks from the proposed model. For both evaluation protocols, we used mean Intersection over Union (mloU) as an evaluation metric, which is widely used to measure segmentation performance. Additionally, we utilized the harmonic mean the results from the seen and unseen categories as the final evaluation metric.
Implementation Details. We build upon CLIPSeg [32, 46], a CLIP-based encoder-decoder model. The implementation details can be found in the supplementary material.
4.2 Performance Evaluation
Zero-Shot Part Segmentation. We compare our PartCLIPSeg to previous methods [11, 32, 46, 52] on three OVPS benchmarks [7, 57]. As shown in Table 1, PartCLIPSeg consistently outperforms previous approaches by significant margins on Pascal-Part- 16. demonstrating its zero-shot ability, with performance improvements of 3. 3.94% in the Pred-All setting and 3.55% in the Oracle-Ob) setting. The more challenging ADE20K Part-234 dataset, which a fine- me-grained segmentation dataset, further highlights the effectiveness of PanCLIPSeg. As shown in Table 2, PartCLIPSeg achieves a harmonic mean mloU of 11 11.3% the Pred-All setting, outperforming the best-performing baseline by 7.85%. In the Oracle-Obj setting, it achieves 38.60%, which is 4.45% higher than the best baseline. Notably, PartCI.IPSeg shows significant performance improvement in unseen categories, demonstrating its strong generalizability. Considering that performance in unseen categories is crucial in zero-shot
7

---

Table 1: Comparison of zero-shot performance with state-of state-of-the-eee the--hh-rrt art methods on Pascal- 16.
Pred-All
Oracle-Obj
Method
Backbone
Seen
Unseen
Harmonic
Seen
Unseen
Hurmonic
7SSeg+ [52]
ResNet-50
38.05
3.38
6,20
54,43
19.04
28.21
VLPatt (40)
ResNet-50
35.21
9.04
14.39
42.61
18.70
99
CLiPSeg [32, 46]
ViT-B/16
27.79
13.27
17.96
48.91
27.54
35.24
CAT-Seg [11, 46]
VIT-B/16
28.17
25.42
26.72
36,20
28.72
32.03
PanCLIPSeg (Ours)
VIT-B/16
23.56 43.4
30.67
50.02.0..1
31.67
38.79
1. 'The
bold and the second best score is underlined. The standard error of an average 5 results is reported. These are the same for all experiments.
Table 2: Comparison of zero-shot performance with state-of-the-art methods on
Pred-All
Oracle-Ohj
Method
Backbone
Seen
Unseen
Harmonic
Seen
Unseen
Harmonic
ZSSeg+ [52]
ResNet-50
32.20
0.89
1.74
43.19
27. 84
33.85
CLIPSeg [32,46]
VIT-B/16
3.14
0.55
0.93
38.15
30.92
34.15
CAT-Seg [11,46] 46]
VTT-B/16
7.02
2.36
3.53
33.80
25.93
29.34
ParCCLIPSeg (Ours)
VIT-B/16
14.15
9.
11,38
38.37
38.82
38.60
(x7.86)
(+4.40)
Table 3: Comparison of zero-shot performance with state-of the--rrrtttt- method on PartImageNet.
Pred-All
Orucle-Obj
Method
Backbone
Seen
Unseen
Harmonic
Seen
Unseen
Harmonic
CLIPSeg [32, 46] ParCLIPScg (Ours)
VIT-B/16 B/16 VIT-B/16
32.39
12.27
17,80
53.91
37.17
44.00
38.82
19.47
25.94
56.26=0.20
#1.65
53.85
(+0.14)
(+5.86)
scenario, these results significant despite some performance degradation in categories. We also evaluated PartCI PartCLIPSeg on PartImageNet. According to Table 3, PanCIIIPSeg shows a notable improvement CLIPSeg.
We present the segmentation results of PartCLIPSeg in comparison to state-of-the-art open-vocabulary part segmentation methods [11, 32, 40] Pascal-Part-11 16. Specifically, we focus on qualitative performance on unseen categories such as "dog", "sheep", "car", and and "bird". As shown Figure for the Pred-All and Figure 6 for the Oracle-Obj setting, the proposed method effectively segments target parts regardless of the need for predefined masks during inference. Notably, ParCLLISSeg excels at identifying smaller, often overlooked part classes such as "eye", "tail", and "headlight" Additionally, our method effectively segments multiple objects and their respective parts, a challenge for other methods, demonstrating the effectiveness of PartCIIPSeg in zero-shot part segmentation. Its improved performance on unseen categories and higher accuracy in challenging environments highlight the robustness and generalization capabilities of ParCCLIPSeg. Consistent improvements on Pascal-Part-11 16, DEE20K-Part-233 and PartImageNet demonstrate that PartCLIPSeg sets new standard in open-vocabulary part segmentation.
Cross- Poos-DDaaset Part Segmentation.
Table 4 validated the efficacy of our approach in cross-dataset setting, where category names, annotation style, and granularity of mask may vary. Additionally, unlike zero-sho shot situations within the same dataset, there are differences in the types and diversity of parts. Initially, we trained our model on PartImageNet and POE:0O-PPrtt-- [7, 46] showed that PartCLIPSeg outperforms CLIPSeg in both the Pred-All and Oracle-Obj settings, confirming our method's superiority on generalization in different datasets.
respectively. Subsequent tests on Pascal-Part
Table 4: Cross-datasee performance.
Table 5: Impact of attention control losses.
Method
Pred-All Oracle-Obj
Loss
Pred-All
Oracle-Obj
PartImageNet > Pascal-Pard 116
Zeeg
Con
Soen
Unseen
Harmonic
Seen
Unseen
Harmonic
Pascal-Patt-116
CLIPSeg 46]
11.72
14.87
PurCLLISSeg (Ours)
14.74
19.86
x
43.86
21.89
29.20
49.09
31.26
38.20
(+3.02)
(+6.99)
x
44.01
28.18
30.37
50.37
31.45
38.72
23.56
30.67
38.79
ADE20K-Part-234 234 - Puscal-Part-10 116
ADEE2KK-Part22
CLIPSeg [32,46]
5.41
17.82
x
10.86
8.33
37.39
36.93
PanCLIPPSeg (Ours)
10.37
17.94
x
12.78
9.38
10.82
39.46
36.04
37.67
(46.96)
(+0.122
14.15
9.52
11.38
38.37
8

---

(a) Ground-truth
(b) VLPart [40] (c) CLIPSeg [38, 46] (d) CAT-Seg [11, 46] (c) ParCLIPSeg (0m)
Figure 5: Qualitative results of zero-shot part segmentation on Pascal- Parl-Parr-11 16 in Pred-All setting.
Annotations for unseen categories (bird, car, dog, sheep, etc.) are not included in the train set.
Table 7: Impact ParCCLIPSeg for small parts on Pascal-Part 116 in Oracle-Obj setting. (mloU)
Table 6: Performance on mean Boundary IoU
(1) on Pascal-Part-I 16 in Oracle-Obj setting,
Part: "eye"
bird
cow
dog
shoep
person
Method
Seen
Unseen
Harmonic
CLIPSeg [32, 46]
3.33
18.77
3.65
16.05
0.00
15.30
PanCLIPSog (Ours)
1.95
31.01
28.16
32.79
0.67
29.16
ZSSeg+ [52]
33.01
26.76
29.56
CLIPSeg [32, 46]
34,67
32.20
33.39
Part: "neck"
cal
cow
dog
sheep
person
CAT-Seg [11, 46]
34.17
30.14
32.03
CLIPSeg [32, 46] ParCCLIPSeg (Ours)
19.09
6.57
0.78
8.12
8.47
30.93
PanCLIPSSe (Ours)
36.15
39.07
37.55
32.51
12.00
2.75
16.37
18.80
50.71
Part: "leg"
bird
Cow
dog
shoop
person
CLIPPSeg [32, 46]
19.61
38.62
27.
39.14
52.63
52.67
PanCLIPSeg (Ours)
31.12
44.82
63.78
41.55
54.73
55.35
4.3
Ablation Study
In this section, analyze the impact of each training loss ParCLIPSee, We focus the roles of the separation and enhancement losses, examining how they contribute to improved segmentation accuracy.
Separation Enhancement Losses. We conducted an ablation study to investigate the effect the separation loss Leep and the enhancement loss Lesh on the performance of PartCLIPSeg in Table 5. Pascal-Part-116. eliminating both losses resulted and harmonic mean of 38.20 in Oracle-Obj. Introducing Leep without Lanh improved the harmonic mean in both Pred-All and Oracle-Obj setups. Using both losses led to the highest harmonic means of 30.67 and 38.79, respectively. Similarly, for ADE20K-Pas P244-234, employing both losses resulted in the
lower harmonic mean of 29.20 Pred-All

---

BBC
(a) Ground-truth
(b) VLPart [40] (c) CLIPSeg [38, 46] (d) CAT-Seg [11, 46] (e) ParCLIPScg
Figure 6: Qualitative results of zero-shot part segmentation on Pascal-Part-116 166 in Oracle-Obj setting.
best performance, with harmonic means of 19.63 in Pred-All and 38.60 in Oracle-Obj These results highlight the importance of both separation and enhancement losses in improving performance.
To verify the effectiveness of boundary creation of PartCLIPSeg, we examined an additional qualita- tive metric, Boundary IoU [8]. The results demonstrated high Boundary IoU performance, confirming that PartCLIPSeg effectively resolves ambiguous boundary issues as shown in Table 6.
Impact of PartCLIPSeg for Underrepresented Parts. We investigate the effect of the enhancement loss Lenh OVPS model performance, especially with respect to underrepresented parts. In Table 7, we compare our PartCLIPSeg with CLIPSeg [32, 46] on small parts such "eye", "neck", and "leg" of animals in Pascal-Part- 116. As shown in the table, PartCLIPSeg consistently outperforms CL.ISSeg with significant improvements in most cases. Notably, there is an impressive performance increase of 35.93%p for "cow's 's leg". These improvements highlight the effectiveness of the enhancement loss in accurately segmenting small and intricate parts, demonstrating its crucial role in improving overall performance.
5 Conclusion
In this study, we introduced PartCLIPSeg, state-of-the-ar OVPS method that addresses three primary challenges in OVPS. PartCLIPSeg utilizes generalized parts and object-level guidance to effectively solve identification issues. Then, it separates parts by minimizing their overlaps attention maps, thus learning ambiguous part boundaries. Additionally, we implemented enhancement loss function to improve the detection of underrepresented parts. Through extensive experimentation, we have confirmed the superior performance of PartCLIPSeg.
10

---

Acknowledgements
This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the MSIP (NRF-2022R CATERRAACC20 A2C301 1154, RSS-023-002199111 KEIT grant funded by the Korean government (MOTIE) (No. 2022-0-00688 No. 2022-0-01045), the HITP grant funded by the Korean government (MSIT) (No. 2021-0-02068 Artificial Intelligence
Innovation Hub, RS- RS-2019-11190075 Artificial Intelligence Graduate School Program (KAIST)), and Samsung Electronics Co., Ltd (I0230508-06190-00)))
Bibliography
[1] Aishwarya Agarwal, Srikrishna Karanam, KJ Joseph, Apoorv Saxena, Koustava Goswami, and Balaji Vasan Srinivasan. A-star: Test-time attention segregation and retention for text-to-image synthesis. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2283-2293, 2023.
[2] Juan Antonio Balbuena, Raúl Míguez-Lozano, and Isabel Blasco-Costa. Paco: a novel procrustes application cophylogenetic analysis. PloS one, 8(4):e61 2013.
[3] Zhipeng Bao, Yijun Li, Krishna Kumar Singh, Yu-Xiong Wang, and Martial Hebert. Separate- and-enhance: Compositional finetuning for text-to-image diffusion models. In ACM SIGGRAPH 2024 Conference Pupers, pages 1-10, 2024.
[4] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. Advances in Neural Information Processing Systems, 32, 2019.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[6] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions Graphics (TOG), 42(4))1-10, 2023.
[7] Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1971-1978, 2014.
[8] Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C Berg, and Alexander Kirillov. Boundary iou: Improving object-centric image segmentation evaluation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15334-15342, 2021.
[9] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems, 34: 17864-17875, 2021.
[10] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290-1299, 2022.
[11] Seokju Cho, Heeseong Shin, Sunghwan Hong, Seungjun An, Seungjun Lee, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. arXiv preprint arXiv:2303.11797, 2023.
[12] Subhabrata Choudhury, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Unsupervised part discovery from contrastive reconstruction. Advances in Neural Information Processing Systems, 34:28 104-28118, 2021.
[13] Daan de Geus, Panagiotis Meletis, Chenyang Lu, Xiaoxiao Wen, and Gijs Dubbelman. Part- aware panoptic segmentation. In Proceedings of the IEEF/VE Conference on Computer Vision and Pattern Recognition, pages 5485-5494, 2021.

---

[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 leee, 2009.
[15] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmenta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11583- 11592, 2022.
[16] Zheng Ding, Jieke Wang, and Zhuowen Tu. with maskelip. arXiv preprini arXiv: 08984, 2022.
universal image segmentation
[17] Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature-wise transformations. Distill, 3(7)e1 11, 2018.
[18] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open-vocabulary image segmen- tation with image-level labels. In European Conference on Computer Vision, pages 540-557. Springer, 2022.
[19] Xiuye Gu, Tsung-Yr Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2 13322, 2021.
[20] Kunyang Han, Yong Liu, Jun Hao Liew, Henghui Ding, Jiajun Liu, Yitong Wang, Yansong Tang. Yujiu Yang, Jiashi Feng, Yao Zhao, al. Global knowledge calibration for fast open-vocabulary segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 797-807, 2023.
[21] Ju He, Shuo Yang. Shaokang Yang, Adam Kortylewski, Xiaoding Yuan, Jie-Neng Chen, Shuai Liu, Cheng Yang. Qihang Yu, and Alan Yuille. Partimagenet: A large, high-quality dataset of parts. In European Conference on Computer Vision, pages 128-145, Springer, 2022.
[22] Ju He, Jieneng Chen, Ming-Xian Lin, Qihang Yu, and Alan I. Yuille. Compositor: Bottom-up clustering and compositing for robust part and object segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11259-1128,, 2023.
[23] Wei-Chih Hung, Varun Jampani, Sifei Liu, Pavlo Molchanov, Ming- Hing-Hsuan Yang, and Jan Kautz. Scops: Self-supervised co-part segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 869-878, 2019.
[24] Chao Jia, Yinfci Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Lc, Kun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021.
[25] Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Woo Ha, and Jun- Yan Zhu. Dense text-to-image generation with attention modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7701-7711, 2023.
[26] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Yen Lo, et al. Segment anything. In Proceedings of the JEEF/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[27] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language- driven semantic segmentation. arXiv preprint arXiv:2201.03544 2022.
[28] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang. Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recognize anything at any granularity. arXiv preprint arXiv:2307.04477, 2023.
[29] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. vocabloccyylary semantic segmentation with mask-adapted clip. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 7061-7070, 2023.
12

---

[30] Yong Liu, Sule Bai, Guanbin Li, Yitong Wang, and Yansong Tang, Open-vocabulary segmenta- tion with comantic-assisted calibration. arXiv preprint arXiv:2312.04089. 2023.
[31] Zongdai Liu, Feixiang Lu, Peng Wang, Hui Miao, Liangjun Zhang, Ruigang Yang, and Bin Zhou. 3d part guided image editing for fine fine-grained object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 1336-11345, 2020.
[32] Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In P Proceedings of the IEEE/CVE conference on computer vision and paltern recognition, pages 7086-7096, 2022.
[33] Umberto Michieli and Pietro Zanuttigh. Edge-aware graph matching network for part-based semantic segmentation. International Journal of Computer Vision, 130(11):2797-2811, 2022.
[34] George Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11):
39-41, 1995.
[35] Tai-Yu Pan, Qing Liu, Wei-Lun Chao, and Brian Price. Towards open-world segmentation of parts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15392-15401, 2023.
[36] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
[37] Quynh Phung, Songwei Ge, and Jia-Bin Huang. Grounded text-to-image synthesis with attention refocusing. arXiv preprint 200.23066.5427, 2023.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 1748-87633 PMLR, 2021.
[39] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li. He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open- open-world models for diverse visual tasks. arXiv preprint 2tti:22001.145599 2024.
[40] Peize Sun, Shoufa Chen, Chenchen Zhu, Fanyi Xiao, Ping Luo, Saining Xic, and Zhicheng Yan. Going denser with open-vocabulary part segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15453-15465, 2023.
[41] Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar Gonzalez-France Diffuse. attend, and segment: Unsupervised seoo-shot segmentation using stable diffusion. arXiv preprint ar3iv:2308. 12469, 2023.
[42] Mehmet Ozgur Turkoglu, Alexander Becker, Hüseyin Anil Gündüz, Mina Rezaci, Bernd Bischl, Rodrigo Caye Daudt, Stefano D' Aronco, Jan Wegner, and Konrad Schindler. cilmeensemble: probabilistic deep learning via feature-wise linear modulation. Advances in neural information processing systems, 22222229-22222,, 2022.
[43] Robert van der Klis, Stephan Alaniz, Massimiliano Mancini, Cassio F Dantas, Dino Ienco, Zeynep Akata, and Diego Marcos. Pdisconet: Semantically consistent part discovery for fine- grained recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1866-1876. 2023.
[44] Zifu Wan, Yaqi Xie, Ce Zhang, Zhiqiu Lin, Zihan Wang, Simon Stepputtis, Deva Ramanan, and Katia P Sycara. Instructpart: Affordance-haseee part segmentation from language instruction. In AAAI-2024 Workshop on Public Sector I.MMs: Algorithmic and Sociotechnical Design, 2024.
[45] Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. In- stancediffusion: Instance-level control for image generation. arXiv preprint Xiv:22402.03299. 2024.
13

---

[46] Meng Wei, Xiaoyu Yue, Wenwei Zhang, Shu Kong, Xihui Liu, and Jiangmiao Pang. Ov-parts: Towards open-vocabulary part segmentation. Advances in Neural Information Processing Systems, 36, 2024.
[47] Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1206-1217, 2023.
[48] Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8256-8265, 2019.
[49] Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. Sed: A simple encoder- decoder for open- veee-vocayllary semantic segmentation. arXiv preprint arXiv:2311.15537, 2023.
[50] Jinheng Xie, Yuexiang Li, Yawen Huang. Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training- box-constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7452- 7461, 2023.
[51] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2955-2966. 2023.
[52] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In European Conference on Computer Vision, pages 136-553. Springer, 2022.
[53] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2945-2954, 2023.
[54] Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang Chich Chen. Convolutions die hard: Open- voeablocabulary segmentation with single frozen convolutional clip. Advances in Neural Information Processing Systems, 36, 2024.
[55] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14393-14402 2021.
[56] Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, and Antonio Torralba. Open vocabulary scene parsing. In Proceedings of the IEEE International Conference on Computer Vision, pages 2002-2010, 2017.
[57] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 633 633-641, 641, 2017.
[58] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pages 696-712. Springer, 2022.
[59] Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegelip: Towards adapting clip for zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11175-1111 185, 2023.
14

---

Supplementary Material
Table of Contents
Discussion
Appendix A
Limitations & Future Work
Appendix A.1
Social Impact
Appendix A.2
Experimental Details
Appendix B
Datasets Details
Appendix B.1
Implementation Details
Appendix B.2
Computational Resource
Appendix B.3
Additional Quantitative Evaluation
Appendix C
Additional Ablation
Appendix D
Impact of Object-Level and Part-Level Guidance
Appendix D.1
Qualitative Ablation on Attention Control Losses
Appendix D.2
Ablation on the Hyperparameter in Attention Control
Appendix D.3
Additional Qualitative Results and Qualitative Analysis
Appendix
CLIP Embedding
Appendix E.1
Additional Qualitative Results
Appendix E.2
A Discussion
A.1 Limitations Future Work
We share some limitations of our model and outline directions for future research. Our model is based on semantic segmentation, which does not allow for the discrimination individual parts as instances. Consequently, parts such "Paw I" from Dog I and "Paw 2" from Dog 2 are assigned the same label. We plan to address this limitation in our future work to enhance the model's capability to distinguish between similar parts from different instances.
Furthermore, we believe that adding more inductive biases related the relationships between parts, similar to key point detection which incorporates structural understanding, could yield higher-quality results.
Currently, our focus has been on object-specific parts, essentially mapping different granularity of vocabulary visually. Advanced methods could allow us more effectively handle broader variety of input categories, further enhancing our model's 's applicability and performance.
A.2 Social Impact
This study explores open- vocabulary part segmentation, technique that expands segmentation models to include ine-grained categories not encountered during training. The approach's robust nature allows for segmentation across various categories, proving invaluable for applications requiring flexibility and adaptability.
Open-vocabulary part segmentation could greatly influence several advanced fields. robotics, for example, robots can precisely identify and handle wide array of objects and components, essential for tasks from manufacturing assembly lines complex medical surgeries. This adaptability allows robots to function in new settings without extensive retraining.
In healthcare, this technology enhances diagnostic processes by allowing for the segmentation of novel anatomical structures in medical imaging. This could facilitate earlier disease detection by identifying subtle, non-cataloged abnormalities essential for diagnosis.
In image editing, open-vocabulary part segmentation enables sophisticated manipulation by letting editors modify image fine-grained components not predefined in their software. This is especially
15

---

beneficial in the creative industries, where precise adjustments can improve output quality and foster innovation.
Adopting open-vocabulary part segmentation promises to enhance the efficiency, accessibility, and effectiveness of these technologies, particularly in handling real-world variability and unpredictability.
B
Experimental Details
B.1 Datasets Details
B.1.1
Pascal-Part-116
In the Pascal-Part- 16 dataset [7, 46], we target the following object-specific category names in Table AI. Among these, "bird", "car", "dog", "sheep", and "motorbike" are designated as unseen categories, encountered for the first time during inference in the zero-shot part segmentation setting.
Table Al: List of object-specific classes Pascal-Part-116
Object-spocific Part Categories
acroplane's hody
acroplane's stemn aeroplane's wing
acroplane's tail bicycle's handlebar bead hird's leg
acroplane's engine
aeroplane's wheel
bicycle's is wheel
hicycle's suddle bird's tail
bicycle's is chainwheel
bicycle's headlight
bird's wing
bird's eye
hird's beak
bird's torso
hird's neck
bird's foot
bottle's hody
bottle's
bus's wheel bus's roof
bus heudlight
front
bus's side
's back
bus's mirroe
bus's license plate
bus's door
bas window
car's whoel
car's headlight
car's front plate
car's door
car's window
cat's tail
cat's bead
cat's eye
cat's
neck
leg.
cat's
cat's ear
cow't tail
cow't head
cow's eye
cow torso
cow's neek
cow's leg
cow's car
cow's muzzle
cow't born
dog's
dog's
dog's eye
dog's
dog's neck
dog's leg
dog's nose
dog's puw
dog's eur
dog's muzzle
horse's eyo
horse's
honse's leg
horse's eur
horse's 's muzzle
horse's 's hoof
matorbike's 's wheel
motorbike's saddle
motorbike's handlebar
motorbike's 's headlight
's head
person's eye
person's i torso
person's noek person's eyehrow
person's 's log
person's 's foot
person's nose
person's ear
person's 's mouth
person's 's hair
person's 's lower arm
person's upper arm shoep's head sheep's
pesson's hand sheep's is eye
pottcdplant' 's pot sheep's 's torso
pottsdppant" plant shoop's 's neok
shoop's
shoep's is leg.
sheep's muzzle train's is side
sheep's horn train's back
train's headlight
train's head
truin's 's front Ivmonitor's screen
train's 's roof
train's is couch
B.1.2 ADE20K-Part-234
In the ADE20K-Paa Part-234 dataset [57], we target specific object categories listed in Table A2. The dataset includes 44 object classes and detailed subdivisions into over 200 part categories. Notably, "bench", "bus", "fan", "desk", "stool", "truck", "van", "swivel chair", "oven", "ottoman", and "kitchen island" identified novel classes and are encountered for the first time during inference in zero-shot part segmentation setting.
B.1.3 PartImageNet
PartImageNet [21] is a dataset derived from ImageNet [14], consisting of approximately 24,000 images across 158 classes. Each class has annotations for parts. All classes belong one of 11 superclasses, organized using the hierarchical information from WordNet [34].
Previous opea-loca part segmentation research [40] primarily used PartImageNet to evaluate cross-dataset settings. In our study, we use PartImageNet not only for cross-datased evaluation but also to assess model performance in zero-shot settings specific to PartImageNet.
To measure more generalized performance, we select 40 classes out the 158. We maintain proportion of existing superclasses as much as possible. For each superclass, at least 50% of the categories are designated as seen categories, with the remaining being unseen categories. Therefore, there are 25 seen classes and 15 unseen classes in our PartImageNet evaluation dataset.
16

---

Table A2: List of object-specific classes ADE20K-Part-22
Object-ppcilic Part Calegories
person's
person's back
person's ' foot
person's gaze person's tanso
person's hand
person's
person's leg
person's neek
door't trame
door's s handle
dooe's s knob
doce's panel
clock's 's face cabinet's is door cabinet's skin
clock's frame cabinet's drawer cabinet's top sink's top
toilet'ss bowl
toilet's 's cistern
toilet's lid
cahinet's front
cabiner's shelf'
cabines's side
sink's faucet
is pedestal
sink's tap
lamp's arm
lamp's hise
lamp's comppy
lamp's colomn
lamp's is cord sconce's arm
lamp's ss highlight
lamp's 's light source
lamp's shude
lamp's tibe
sconce's backsplate
scance's highlight
light
sconce's shode
chair's apron
chair's arm
chair's hack
chair's
chair's leg.
is chest drowers's doce
chair's ' sent cushion
chair's 's skirt
chair's stretcher
chest drawens's 'ss apron
chest of drawers's is drawer
chest of drawers's front chundeller's chain bed's footboard table's s drower amnchair's apron armchair's
chest of drawers's log
chandelier's arm
chandelier's bulb
chandelier's canopy
chandelier's cord
chankliie's highlight bed's leg
clandelier's light hod's rail
chandelier's shode
headhoard
table's is spoon
table's leg
table's s shelf
table' 's top
thil's wheel
armehair's arm
armehair's ' back
arnchair's back pillow
armchair's
armehair's base
armehali's cushice
ottoman's back
ottorunn's log
ottomnnn's seadl swivel chair's hack
shelf's door
shell's drawer
shell's ' front
shelf's shelf
swivel chulr'ss hase
swivel chair's sear
swivel chair's wheel
fan's is blade
fan canopy
fan's tube
coffee table's aale's leg
coffee table's top
stoel's
stool's
sofa's
sofa's back
sola's buck pillow
sofa's leg
sost hase
sofa's cushion
sofa's skin
computer's computer case
computer's i keyboard
computer's monitor
computer's is monse
desk's is apron
desk't door
desk's 's drawer
desk's log
desk's shelf
desk's top
wardrohe's door
wardmbe's drawer
wandrohe's front
wandmohe'
wardmbe's minme
wardrohe's top
car's bumper
car's dour
car's headllggtt
car's hood
car's license platic
car's logo
car's mirror
car's wheel
car's window
car's wiper
has's bumper
bus's door
bus's headlight
bus's license plate
has's Ingo
bus's
bus's
window
bus's wiper
puncl
's door
oven's drawer
oven's top
cooking stove's humerr
cookinng stove's button panel
cookinng stove's door
cooking stove's drwwer
cooking stave's oven microwave's s sick
cooking stove's stove microwave's top
microwave's 's button panel
microwave's 's door
microwave's 's froni refrigerator's door
microwase's window
refingerattr'
panel
refrigeratoe's
refiigerator's
kitchen island's door
kiichen island's drawer
kitchen island's 's front dishwasher's 's skirt
kischen island's 's side bookcanse's 's door
kilehhen ishan's top
diskwasher's ' button panel
disloassher's 's handli
bookease's is drawer
bookcase's 's front
bookease's side
television receiver's 's hase Iclevision receivers speaker pool table's 's had van's headlight
relevision receiver's 's buttons
television receiver's Thame
tolevision receiver's keys
television receiver's soreen
glasss
glass's
glass' 's opening
glass's ' stem
pool tuble's log
pool luble's pocket
van't ' bumper
van's door
van's license plate
van't logo
van't is mirror
van's taillight
van's wheel
window
wiper
airplane's door
airplane's fuseluge
airplane's handing year
sirphane's 's propeller
airplane's stabilizer
airplane's turbine engine
airplane's 's wing
truck's 's bumper
tuck's door
truck's ' headlight
truck's 's license plate
truck's 's logo
truck's 's mirroe minibike's 's
tuck'ss wheel
truck's 's window
minitike's is license plate
minibike's mirror
minibike's is wheel arm tralliic light's pole light's light source
washer's buiton panel
washer's door
washer's frent
wadher's
bench's
beach's log
bench's 's scal
trallic light's 's housing light's is highlight
light's aperture
light's canopy
light's is diffussor
lightss shude
We conduct the dataset evaluation as follows: Models are trained training dataset composed of seen classes. Segmentation performance are then assessed on a validation dataset containing both seen and unseen classes. Evaluations were conducted in both Pred-All and Oracle-Obj settings.
Table A3: List of selected object classes per superclass. We choose 40 object classes from 158 categories to evaluate performance on PartImageNet and a cross-dataset setting. Object categories that are both underlined and in bold represent the unseen classes, which are emphasized for their unique characteristics within each superclass.
Superclass
Object Categories
Quadruped
tiger, giant panda, leopard, gazelle, ice bear, impala, golden retriever green mamba. Indian cobra green lizard, Komodo dragon, tree frog, box turtle, American alligator yawl, pirate, schooner
Snake
Reptile
Boat
Fish
barracouta, goldfish. killer whale, tench albatross, goose, bald eagle garbage truck, minibus, ambulance, jeep. school bus mountain bike, moped, motor scooter gorilla, orangutan, chimpanzee beer bottle, water hottle, wine hottle warplane, airliner
Bird
Car
Bicycle
Biped
Bottle
Aeroplane
17

---

gyalohanns
Doldiions
Oadrianss boch
goldnish's tail
colonah's
pold 6I''
abldtiin's tall
(a) Ground Truth
(b) Result from ParCLIPSeg (Oracle-Ob))
Figure Al: Example of part annotations in PartImageNet on our experiment.
B.2 Implementation Details
Our model implementation is based on the CLIPSeg [32] architecture, as described in the OV- PARTS [46]. We utilized the pre-trained CLIP ViT-B/16 [38, 59] image encoder and text encoder for our experiments.
The model is trained using the ADAMW optimizer with base learning rate of 0.0001 over 20,000 iterations, with a batch size of images. We employ a WarmupPolyLR learning rate scheduler to manage the learning rate throughout the training process. To ensure model stability, we apply gradient clipping with a maximum gradient norm of 0.01.
We save model parameters every 1,000 iterations during training. The best-performing parameters are selected based on the highest validation evaluation scores. For example, the evaluation result on the Pascal-Pa BarccllPart-lll 16 dataset in the Oracle-Obj setting is derived from the checkpoint saved at the 5,000-step mark, which yields the best validation performance.
We evaluated several baseline methods -ZSegt, CLIPSeg [32], and CAT-Seg [1]]_wick] which are fine-tuned on our datasets. ZSSeg+ modified version of ZSseg [52], utilizing different fine-tuning methods according to [46]. It employs a ResNet-101 backbone and Compositional Prompt Tuning based on CoOp.
CLIPSeg and CAT-Seg models are pre-trained on object datasets; however, we fine-tuned these models each part-level dataset. CAT-Seg, based on ResNet-101 and using ViT-B/16 as CLIP's visual encoder, achieved comparable performance by computing cost volumes and subsequently applying cost aggregation- process that enhances segmentation by aggregating matching costs between image features. Specifically, CAT-Seg uses the frozen upsampling decoder but fine-tuned CLIP's image and text encoders. Conversely, fine-tune the CLIPSeg decoder to better identify small segments and define clear boundaries. CLIPSeg, based on the ViT-B/16 architecture, is fine- tuned on the visual adapter, text embeddings, and transformer decoder to enhance its segmentation capabilities.
B.3 Computational Resource
All our experiments are conducted on 8 x NVIDIA A6000 GPUs.
Table A4: Computational resources Pascal-
Part- 16 with batch size 8.
As shown in Table A4, PartCLIPSeg offers advan- tages in both the number of parameters and mem- ory consumption compared to other baselines on the Pascal-Part-11 116 dataset. With 152.4 million param- eters, it is more efficient than ZSSeg+ and CAT-Seg, and comparable to CLIPSeg. In terms of GPU mem- ory usage, PanCLIPSeg requires 24.4 GB, which lower than both CAT-Seg and CLIPSeg.
Method
Params
Memory
ZSSeg+ [52]
191.6
11.1
CLIPSeg [32, 46]
151.7 M
25.5
CAT-Seg [11]
180.6 M
29,0
PanCLIPSeg
152.4 M
24.4
For PartCL.IS although the number parameters is larger than CLIPSee because computations
related to attention control, there is an advantage in not having maintain weights for each object-
18

---

specific part due to the use of generalized parts. These efficiencies become more pronounced as the number of generalized parts shared among object classes increases. By leveraging shared representations for generalized parts, PartCIIPSeg reduces redundancy and memory requirements. This makes our model particularly advantageous in datasets where object classes have many common parts, leading to more efficient training and inference without compromising performance.
C Additional Quantitative Evaluation
Table A5: Recall performance Pascal-Parts
Table A6: Recall performance ADE20K-Part 234 under the Oracle-Obj setting.
116 under the Oracle-Ohj setting.
Method
Unseen
Harmonic
Method
Seen
Unseen
Harmonic
ZSSeg+ [52]
65.47
32.13
ZSSeg+ [52]
55.78
40.71
47.07
CLIPSeg [32.46]
55.71
43.35
48.76
CLIPSeg [32.46]
49.59
48.11
48.84
CAT Seg [1]]
56.00
43.20
48.77
CAT-Seg [1]]
43.48
39.87
41.60
PanCLIPScg (wlo Lus Leah)
46.47
PanCLIPSeg Luv Luat)
51.64
50.99
51.31
ParCCLIPSeg (w/ Ceep + Lash) 58.46
47.93
52.67
ParCCLIPSeg tw/ Ceep Cuih)
53.31
51.52
52.40
In this section, we present an additional evaluation metric that focuses on specific challenges within the Open- Vooabulary Part Segmentation (OVPS) task as shown in Figure 2. The Recall metric is used to assess how well the model captures underrepresented parts, addressing the challenge of underrepresented parts. Higher values in recall indicate that the model effectively captures these seldom-occurring parts, thereby addressing the challenge of underrepresented parts in OVPS.
ParCLLIPSeg consistently achieves higher recall both seen and unseen classes across both datasets as shown in Tables A5 and A6. The improved harmonic mean indicates that our model more effective at identifying underrepresented parts, thereby addressing one of the core challenges in OVPS.
We further analyze the impact of the attention control losses Laep and Lunh on the recall. By comparing the recall metric with and without these losses, we assess their effectiveness in enhancing the representation of seldon the attention control losses enhances the model's performance on unseen classes, which often include underrepresented parts. The increases in harmonic the model to better capture these seldom -ccuurinn or small parts.
parts. From Tables A5 and A6, we observe that incorporating
suggest that the attention control losses help
D Additional Ablation
D.1 Impact of Object-Level and Part-Level Guidance
Table A7: Ablation on Aooj. Apart, and attention control on Pascal- 1 16 in Oracle-Obj setting.
Aabj
Apart
Lzop + Lanh
Seen
Unseen
Harmonic mloU
0.0
0.0
48.36
29.42
36.58
1.0
0.0
V
48.61
31.28
38.07
0.0
1.0
48.94
31.68
38.46
1.0
1.0
x
49.09
31.26
38.20
1.0
1.0
50.02
31.67
38.79
We conduct additional experiments to verify the impact of object-level and part-level label guidance on model performance as shown in Table A7. Specifically, we vary the weights λobi and Apart in Equation (5), setting each to or 1, to assess the influence of object-level and part-level supervision on the overall performance. Additionally, we evaluate the effect of the attention control losses, Czep and Lenh. by including or excluding them.
As shown in Table A7, both object-level and part-level guidance positively impact model performance on the Pascal-Part-11 16 dataset under the Oracle-Obj setting. When neither object-level nor level supervision is applied, the harmonic mean is 36.58. Introducing object-level guidance alone increases the harmonie mean IoU to 38.07, while part-level guidance alone raises 38.46. Combining both guidances yields the best performance with harmonic mean IoU of 38.79.
19

---

Sheep's 5 car
sheep's s car
sheep muzzic
Sheep's muzzic
Eneep's RESO
Sheep's head
Sheep's torso torsec
Sheep's sheep's
sheep's S leg
Sheep's leg
sheep's beck
sheep's
neck
Figure A2: Comparison of results using only Leep (top) with both Leep and Lenh (bottom). The heatmap illustrates attention activation for the "sheep's 's neck" class.
Table A8: Effect of varying threshold y Pascal-Part. I 116 in Oracle-O0 Obj setting.
Threshold (7)
Seen
Unseen
Harmonic mloU
0.1
47.34
32.24
38.35
0.2
47.45
32.20
38.37
0.3
50.02
31.67
38.79
0.4
51.10
31.18
38.73
0.5
48.71
31.16
38.01
Additionally, removing the attention control losses Leep and Laab while keeping both Xaos and Apart at 1.0 results in lower Harmonic mean of 38.20. This indicates that the attention control losses contribute better distinguishing between seen and unseen classes.
D.2 Qualitative Ablation on Attention Control Losses
The separation loss reduces the overlap between different parts, while the enhancement loss strength- ens the activation of underrepresented parts. As shown in Figure A2, when only the separation loss Leep is applied (top), smaller parts adjacent to larger parts may be diminished. Specifically, "sheep's neck" is not properly highlighted because minimizing the intersection as the "sheep's 's torso" and "sheep's 's head", to overshadow smaller ones. When both losses Leep and Lanh utilized (bottom), the model accurately segments the small part-"sheep's neck"-as the enhancement loss boosts its representation, preventing it from being overwhelmed by larger neighboring parts.
cause larger parts, such
This demonstrates that the separation and enhancement losses complement each other. Their combined use is essential to effectively distinguish and represent both large and small parts within object, leading to improved segmentation performance.
D.3 Ablation on the Hyperparameter in Attention Control
To evaluate the sensitivity our method to the hyperparameter threshold y in Equation (8), we conducted experiments the Pascal-Part-I1 16 dataset under the Oracle-Obj setting, We varied from 0.1 to 0.5 and measured the performance terms of mloU for seen and unseen classes, as well as the harmonic mean.
As shown Table A8, our method robust the choice of within range of 3.1 to 0.5. The harmonic mean remains relatively stable, with the best performance achieved at 7 0.3. While there slight variation in performance across different values of 7, the changes are not significant, indicating that our method does not heavily depend on the exact value of this hyperparameter.
20

---

E
Additional Qualitative Results and Qualitative Analysis
E.1
CLIP Embedding
zation Embeddinng
Shwap's oar SW dog le ou
shoup's ell scti a
poctodslonts pot pottelpanne pwnnt
20
row Shopp
europlane start oorbherss utig Ter torig
CoRFERALTTA resatte eye cat's Tood
ChoepHe hoad Stea grek
30-
baltl 1.00 bentl
disie heed
witts sen drgerpa
pereptinets ergane
Ned's hoad
berry reck birT
Led's tox.
pyyters masreneet tkyclellwwo wild
bed' eyt peftras
po RRWSttaa
"wp's
bu bu99iiibeess
parooy PEESS orm.
1016
BONARES A 13
persor's Rig
Scans diate
cat's beso
per eyebrow
deg'
Wath Ce
Traiess headligt
snoap
ovep 12
sindzaarcio Bus's bock
brar's
hend
drs
tsxentchst trant ul
curs St2
Figure A3: The t-SNE visualization of text embeddings from a pre-trained CLIP model on the classes of the Pascal-Part 1 16 dataset.
The t-SNE visualization of text embeddings from a pre-trained CLIP [38, 32,58) 58] model on the Pascal- Part-1 16 dataset [7, 46] reveals intriguing insights into the model's understanding of categories. Notably, similar classes such as "cats" and "dogs" are clustered closely within the embedding space. This proximity indicates shared semantic space for categories that are visually or contextually related.
Additionally, observed that opject-specific parts sharing generalized parts, such as "car's license plate" and "bus's 's license plate", are also positioned near each other. This clustering suggests that the CLIP recognizes and leverages common parts across different objects that share common characteristics. Further analysis shows that object-specific classes containing parts like "muzzle" and "paw" are distributed in similar regions of the space. This consistency across different object categories emphasizes the CLIP's ability to generalize part-level features effectively.
Leveraging CLIP's text embeddings provides significant zero-shot capability in the visual domain. This capability can be extended to part-level categories, demonstrating the potential for sophisticated unsupervised zeoo learning approaches in fine-gra tineggranedd object and part recognition tasks.
21

---

E.2 Additional Qualitative Results
E.2.1 Oracle-Obj Setting
NXT:
Figure A4: Comparison of VLPart, CLIPSeg, CAT-Seg, and our model on the Pascal-Part-11 16 dataset in Oracle-Obj setting.
22

---

E.2.2 Pred-All Setting
00
VIN
3P
1
AVT: ArTLiiPg
Figure A5: Comparison of VLPart, CLIPSeg, CAT-Seg, and our model on the Pascal-Part 1 16 dataset Pred-Al All setting.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's 's contributions and scope in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations & future works are included at supplementary materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The anonymized repository our implementation and instructions of repro- duction provided. (in the abstract section)
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification: The anonymized repository of our implementation and instructions of repro- duction are provided. (in the abstract section)
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.cm co/
oolccycylyy for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips.ccoooccc wwtpppccccgeeeeeeecceccccceeeeee geggguu/oodesssie peteeuummssiooPooii piionnolcyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental setting and details are provided in section 4 and supplemen- tary materials. Also, the anonymized repository of our implementation and reproduction instructions are provided. (URL is in the abstract section)
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: The standard error of an average of 5 results is reported in Section 4 of the proposed model.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The experimental resources, setting and details are provided in section and supplementary materials. Also, the anonymized repository of our implementation and reproduction instructions are provided. (URL the abstract section)
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / /t/.risuuppppp/e wttpppi//ttiiaaaiiiiiiaeeaaaaaee
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader of impact is discussed in supplementary materials.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credited previous works and codes in section 4.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi wwwode.hho....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The anonymized repository our implementation and reproduction instruc- tions are provided. (URL is in the abstract section)
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines: [NA]
The answer NA means that the paper does involve crowdsoureing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "5GzWKcwShW", "reviewer": "Reviewer_KLx2", "summary": "This paper identifies three key problems of current open-vocabulary part segmentation, namely, lack of generalization, ambiguous boundaries, and missing underrepresented parts. The paper then proceeds to propose solutions to fix these problems. Experimental results show that the proposed method outperforms previous state-of-the-art methods.", "questions": "1. What is \"incomplete guidance from knowledge-based, multi-granularity characteristics of parts\"? It is unclear from the texts. It seems like an important motivation for the design of the attention control. Some more analysis or discussion would help here.\n\n2. In Table 1 and Table 3, why are the performance of PartCLIPSeg on \"seen\" classes so low? From the model design, I don't see a reason for this. Do the authors have an explanation? \n\n3. How to understand the \"failures\" in Table 6 where PartCLIPSeg cannot improve the performance from the CLIPSeg (such as \"bird's eye\") or PartCLIPSeg also has very low performances (such as \"sheep's eye\", \"cow's neck\")? And are all these parts indeed *small*? For example, \"cow's leg\" doesn't seem to be small to me. How do the authors define what are small parts?", "limitations": "The authors have addressed the limitation and potential societal impact of the work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper identified some key issues in current open-vocabulary part segmentation and proposed effective solutions respectively. The paper was easy to follow, with good figures to demonstrate key concepts. The experiments also demonstrate that the methods can lead to nontrivial gains.", "weaknesses": "1. **Limited discussion of previous open-vocabulary part segmentation work and their relationship to this work.** For example, I quickly searched and read the VLPart paper [1], which seems to parse objects first, then parts, thus avoiding the object-level misclassification issue (or the \"lack of generalization\" as named by the authors). Indeed, the VLPart seems very natural to me. So I am actually surprised to find that there is such an issue (mixing objects and parts during classification) as indicated by this paper. In general, the related work section provides very limited introduction of the previous open-vocabulary part segmentation methods, esp. how they are related to the proposed approach. There are also **no descriptions of the compared baselines in the experiment section**.This made readers hard to make a fair assessment of the contribution of this paper.\n\n2. **Limited ablation studies to understand the proposed approach.** Some key questions are unanswered: how much does each proposed component (object-level context, attention control) contribute to the performance gain? In what way or what types of data? It might also be helpful to compare the Pred-Obj performance because that helps disentangle the localization and classification ability of the approaches. Current comparison mixes many components together and it's difficult to gain insights about how the gain is achieved. \n\n3. **Lack of addressing directly the identified key problems**. The paper starts with three key issues and then propose different ways to fix them. However, in the experiments, there are no direct addressing to these problems any more (except Table 6 for small parts). It would be great if the experiments can clearly demonstrate how the three problems are mitigated by the proposed approaches. \n\n4. **Lack of discussion on the training details in the paper**. For readers who are not familiar with CLIPSeg, it is impossible to tell how the model is trained and what datasets are used for training. There is also **a lack of descriptions of the compared baselines in the paper**. Without these information, it's hard to understand the performance gains.\n\nMinor issues: \n\n1. I found it confusing to call it \"lack of generalization\" when the dog's parts are misclassified as sheep's or cat's. Isn't it just misclassification, or object-level misclassification?\n\n2. Texts on the figures are way too small, esp. in Figure 2 and Figure 4.\n\n3. The \"supplementary material\" for this paper should be put as an appendix to the main paper file, instead of as a separate file.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and suggestions.\n\n## **Weakness 1**: Discussion of Previous OVPS\n\nWe have now included a performance comparison with VLPart (ICCV 2023).\n\nVLPart is a pioneering study in Open-vocabulary Part Segmentation (OVPS).\nVLPart focuses on instance segmentation with masks, whereas our research compares semantic segmentation methods like ZSSeg, CATSeg, and CLIPSeg. \nThis distinction also led to the omission of a direct comparison in the OpenReview of OV-PARTS (NeurIPS 2023).\n\nHowever, based on the feedback, we recognize that a performance comparison would enhance the overall delivery of context.\nTherefore, we conducted experiments under the same conditions to compare VLPart with our method.\n\nOur results show that PartCLIPSeg outperforms VLPart in both the Pred-All and the Oracle-Obj settings.\nVLPart is based on a Mask R-CNN model; it may encounter errors at the object-level classification (in the Pred-All setting).\nEven if it detects parts within an object, the overall performance can be compromised if the object-level detection itself is not accurate.\nWe have included the qualitative and quantitative evaluation results in Table R1 and Fig. R3.\n\n(Due to space limitations with other questions, we kindly ask you to refer to the response to Reviewer LDit's Weakness 2 for additional information.)\n\n## **Weakness 2**: Ablation Studies\n\nAdditional ablation studies related to object-level context and attention control were conducted.\nThe experiments include various object-level and part-level guidance with and without attention control.\nThis allowed us to analyze the influence of object-level guidance, part-level guidance, and attention loss through their respective coefficients.\n\nDetailed information is provided in Table R2.\n\n(Due to space limitations with other questions, we kindly ask you to refer to the response to Reviewer LDit's Weakness 1.1 for additional information.)\n\n## **Weakness 3**: Addressing of Key Problems\n\nBased on the precious feedback, we have included additional qualitative assessments to demonstrate how each issue has been resolved in Fig. R1.\n\nAlso, we conducted additional quantitative studies on boundary IoU(Intersection of Union) as in Table R3.\n\n(Due to space limitations with other questions, we respectfully refer you to the response to Reviewer LDit’s Weakness 1.2 for detailed information.)\n\n\n## **Weakness 4**: Training Details and Baseline Descriptions\n\nWe add more explanations regarding our training details and CLIPSeg, as well as descriptions of the baseline methodologies: ZSSeg+ and CATSeg in the main text.\n\nCLIPSeg: Extends a frozen CLIP model with a transformer-based decoder. It operates with both text and image prompts, using skip connections from the CLIP encoder to the decoder and modulating the decoder's inputs using FiLM conditioning. Through this, CLIPSeg inherits CLIP’s zero-shot ability and can be fine-tuned to different downstream tasks.\n\nOur method utilizes these CLIPSeg pre-trained on the PhraseCut dataset and frozen CLIP. We fine-tuned the model respectively for each experiment using the training set of datasets.\n\n## **Minor issues**\n\nRegarding incorrect object-level prediction, it can indeed be seen as misclassification.\nHowever, we refer to it as a “lack of generalization” because it indicates an insufficient disentanglement of object-level and part-level information.\n\nMoreover, the misclassification in the unseen class suggests that the model does not fully generalize the holistic understanding and contextual information from the seen classes to the unseen classes. We will clarify this in the revised version.\n\nThank you. We made the corrections for minor issues 2 and 3.\n\n## **Question 1**: Characteristics of Parts\n\nOur intention was to convey that, unlike object-level segments, parts may not have clear, well-defined boundaries that may be defined by human convention or knowledge, making perfect guidance challenging.\nFor instance, the \"head\" may include the neck or only the face.\nWe highlight that supervision can be inherently ambiguous since the boundaries of parts are subjective and competitive, based on agreed terminology.\n\nThis challenge motivates the design of our attention control mechanism.\nInstead of relying solely on ambiguous supervision, we use common knowledge that parts should not overlap significantly.\nOur approach focuses on reducing the overlap between parts while simultaneously enhancing underrepresented parts. We have further clarified this in the main text.\n\n\n## **Question 2**: Performance on \"Seen\" Classes\n\nPartCLIPSeg is designed to improve generalization by focusing both on object-level context and generalized parts, which significantly enhances performance on unseen classes. However, the emphasis on generalization can sometimes lead to sub-optimal performance in seen classes.\n\n(Due to space limitations with other questions, we respectfully refer you to the response to Reviewer Vaxb's Weakness 2 for detailed information.)\n\n## **Question 3**: Understanding Failures in Small Parts\n\nWe define a small part as any part that occupies less than 20% of the object mask. We have added this definition for clarity.\n\nOur model, along with other OVPS methods, faces challenges in accurately predicting small parts. Specifically, for unseen classes such as \"bird's eye,\" the proposed model still exhibits lower prediction accuracy due to the limited resolution of the attention map.\n\nWe have included detailed information on the proportion of object-specific parts, like the “cow’s leg”, relative to the object-level mask in Figure R4. This provides additional context on the size of these parts in relation to the entire object.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "anMS2Ws4Yc", "reviewer": "Reviewer_LDit", "summary": "This paper proposes PartCLIPSeg, which builds upon the CLIPSeg model and extends it to handle the unique challenges of part segmentation. This paper introduces several components to address the challenges in OVPS.\n\n1. Generalized parts with object-level contexts: This approach combines generalized part information with object-level context to mitigate the lack of generalization in fine-grained parts.\n\n2. Attention control: This method introduces competitive part relationships and attention control techniques to minimize overlap between predicted parts and enhance activation for underrepresented parts. This mechanism is a new addition to the field.\n\nThe paper conducts experiments on OVPS’s baseline datasets. The result shows that PartCLIPSeg outperforms existing SOTA methods in most settings. The Ablation primarily focuses on the impact of the proposed attention control loss, which lacks a comprehensive analysis of other key components of this paper.", "questions": "See Weakness.", "limitations": "No societal impact.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Introducing generalized parts with object-level contexts: The paper proposes an approach that combines generalized part information with object-level context to enhance the model’s ability to generalize to unseen object-part combinations.\n- Developing an attention control mechanism: The paper introduces an attention control mechanism which helps to address the issues of ambiguous part boundaries and missing small or infrequent parts.\n- Promising result on baseline benchmark: The paper demonstrates state-of-the-art performance in most settings of OVPS.", "weaknesses": "- Insufficient ablation studies: The ablation studies primarily focus on the proposed attention control loss, which is not sufficient to prove the effectiveness of the proposed method. \n     1) The impact of combining object-level and part-level pseudo-label is not explored. \n     2) The paper does not provide sufficient evidence that the proposed losses work as intended, beyond their impact on the final metric. Further discussion is needed to validate whether the losses effectively address the challenges they were designed to tackle.\n- Limited comparison with other OVPS methods: The paper compares PartCLIPSeg primarily with CLIPSeg and CATSeg, which are designed for open-vocabulary segmentation and not specifically for OVPS. The comparison with other OVPS methods is lacking.\n- Limited discussion on computational complexity: Since the paper introduces a new attention control mechanism, a detailed analysis of the computational requirements and speed trade off could be helpful for a more comprehensive understanding of the proposed method.\nInadequate qualitative analysis: Insufficient exploration and insights of the model’s performance on challenging and failure cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and suggestions.\n\n## **Weakness 1.1**: Impact of Object-Level and Part-Level Labels\n\n\nWe conducted additional experiments to verify the impact of object-level and part-level pseudo-labels.\nWe varied $\\lambda_1$ and $\\lambda_2$ in Equation 5, setting each to 0 and 1 in turn, to assess the influence of each guidance on the overall model performance.\n\nTable R2 shows that reducing the weight of the object-level loss ($\\lambda_1$) improved performance in the Oracle-Obj setting (with provided object-level masks). On the other hand, increasing the weight of object-level loss enhanced performance in the Pred-All setting. This indicates the advantage of focusing on part information prediction when object-level masks are available (Oracle-Obj setting).\n\n\n## **Weakness 1.2**: Validation of Addressing Challenges\n\nWe initially integrated the solutions to the three challenges into the main body of the paper.\nHowever, we realized that this approach compromised clarity.\nBased on the precious feedback, we have included additional qualitative assessments to demonstrate how each issue has been resolved in Fig. R1 of the attached PDF.\n\nAlso, we conducted additional quantitative studies on boundary IoU [C1] to validate the accuracy of the boundaries as shown in Table R3.\n\nOverall (in addition to Fig. R1)\n\n* (a) Lack of generalization:\n    * Fig. 5, row 3 shows improvements in detecting areas that models like CLIPSeg struggled with in the Pred-All setting, indicating enhanced generalization performance.\n* (b) Ambiguous boundaries:\n    * Fig. 5, row 1 illustrates how competitive part segmentation helps mitigate ambiguous boundaries.\n    * Additional metric mean boundary IoU (in Table R3) verified the superiority of PartCLIPSeg.\n* (c) Missing underrepresented parts:\n    * Fig. 5, row 2, and Fig. 4 highlight improvements in the segmentation of underrepresented parts.\n    * We also have detailed this in Section 4.3 through an ablation study.\n    * Fig. R4 demonstrates that our method helps identify small part classes.\n\n\n[C1] Cheng, Bowen, et al. \"Boundary IoU: Improving object-centric image segmentation evaluation.\" CVPR 2021.\n\n\n## **Weakness 2**: Comparison with Other OVPS Methods\n\nWe have now included a performance comparison with VLPart (ICCV 2023) [C1].\n\nVLPart is a pioneering study in OVPS that uses a Mask R-CNN-based approach leveraging the DINO feature for correspondences.\n\nNotably, VLPart focuses on instance segmentation with masks. In contrast, our research compares semantic segmentation methods like ZSSeg, CATSeg, and CLIPSeg, following the recent OVPS research [C2]\nThis distinction also led to the omission of a direct comparison with VLPart in the OpenReview of OV-PARTS (NeurIPS 2023) [C2].\n\nBased on the feedback, we recognize that a performance comparison would enhance the overall understanding and context.\nTherefore, we conducted experiments under the same conditions to compare VLPart with our method.\nWe adapted the mask outputs of VLPart, which uses mAP-based evaluation, by converting them to the semantic segmentation label with the highest pixel-wise confidence.\n\nIt's important to note that VLPart's original experiments on Pascal-Part only included dog and bus as unseen classes, while our current experimental setup on Pascal-Part-116 (following OV-PARTS) includes bird, car, dog, sheep, and motorbike. Therefore, we re-trained VLPart on Pascal-Part-116 to ensure a fair comparison with our method.\n\nAs shown in Table R1, our results show that PartCLIPSeg outperforms VLPart in both the Pred-All setting and the Oracle-Obj setting.\nThis is because VLPart relies on the nearest seen class when predicting unseen classes, whereas our model understands disentangled object-level and part-level contexts and utilizes attention control.\nWe believe leveraging the nearest seen class can be disadvantageous when the number of unseen classes increases, especially if these unseen classes are unrelated or distant from the seen classes.\n\nThe qualitative results can be found in Fig. R3.\n\n[C1] Sun, Peize, et al. \"Going denser with open-vocabulary part segmentation.\", ICCV 2023.\n\n[C2] Wei, Meng, et al. \"OV-PARTS: Towards open-vocabulary part segmentation.\", NeurIPS 2023.\n\n\n## **Weakness 3**: Computational Requirements\n\n**[Computational complexity]** For PartCLIPSeg, although the parameters increase due to the computations related to attention control, there is an advantage in not having to maintain weights for each object-specific part due to the use of generalized parts. Detailed information regarding parameters and memory is provided in Table R4.\n\n**[Challenging and failure cases]** PartCLIPSeg uses ViT-based CLIP, and since the proposed attention modulation is dependent on the resolution of the attention map, it has limitations on achieving optimal performance for overlapping small parts.\n\n\n## **Limitation**\n\nThe section related to societal impact was added to the supplementary materials (Ln 30 - 47). Although this is still in its early stages and immediate applications are limited, we aim to present a foundation for future applications in areas such as autonomous robot guidance, image editing, and fine-grained understanding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qfSE1xgE33", "reviewer": "Reviewer_Vaxb", "summary": "This paper proposed PartCLIPSeg for open-vocabulary part segmentation. This framework leverages generalized parts and object-level contexts for generalization in fine-grained parts, integrating competitive part relationships and attention control techniques. Extensive experiments on three datasets demonstrate the superior performance of PartCLIPSeg.", "questions": "How is the threshold γ in Equation 7 determined? How does this threshold affect the results?", "limitations": "The unseen categories of the three datasets in the experiment may have been seen during the CLIP pre-training, and we want to explore how well PartCLIPSeg generalizes to the “truly” unseen categories that have not been seen in the pre-training or from other fields", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper is well-organized, good writing and easy to understand.\n2) The topic of open-vocabulary part segmentation is meaningful and underexplored.\n3) Significant improvement compared to the baseline methods.", "weaknesses": "1) Figure 3 (overall architecture) can be refined to clearly emphasize the contribution. \n2) The analysis and explanation of the experiment are not sufficient. Although the method achieves a significant improvement on unseen categories, the results of seen categories on the Pred-All setting of the three datasets show different trends. Specifically, the results of the seen categories show a decrease compared to the baseline in Tables 1 and 3 (There exists an annotation mistake (41.70).), and a significant improvement in Table 2. Please discuss this result.\n3) Ablations of loss functions. We notice that adding some loss function in some cases may lead to suboptimal results and show different trends on different datasets. For example, the second column of the Pred-All setting in Table 5. Please discuss this difference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and valuable suggestions.\n\n## **Weakness 1**: Fig. 3 Refinement\n\nWe refined Fig. 3 to make it clearer and to effectively emphasize our contribution.\nSpecifically, we had made modifications to distinguish between the modules involved in training, fine-tuning, and the frozen.\nAdditionally, we have delineated the data and operational components and added dotted lines to expand the explanation, highlighting our contribution more clearly.\n\n## **Weakness 2**: Discussion in Experiment Results\n\n\nThank you for pointing out the annotation mistake. We have corrected it.\n\nPartCLIPSeg enhances generalization ability by disentangling object-specific part guidance into object-level context and generalized parts. This improves performance for unseen classes compared to previous methods. However, this focus on generalization may cause some performance degradation for seen classes.\n\nPrevious approaches often overfit to seen classes, resulting in high performance on seen classes but low performance on unseen classes.\nOn the other hand, our approach leverages generalized information extracted from supervised seen classes, which can sometimes reduce performance on seen classes.\n\nDifferent trends depending on datasets: In ADE20K-Part-234, which contains many diverse and small objects (as shown in Fig. R5), the use of object-level priors enhances generalization performance.\nThis is because directly predicting parts smaller than the small objects is much more challenging.\nHowever, in Pred-All setup of Pascal-Part-116 and PartImageNet, can lead to sub-optimal outcomes compared to directly using only object-specific part information if the object-level prior is noisy.\n\nNonetheless, this framework allows us to leverage advanced methods for acquiring the object level prior, as demonstrated by gains in the Oracle-Obj setting.\nSpecifically, modern approaches like SAM [C1], BEiT [C2], and EVA [C3], which excel in object-level predictions can support the validity of our approach.\nSince these models improve the accuracy of object-level prior, they can mitigate the performance degradation in the seen classes.\n\nFuture research on OVPS can explore methods to maintain performance on seen categories while improving performance on unseen categories.\n\n\n[C1] Kirillov, Alexander, et al. \"Segment anything.\" ICCV 2023.\n\n[C2] Bao, Hangbo, et al. \"Beit: Bert pre-training of image transformers.\" ICLR 2022\n\n[C3] Fang, Yuxin, et al. \"Eva: Exploring the limits of masked visual representation learning at scale.\" CVPR 2023.\n\n\n\n\n\n\n## **Question 1**: Threshold γ\n\nThe threshold was set empirically, and we have included the evaluation results for other values (0.1, 0.2, …, 0.5) as in the below table. It shows that our method is robust to the choice of threshold. We will add more detailed information about gamma in the main text and supplementary materials to provide further clarity.\n\n(Pascal-Part-116 / Oracle-Obj / mIoU)\n\n|threshold (γ)|seen|unseen|harmonic mean|\n|:-:|:-:|:-:|:-:|\n|.1|47.34|32.24|38.35|\n|.2|47.45|32.20|38.37|\n|.3|50.02|31.67|38.79|\n|.4|51.10|31.18|38.73|\n|.5|48.71|31.16|38.01|\n\n## **Limitation**: Generalization to Truly Unseen\n\nPartCLIPSeg leverages the results of CLIP's pretraining, which, as you noted, presents a limitation in generalizing to categories not seen during CLIP's pretraining. This is a common limitation shared by methods across various open-vocabulary tasks, not just Part Segmentation. We agree that further research is needed to address the challenge of \"truly\" unseen categories in the future.\n\nThe potential future work to address the challenge of truly unseen categories could explore the utilization of one-shot training and visual correspondence methods. One-shot training offers the potential to extend the model's capabilities to new categories with minimal training examples. By incorporating visual correspondence, there is a possibility to enhance the model's adaptability and improve its performance in recognizing and segmenting unseen categories. This approach could mitigate the limitations of pretraining and enable more flexible and effective handling of truly unseen scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We sincerely appreciate all reviewers for their thorough and insightful feedback.
The valuable reviews have significantly enhanced the overall delivery of the proposed method.

We are particularly thankful that all reviewers (Vaxb, LDit, KLx2) found our paper show promising results and non-trivial improvement.
We appreciate the reviewers for highlighting that our paper is well-organized and easy to follow (Vaxb, KLx2).
Additionally, we are grateful for noting that the task of OVPS is meaningful and underexplored (Vaxb).
Finally, we are thankful for acknowledging our identification of key issues in OVPS and our effective approach to addressing these issues (LDit, KLx2).

Through this review, we were able to answer the following questions. We kindly ask you to refer to the individual responses and the attached PDF.

The progress made based on the review can be summarized as follows.

* Additional Baseline: VLPart (ICCV 2023)
    * Additional experiments related to the OVPS baseline were conducted.
    * Qualitative and quantitative results of VLPart can be found in Table R1, Fig. R2.
* Qualitative and Quantitative Results of Initial Challenges
    * New qualitative results have been added in Fig. R1.
    * We conducted extra quantitative experiments of boundary IoU for Challenge B (Ambiguous boundaries), and results have been added in Table R3.
* Additional Ablation Studies and Analysis
    * Futher ablation studies have been conducted for different hyperparameters of $\lambda_1$, $\lambda_2$, attention control ($ {\mathcal{L}\_{sep}} $, $ {\mathcal{L}\_{enh}} $), and $ \gamma $ as in Table R2.
    * New qualitative explanations of attention losses are added in Fig. R2.
    * The analysis of the datasets in the aspect of object-specific part size has been added in Fig. R4 and the overall object size in Fig. R5.
    * Detailed computation resource information has been added in Table R4.


We refined the paper for clarity and hope our responses address the reviewers' concerns.
We are happy to answer any further questions.

We sincerely thank you again for the valuable feedback.

Meta-review: This is a borderline paper; three reviewers give positive ratings and and recommend to accept it. Reviewers think the module design is novel and the performance is good. However, reviewers are also concerned about that the paper (1) misses many related works and descriptions of training details, (2) fails to position in the literature of OV Part Seg to better motivate or compare the work, (3) has a large space to improve the writing quality. To prepare the camera-ready, authors should:
- include the missing citations,
- carefully re-write related paragraphs to better motivate the work and better position it in the literature,
- describe important implementation details
- polish the language and improve the readability of the paper.
Decision: Accept (poster)