Understanding Multi-Granularity for Open-Vocabulary Part Segmentation

Authors: jiho  choi
seonho  lee
seungho  lee
minhyun  lee
hyunjung  shim
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 15:50:29

Abstract: Open-vocabulary part segmentation (OVPS) is an emerging research area focused on segmenting fine-grained entities using diverse and previously unseen vocabularies.
Our study highlights the inherent complexities of part segmentation due to intricate boundaries and diverse granularity, reflecting the knowledge-based nature of part identification.
To address these challenges, we propose PartCLIPSeg, a novel framework utilizing generalized parts and object-level contexts to mitigate the lack of generalization in fine-grained parts.
PartCLIPSeg integrates competitive part relationships and attention control, alleviating ambiguous boundaries and underrepresented parts.
Experimental results demonstrate that PartCLIPSeg outperforms existing state-of-the-art OVPS methods, offering refined segmentation and an advanced understanding of part relationships within images.
Through extensive experiments, our model demonstrated a significant improvement over the state-of-the-art models on the Pascal-Part-116, ADE20K-Part-234, and PartImageNet datasets.
Our code is available at https://github.com/kaist-cvml/part-clipseg.

Main paper: 
 introduction: 
 The pursuit of understanding parts and multi-granularity in computer vision [7, 13, 21] mirrors the innate complexities of animal instincts. For example, a "cheetah" instinctively targets an "impala's neck" during a hunt, demonstrating its ability to distinguish specific parts. This ability extends to applications such as robot commands [44] , fine-grained controls on image editing [31] , and more sophisticated image generation [45] . Part segmentation aims to mimic this ability by recognizing intricate details (e.g., parts) within objects, going beyond simple object-level segmentation to achieve detailed and diverse entity recognition.Recognizing parts is more challenging than recognizing whole objects due to their complexity and diversity. Parts often have ambiguous boundaries not only defined by visual cues but also require a broader spectrum of contextual information, reflecting their knowledge-based nature. For example, the "head" of a "dog" may include only the "face" or also the "neck" depending on the annotators' perspective [7, 21] .To address difficulties in part segmentation, Open-Vocabulary Part Segmentation (OVPS) [40, 44, 46] has evolved by leveraging the knowledge of powerful Vision-Language Models (VLMs) like CLIP [38] or ALIGN [24] . Especially, it aims to achieve adaptive recognition and processing of previously unseen categories with the aid of pre-trained VLMs, pushing the boundaries of vocabularies in traditional part segmentation. By utilizing Oracle supervision of base classes during training, recent studies in OVPS exploit part-level knowledge of base classes to generalize to novel classes. Recently, VLPart [40] uses DINO [5] features to map correspondences between base and novel classes and creates pseudo labels for the novel categories. OV-PARTS [46] addresses the ambiguity of part boundaries by introducing object mask prompts and transferring knowledge of base class through a few-shot approach. These methods successfully extract knowledge from VLMs and extend it to novel classes, achieving significant performance improvements in open-vocabulary settings. Classification of a "dog's parts" involving categories like "cats" and "sheep", "dog's tail" misclassified as "sheep's ear". (VLPart [40] ) (b) Ambiguous boundaries: Vague boundary output of "aeroplane's body". (c) Missing underrepresented parts: Neglecting parts such as "beak" and "leg".(CLIPSeg [32, 46] ).However, through empirical analysis of existing OVPS methods, we observed several common limitations in Figure 2 . (Lack of generalization in (a)) Despite understanding part-level information, they often misidentify parts at the object level, e.g., a "dog's leg" as a "cat's leg". Also, part-level misclassification occurs as the knowledge of parts in the base class fails to generalize to a novel class, e.g., "dog's tail" as a "sheep's ear". (Ambiguous boundaries of parts in (b)) They fail to maintain non-overlapping relationships between parts, frequently resulting in overlaps, e.g., an "airplane's wing" overlapping with its "body" or the presence of empty spaces where no part is predicted.(Missing underrepresented parts in (c)) They ignore small and less frequent parts, causing prediction bias based on part size.To overcome these limitations, we propose a novel framework called PartCLIPSeg, which consists of three main components. First, we devise generalized parts with object-level contexts to address the lack of generalization issue as the upper side of Figure 1 . It explicitly obtains object-level and part-level pseudo-labels from VLMs and trains the OVPS model to satisfy both types of supervision. This guides the model to learn object boundaries while recognizing both part and object-level classes. Then, we suggest an attention control for minimizing the overlap between predicted parts, ensuring that parts are clearly separated as the lower side of Figure 1 . In this way, we effectively leverage Part Segmentation. Part segmentation aims to identify the individual parts of objects, a task that is more complex and costly due to the smaller and more diverse nature of parts compared to whole objects. To tackle this, various datasets like Pascal-Part [7] , PartImageNet [21] , ADE20k-Part [57] , Cityscapes-Panoptic-Part [13] , and PACO [2] provide diverse and detailed part annotations. Earlier studies [7, 12, 22, 23, 43] used self-supervised constraints and contrastive settings for effective part-level entity segmentation. Recent studies extended this to open-vocabulary scenarios [35, 40, 46] , opening new avenues for handling diverse parts. By leveraging class-agnostic detectors [35] and Vision-Language Models like CLIP [40, 46] , part segmentation has extended its generalization ability to unseen parts. Our work builds upon and extends methodologies from these studies. 
methodology: 
 As illustrated in Figure 2 , we identified three primary challenges of open-vocabulary part segmentation (OVPS): lack of generalization, overlapping parts, and missing underrepresented parts. Recognizing object-specific parts (such as "dog's torso") cannot be determined solely by looking at each part in isolation; it is imperative to consider both generalized part information and the overall context of the object. Furthermore, some parts may have overlapping meanings across different granularity labels (e.g., "eye", "face", and "head"). This implies that predictions should consider direct guidance for each part as well as the relationships between different parts. These intricate spatial and functional dependencies between parts are crucial for achieving a holistic understanding and precise predictions in fine-grained entity segmentation tasks.Based on this motivation, we propose a novel OVPS method, PartCLIPSeg. This method leverages generalized part information combined with object-level context to tackle the lack of generalization problem (see Section 3.2). Also, we directly minimize the overlap among part predictions to improve the part boundaries (see Section 3.3.1). Finally, we normalize the scale of attention activation from various parts for handling missing underrepresented parts (see Section 3.3.2). The overall architecture of our method is shown in Figure 3 . Considering the domain gap between the datasets, such as differences in granularity, this setting is more challenging.To address the lack of generalization problem, we propose leveraging generalized parts with objectlevel contexts. The concept of generalized parts involves identifying and utilizing common structural components that are shared across different object-level categories. For instance, many animals have parts like "head" or "torso" which, although functionally and visually distinct, may share certain underlying characteristics. By introducing generalized parts from object-specific parts, our PartCLIPSeg can efficiently recognize and segment these object-specific parts across diverse object classes, significantly enhancing the model's ability to generalize from seen to unseen categories.Although generalized parts help distinguish the part-level categories, the visual information of a part may not suffice for accurately classifying their object-level categories. For instance, predicting the "leg" part of an animal can be challenging to identify when solely examining the part as it may not clearly indicate to which animal it belongs. For this reason, there have been attempts to incorporate object-level guidance [33, 40, 46] in part segmentation. However, object-level guidance without a generalized part may lose contextual information and miss hierarchical relationships.By integrating object contexts with generalized parts, PartCLIPSeg employs object-level guidance that captures the holistic essence of the object to which parts belong. This integration allows for a more precise understanding and classification of parts, improving the overall performance of OVPS.Object and Part Embedding Generation. We modified the architecture of CLIPSeg [32, 46] , which adopted CLIP [38] encoder-decoder architecture for semantic segmentation. However, it is worth noting that our approach of utilizing generalized parts with object-level context is orthogonal to other previously proposed object-level segmentation methods [5, 26, 28, 39] .The proposed approach begins by parsing an object-specific part category name, c obj-part ∈ C obj-part , into separate components: an object category name (c obj ) and a generalized part category name (c part ), e.g., "cat" and "torso". Then, the CLIP text encoder, CLIP * T (•), is used to transform these category names into their respective CLIP embeddings (e T obj and e T part ). It will condition the image features, e I , derived from the CLIP image encoder, CLIP * I (•) as:EQUATIONwhere * denotes frozen pre-trained models. By using Feature-wise Linear Modulation (FiLM) [36, 42] , each category name embeddings respectively modulate the image features as:EQUATIONwhere ⊕ is an element-wise sum. FiLM is an adaptive affine transformation widely used for multimodal or conditional tasks. It helps retrieve adequate conditioning for the image features. The modulated image features, e I [obj | part] , corresponding to each object and part category name, pass through a decoder module. The decoder module will be discussed in detail in Section 3.3. They then proceed through a transposed convolution model. Finally, the output mask of the object ŝo and part ŝp are evaluated with ground-truth mask of objects, s o , and parts, s p . Oracle supervision for the object and parts mask is simply computed from a combination of object-specific parts annotations: s ∈ M.Object-specific Part Construction. We utilize previously computed generalized part embeddings (e I part , e T part ) and object embeddings (e I obj , e T obj ) to reconstruct object-specific part embeddings. This process involves separate operations on modulated image features and category name embeddings.Initially, we project the concatenated results of the object category name with the generalized part category name. This is to synthesize the embeddings for the target object-specific part category name. The approach ensures that the resultant embeddings are highly representative of parts and contextually relevant. The equivalent operation is applied to both object-level image features and part-level image features to generate object-specific image features as: ).(EQUATIONThe resulting object-specific part embeddings are further refined by a FiLM process. Combined with the respective object-specific image features, final modulated object-specific part embeddings, e obj-part is computed as: e obj-part = e I obj-part ⊕ FiLM(e T obj-part ).These embeddings are then processed through a deconvolution layer to produce the final segmentation masks s ∈ M. This step ensures that the embeddings are precisely aligned to enhance the definition and accuracy of the object-specific part masks. It effectively bridges the gap between object and part-level categorical information with object-specific parts information.Object, Part, and Object-specific Part Mask Supervision. The mask supervision is provided for three distinct categories: object-specific parts, objects, and generalized parts. This multi-faceted supervision enables our model to effectively disentangle generalized parts from objects, thereby facilitating a more nuanced learning process for OVPS. This disentanglement is crucial for the model to accurately recognize and differentiate between various object categories and their corresponding parts. It enhances the model's ability to handle complex segmentation tasks with unseen objectspecific parts. The overall mask guidance loss can be defined as follows:EQUATIONwhere Figure 4 : Example of attention control using separation and enhance losses. The proposed method manipulates attention maps to accurately identify and segment small parts.In this subsection, we address the previously mentioned challenges: (1) ambiguity in part boundaries and (2) omission of small or infrequently appearing parts. The main reason for these challenges is the incomplete guidance from knowledge-based, multi-granularity characteristics of parts. To overcome these, we adopt unsupervised methods traditionally used in fine-grained recognition and part discovery studies [7, 12, 43] . Specifically, we utilize approaches for adjusting self-attention activation inspired by the recent diffusion methods [6, 25, 41] .We assume that the distribution of self-attention activation maps for visual tokens belonging to the same object-specific part mask should exhibit inter-similarity characteristics [41] , implying similar distributions. To this end, we first compute the average self-attention map A Mc for each objectspecific part mask M c , where c ∈ C obj-part represents an object-specific part category. This is done by summing the self-attention activation maps from channels specifically corresponding to object c obj and part c part , across all spatial tokens (h, w) within the mask, as follows:EQUATIONSubsequently, the self-attention map A Mc for the object-specific part mask is refined through minmax normalization, followed by the application of a Gaussian filter to smooth the initial activation as in [6, 50] . Therefore, the dimensions of both the original and normalized self-attention maps for the object-specific part masks are as follows:A Mc , A norm Mc ∈ R H×W .In the self-attention of the decoder layers, competition between object-specific parts helps define boundaries that cannot be sufficiently established by supervision alone. Using the previously obtained normalized attention map, our method generates parts with minimized intersections, inspired by [1, 3, 25, 37, 47] . This approach effectively mitigates the ambiguity issue in part boundaries.Mc is first binarized based on an arbitrary threshold γ as:EQUATIONwhere B Mc denotes binarized attention map for part mask M c . From now on, C obj-part is simply denoted as C. The separation loss L sep , which indicates the degree of intersection between objectspecific parts, is as follows:EQUATIONwhere separating activation mitigates the challenge of ambiguous boundaries between parts.To address the omission problem, we employ a method inspired by attention controls in modern diffusion-based approaches [3, 6] . This approach enhances the activation within the self-attention activation map to enhance underrepresented parts before normalization. Specifically, for each objectspecific part mask, the maximum value within the attention map is identified. Subsequently, among all object-specific parts, the minimum activation of the part with the maximum value is enhanced as:EQUATIONthereby boosting its representational efficacy. In this way, the enhancement loss L enh provides sufficient guidance for small or infrequently occurring parts, effectively mitigating the omission problem.The training objective for PartCLIPSeg integrates three key loss components as:EQUATIONwhere (1) L mask for generalized parts with object-level context, (2) L sep for addressing ambiguous boundaries, (3) L enh for handling missing underrepresented parts, and λ sep and λ enh are hyperparmeters.4 Experiments 
experimental setups: 
 Datasets. We evaluate our method on three part segmentation datasets: Pascal-Part-116 [7, 46] , ADE20K-Part-234 [46, 57] , and PartImageNet [21] . Pascal-Part-116 [7, 46] [14] and organizes them into 11 super-categories. For this study, we select 40 object classes that represent common categories to assess cross-dataset performance effectively. More details about the datasets can be found in the supplementary materials.Evaluation Protocols. We use two evaluation protocols for the performance of OVPS: (1) Pred-All setting, where the ground truth object-level mask and object class are not provided, and (2) Oracle-Obj setting, where the ground truth object-level mask and object class are known. In particular, the Pred-Obj setting in OV-PARTS [46] uses predicted masks from the off-the-shelf segmentation model. In contrast, our Pred-All setting is a more challenging and practical setting because it does not rely on additional predicted masks or foundation models but solely uses the predicted object masks from the proposed model. For both evaluation protocols, we used mean Intersection over Union (mIoU) as an evaluation metric, which is widely used to measure segmentation performance. Additionally, we utilized the harmonic mean of the results from the seen and unseen categories as the final evaluation metric.Implementation Details. We build upon CLIPSeg [32, 46] , a CLIP-based encoder-decoder model. The implementation details can be found in the supplementary material.Zero-Shot Part Segmentation. We compare our PartCLIPSeg to previous methods [11, 32, 46, 52] on three OVPS benchmarks [7, 57] . As shown in scenario, these results are significant despite some performance degradation in seen categories. We also evaluated PartCLIPSeg on PartImageNet. According to Table 3 , PartCLIPSeg shows a notable improvement over CLIPSeg.We present the segmentation results of PartCLIPSeg in comparison to state-of-the-art open-vocabulary part segmentation methods [11, 32, 40] on Pascal-Part-116. Specifically, we focus on qualitative performance on unseen categories such as "dog", "sheep", "car", and "bird". As shown in Figure 5 for the Pred-All and Figure 6 for the Oracle-Obj setting, the proposed method effectively segments target parts regardless of the need for predefined masks during inference. Notably, PartCLIPSeg excels at identifying smaller, often overlooked part classes such as "eye", "tail", and "headlight". Additionally, our method effectively segments multiple objects and their respective parts, a challenge for other methods, demonstrating the effectiveness of PartCLIPSeg in zero-shot part segmentation. Its improved performance on unseen categories and higher accuracy in challenging environments Cross-Dataset Part Segmentation. Table 4 validated the efficacy of our approach in a cross-dataset setting, where category names, annotation style, and granularity of mask may vary. Additionally, unlike zero-shot situations within the same dataset, there are differences in the types and diversity of parts. Initially, we trained our model on PartImageNet and ADE20K-Part-234 respectively. Subsequent tests on Pascal-Part-116 [7, 46] showed that PartCLIPSeg outperforms CLIPSeg in both the Pred-All and Oracle-Obj settings, confirming our method's superiority on generalization in different datasets. In this section, we analyze the impact of each training loss on PartCLIPSeg. We focus on the roles of the separation and enhancement losses, examining how they contribute to improved segmentation accuracy.Separation & Enhancement Losses. We conducted an ablation study to investigate the effect of the separation loss L sep and the enhancement loss L enh on the performance of PartCLIPSeg in To verify the effectiveness of boundary creation of PartCLIPSeg, we examined an additional qualitative metric, Boundary IoU [8] . The results demonstrated high Boundary IoU performance, confirming that PartCLIPSeg effectively resolves ambiguous boundary issues as shown in table 6 .Impact of PartCLIPSeg for Underrepresented Parts. We investigate the effect of the enhancement loss L enh on OVPS model performance, especially with respect to underrepresented parts. In Table 7 , we compare our PartCLIPSeg with CLIPSeg [32, 46] on small parts such as "eye", "neck", and "leg" of animals in Pascal-Part-116. As shown in the table, PartCLIPSeg consistently outperforms CLIPSeg with significant improvements in most cases. Notably, there is an impressive performance increase of 35.93%p for "cow's leg". These improvements highlight the effectiveness of the enhancement loss in accurately segmenting small and intricate parts, demonstrating its crucial role in improving overall performance. 
conclusion: 
 In this study, we introduced PartCLIPSeg, a state-of-the-art OVPS method that addresses three primary challenges in OVPS. PartCLIPSeg utilizes generalized parts and object-level guidance to effectively solve identification issues. Then, it separates parts by minimizing their overlaps in attention maps, thus learning ambiguous part boundaries. Additionally, we implemented an enhancement loss function to improve the detection of underrepresented parts. Through extensive experimentation, we have confirmed the superior performance of PartCLIPSeg.beneficial in the creative industries, where precise adjustments can improve output quality and foster innovation.Adopting open-vocabulary part segmentation promises to enhance the efficiency, accessibility, and effectiveness of these technologies, particularly in handling real-world variability and unpredictability. In the Pascal-Part-116 dataset [7, 46] , we target the following object-specific category names in Table A1 . Among these, "bird", "car", "dog", "sheep", and "motorbike" are designated as unseen categories, encountered for the first time during inference in the zero-shot part segmentation setting.  
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The experimental setting and details are provided in section 4 and supplementary materials. Also, the anonymized repository of our implementation and reproduction instructions are provided. (URL is in the abstract section)Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: The standard error of an average of 5 results is reported in Section 4 of the proposed model.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: The experimental resources, setting and details are provided in section 4 and supplementary materials. Also, the anonymized repository of our implementation and reproduction instructions are provided. (URL is in the abstract section)Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The broader of impact is discussed in supplementary materials.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We properly credited previous works and codes in section 4. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The anonymized repository of our implementation and reproduction instructions are provided. (URL is in the abstract section) Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In the ADE20K-Part-234 dataset [57] , we target specific object categories listed in Table A2 . The dataset includes 44 object classes and detailed subdivisions into over 200 part categories. Notably, "bench", "bus", "fan", "desk", "stool", "truck", "van", "swivel chair", "oven", "ottoman", and "kitchen island" are identified as novel classes and are encountered for the first time during inference in our zero-shot part segmentation setting.PartImageNet [21] is a dataset derived from ImageNet [14] , consisting of approximately 24,000 images across 158 classes. Each class has annotations for parts. All classes belong to one of 11 superclasses, organized using the hierarchical information from WordNet [34] .Previous open-vocabulary part segmentation research [40] primarily used PartImageNet to evaluate cross-dataset settings. In our study, we use PartImageNet not only for cross-dataset evaluation but also to assess model performance in zero-shot settings specific to PartImageNet.To measure more generalized performance, we select 40 classes out of the 158. We maintain the proportion of existing superclasses as much as possible. For each superclass, at least 50% of the categories are designated as seen categories, with the remaining being unseen categories. Therefore, there are 25 seen classes and 15 unseen classes in our PartImageNet evaluation dataset. We conduct the dataset evaluation as follows: Models are trained on a training dataset composed of seen classes. Segmentation performance are then assessed on a validation dataset containing both seen and unseen classes. Evaluations were conducted in both Pred-All and Oracle-Obj settings. Our model implementation is based on the CLIPSeg [32] architecture, as described in the OV-PARTS [46] . We utilized the pre-trained CLIP ViT-B/16 [38, 59] image encoder and text encoder for our experiments.The model is trained using the ADAMW optimizer with a base learning rate of 0.0001 over 20,000 iterations, with a batch size of 8 images. We employ a WarmupPolyLR learning rate scheduler to manage the learning rate throughout the training process. To ensure model stability, we apply gradient clipping with a maximum gradient norm of 0.01.We save model parameters every 1,000 iterations during training. The best-performing parameters are selected based on the highest validation evaluation scores. For example, the evaluation result on the Pascal-Part-116 dataset in the Oracle-Obj setting is derived from the checkpoint saved at the 5,000-step mark, which yields the best validation performance.We evaluated several baseline methods-ZSSeg+, CLIPSeg [32] , and CAT-Seg [11] -which are fine-tuned on our datasets. ZSSeg+ is a modified version of ZSseg [52] , utilizing different fine-tuning methods according to [46] . It employs a ResNet-101c backbone and Compositional Prompt Tuning based on CoOp.CLIPSeg and CAT-Seg models are pre-trained on object datasets; however, we fine-tuned these models on each part-level dataset. CAT-Seg, based on ResNet-101 and using ViT-B/16 as CLIP's visual encoder, achieved comparable performance by computing cost volumes and subsequently applying cost aggregation-a process that enhances segmentation by aggregating matching costs between image features. Specifically, CAT-Seg uses the frozen upsampling decoder but fine-tuned CLIP's image and text encoders. Conversely, we fine-tune the CLIPSeg decoder to better identify small segments and define clear boundaries. CLIPSeg, based on the ViT-B/16 architecture, is finetuned on the visual adapter, text embeddings, and transformer decoder to enhance its segmentation capabilities.Table A4 : Computational resources on Pascal-Part-116 with batch size 8.ZSSeg [52] 191.6 M 11.1 G CLIPSeg [32, 46] For PartCLIPSeg, although the number of parameters is larger than CLIPSeg because of computations related to attention control, there is an advantage in not having to maintain weights for each object-specific part due to the use of generalized parts. These efficiencies become more pronounced as the number of generalized parts shared among object classes increases. By leveraging shared representations for generalized parts, PartCLIPSeg reduces redundancy and memory requirements. This makes our model particularly advantageous in datasets where object classes have many common parts, leading to more efficient training and inference without compromising performance. In this section, we present an additional evaluation metric that focuses on specific challenges within the Open-Vocabulary Part Segmentation (OVPS) task as shown in Figure 2 . The Recall metric is used to assess how well the model captures underrepresented parts, addressing the challenge of underrepresented parts. Higher values in recall indicate that the model effectively captures these seldom-occurring parts, thereby addressing the challenge of underrepresented parts in OVPS.PartCLIPSeg consistently achieves higher recall on both seen and unseen classes across both datasets as shown in Tables A5 and A6 . The improved harmonic mean indicates that our model is more effective at identifying underrepresented parts, thereby addressing one of the core challenges in OVPS.We further analyze the impact of the attention control losses L sep and L enh on the recall. By comparing the recall metric with and without these losses, we assess their effectiveness in enhancing the representation of seldom-occurring parts. From Tables A5 and A6 , we observe that incorporating the attention control losses enhances the model's performance on unseen classes, which often include underrepresented parts. The increases in harmonic mean suggest that the attention control losses help the model to better capture these seldom-occurring or small parts.D.1 Impact of Object-Level and Part-Level Guidance We conduct additional experiments to verify the impact of object-level and part-level label guidance on model performance as shown in Table A7 . Specifically, we vary the weights λ obj and λ part in Equation ( 5), setting each to 0 or 1, to assess the influence of object-level and part-level supervision on the overall performance. Additionally, we evaluate the effect of the attention control losses, L sep and L enh , by including or excluding them. Additionally, removing the attention control losses L sep and L enh while keeping both λ obj and λ part at 1.0 results in a lower Harmonic mean of 38.20. This indicates that the attention control losses contribute to better distinguishing between seen and unseen classes.The separation loss reduces the overlap between different parts, while the enhancement loss strengthens the activation of underrepresented parts. As shown in Figure A2 , when only the separation loss L sep is applied (top), smaller parts adjacent to larger parts may be diminished. Specifically, "sheep's neck" is not properly highlighted because minimizing the intersection can cause larger parts, such as the "sheep's torso" and "sheep's head", to overshadow smaller ones. When both losses L sep and L enh are utilized (bottom), the model accurately segments the small part-"sheep's neck"-as the enhancement loss boosts its representation, preventing it from being overwhelmed by larger neighboring parts.This demonstrates that the separation and enhancement losses complement each other. Their combined use is essential to effectively distinguish and represent both large and small parts within an object, leading to improved segmentation performance.To evaluate the sensitivity of our method to the hyperparameter threshold γ in Equation ( 8), we conducted experiments on the Pascal-Part-116 dataset under the Oracle-Obj setting. We varied γ from 0.1 to 0.5 and measured the performance in terms of mIoU for seen and unseen classes, as well as the harmonic mean.As shown in Table A8 , our method is robust to the choice of γ within the range of 0.1 to 0.5. The harmonic mean remains relatively stable, with the best performance achieved at γ = 0.3. While there is a slight variation in performance across different values of γ, the changes are not significant, indicating that our method does not heavily depend on the exact value of this hyperparameter. The t-SNE visualization of text embeddings from a pre-trained CLIP [38, 32, 58] model on the Pascal-Part-116 dataset [7, 46] reveals intriguing insights into the model's understanding of categories. Notably, similar classes such as "cats" and "dogs" are clustered closely within the embedding space. This proximity indicates a shared semantic space for categories that are visually or contextually related.Additionally, we observed that object-specific parts sharing generalized parts, such as "car's license plate" and "bus's license plate", are also positioned near each other. This clustering suggests that the CLIP recognizes and leverages common parts across different objects that share common characteristics. Further analysis shows that object-specific classes containing parts like "muzzle" and "paw" are distributed in similar regions of the space. This consistency across different object categories emphasizes the CLIP's ability to generalize part-level features effectively.Leveraging CLIP's text embeddings provides a significant zero-shot capability in the visual domain. This capability can be extended to part-level categories, demonstrating the potential for sophisticated unsupervised or zero-shot learning approaches in fine-grained object and part recognition tasks. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Justification: -Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This is a borderline paper; three reviewers give positive ratings and and recommend to accept it. Reviewers think the module design is novel and the performance is good. However, reviewers are also concerned about that the paper (1) misses many related works and descriptions of training details, (2) fails to position in the literature of OV Part Seg to better motivate or compare the work, (3) has a large space to improve the writing quality. To prepare the camera-ready, authors should:
- include the missing citations,
- carefully re-write related paragraphs to better motivate the work and better position it in the literature,
- describe important implementation details
- polish the language and improve the readability of the paper.

Reviews: review: paper_summary: paper_summary: This paper identifies three key problems of current open-vocabulary part segmentation, namely, lack of generalization, ambiguous boundaries, and missing underrepresented parts. The paper then proceeds to propose solutions to fix these problems. Experimental results show that the proposed method outperforms previous state-of-the-art methods.
strength_weakness: strength_weakness: 1. **Limited discussion of previous open-vocabulary part segmentation work and their relationship to this work.** For example, I quickly searched and read the VLPart paper [1], which seems to parse objects first, then parts, thus avoiding the object-level misclassification issue (or the "lack of generalization" as named by the authors). Indeed, the VLPart seems very natural to me. So I am actually surprised to find that there is such an issue (mixing objects and parts during classification) as indicated by this paper. In general, the related work section provides very limited introduction of the previous open-vocabulary part segmentation methods, esp. how they are related to the proposed approach. There are also **no descriptions of the compared baselines in the experiment section**.This made readers hard to make a fair assessment of the contribution of this paper.

2. **Limited ablation studies to understand the proposed approach.** Some key questions are unanswered: how much does each proposed component (object-level context, attention control) contribute to the performance gain? In what way or what types of data? It might also be helpful to compare the Pred-Obj performance because that helps disentangle the localization and classification ability of the approaches. Current comparison mixes many components together and it's difficult to gain insights about how the gain is achieved. 

3. **Lack of addressing directly the identified key problems**. The paper starts with three key issues and then propose different ways to fix them. However, in the experiments, there are no direct addressing to these problems any more (except Table 6 for small parts). It would be great if the experiments can clearly demonstrate how the three problems are mitigated by the proposed approaches. 

4. **Lack of discussion on the training details in the paper**. For readers who are not familiar with CLIPSeg, it is impossible to tell how the model is trained and what datasets are used for training. There is also **a lack of descriptions of the compared baselines in the paper**. Without these information, it's hard to understand the performance gains.

Minor issues: 

1. I found it confusing to call it "lack of generalization" when the dog's parts are misclassified as sheep's or cat's. Isn't it just misclassification, or object-level misclassification?

2. Texts on the figures are way too small, esp. in Figure 2 and Figure 4.

3. The "supplementary material" for this paper should be put as an appendix to the main paper file, instead of as a separate file.
questions: questions: 1. What is "incomplete guidance from knowledge-based, multi-granularity characteristics of parts"? It is unclear from the texts. It seems like an important motivation for the design of the attention control. Some more analysis or discussion would help here.

2. In Table 1 and Table 3, why are the performance of PartCLIPSeg on "seen" classes so low? From the model design, I don't see a reason for this. Do the authors have an explanation? 

3. How to understand the "failures" in Table 6 where PartCLIPSeg cannot improve the performance from the CLIPSeg (such as "bird's eye") or PartCLIPSeg also has very low performances (such as "sheep's eye", "cow's neck")? And are all these parts indeed *small*? For example, "cow's leg" doesn't seem to be small to me. How do the authors define what are small parts?
limitations: limitations: The authors have addressed the limitation and potential societal impact of the work.
paper_summary: paper_summary: This paper proposes PartCLIPSeg, which builds upon the CLIPSeg model and extends it to handle the unique challenges of part segmentation. This paper introduces several components to address the challenges in OVPS.

1. Generalized parts with object-level contexts: This approach combines generalized part information with object-level context to mitigate the lack of generalization in fine-grained parts.

2. Attention control: This method introduces competitive part relationships and attention control techniques to minimize overlap between predicted parts and enhance activation for underrepresented parts. This mechanism is a new addition to the field.

The paper conducts experiments on OVPS’s baseline datasets. The result shows that PartCLIPSeg outperforms existing SOTA methods in most settings. The Ablation primarily focuses on the impact of the proposed attention control loss, which lacks a comprehensive analysis of other key components of this paper.
strength_weakness: strength_weakness: - Insufficient ablation studies: The ablation studies primarily focus on the proposed attention control loss, which is not sufficient to prove the effectiveness of the proposed method. 
     1) The impact of combining object-level and part-level pseudo-label is not explored. 
     2) The paper does not provide sufficient evidence that the proposed losses work as intended, beyond their impact on the final metric. Further discussion is needed to validate whether the losses effectively address the challenges they were designed to tackle.
- Limited comparison with other OVPS methods: The paper compares PartCLIPSeg primarily with CLIPSeg and CATSeg, which are designed for open-vocabulary segmentation and not specifically for OVPS. The comparison with other OVPS methods is lacking.
- Limited discussion on computational complexity: Since the paper introduces a new attention control mechanism, a detailed analysis of the computational requirements and speed trade off could be helpful for a more comprehensive understanding of the proposed method.
Inadequate qualitative analysis: Insufficient exploration and insights of the model’s performance on challenging and failure cases.
questions: questions: See Weakness.
limitations: limitations: No societal impact.
paper_summary: paper_summary: This paper proposed PartCLIPSeg for open-vocabulary part segmentation. This framework leverages generalized parts and object-level contexts for generalization in fine-grained parts, integrating competitive part relationships and attention control techniques. Extensive experiments on three datasets demonstrate the superior performance of PartCLIPSeg.
strength_weakness: strength_weakness: 1) Figure 3 (overall architecture) can be refined to clearly emphasize the contribution. 
2) The analysis and explanation of the experiment are not sufficient. Although the method achieves a significant improvement on unseen categories, the results of seen categories on the Pred-All setting of the three datasets show different trends. Specifically, the results of the seen categories show a decrease compared to the baseline in Tables 1 and 3 (There exists an annotation mistake (41.70).), and a significant improvement in Table 2. Please discuss this result.
3) Ablations of loss functions. We notice that adding some loss function in some cases may lead to suboptimal results and show different trends on different datasets. For example, the second column of the Pred-All setting in Table 5. Please discuss this difference.
questions: questions: How is the threshold γ in Equation 7 determined? How does this threshold affect the results?
limitations: limitations: The unseen categories of the three datasets in the experiment may have been seen during the CLIP pre-training, and we want to explore how well PartCLIPSeg generalizes to the “truly” unseen categories that have not been seen in the pre-training or from other fields

Comments: title: Thanks for the authors' responses
Thanks for the response and a few more comments
kicking off reviewer-author discussion
comment: We appreciate your thorough review and your acknowledgment of the significance of our research on open-vocabulary part segmentation. We are glad that the concerns raised have been fully resolved and are thankful for your recommendation to accept the paper.
We sincerely appreciate your thoughtful review and valuable comments.
We are pleased to have resolved the questions raised and appreciate your positive decision.
Thank you again for your time and insight.
After revisiting the original paper, reviewing the comments from other reviewers, and considering the authors' responses, I find that my raised issues have been addressed. I believe the research topic of open-vocabulary part segmentation is meaningful, and this paper provides valuable research and very good results. I recommend accepting this paper. At this point, I tend towards keeping my original score.
Hi dear reviewer,

Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments.

Thanks,

AC
Hi dear reviewer,

Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments.

Thanks,

AC
I want to thank the authors for their detailed responses and clarifications. They help clarify some of my misunderstandings. I re-read the paper and the other comments. Based on my final understanding, I decided to keep my current rating as borderline acceptance.
## **3. Final comments**

Thank you for your feedback. We understand your concerns. However, we had a thorough understanding of the existing research (including VLPart) and excluded VLPart from the experimental comparison only due to differences in experimental protocols. We also emphasize that our experiments on semantic segmentation protocols are thorough and more competitive in terms of performance. Besides, we even suggest a new evaluation criterion, the Pred-All setting.

* We focused on the semantic segmentation task of OVPS for the reasons mentioned earlier in answer 1.
    * We adopted the state-of-the-art OVPS baseline from OV-PARTS and conducted a thorough comparison with all comparable methodologies in this task.
    * Although VLPart was mentioned in the introduction and related work, a detailed comparison was initially omitted because VLPart adheres to the instance segmentation protocol. However, this has been strengthened thanks to the reviews (Table R1).
    * Baselines such as ZSSeg+, CATSeg, and CLIPSeg (introduced in OV-PARTS) along with their training details were previously introduced in Section 2.2 of the supplementary materials under Implementation Details (L77-L98). However, based on the feedback, we will revise the related work to make these descriptions more prominent and ensure that they are not overlooked.
* We were aware of existing OVPS studies, including VLPart, from the initial submission. OVPS is an emerging and underexplored research area with limited research focused on improving the direct alignment between images and part-level text.
    * In the Part Segmentation section of the introduction and related work, we briefly described all the latest OVPS-related methods we could find at the time of submission, including VLPart [35] (L33-34, L84-86), OV-PARTS [40] (L34-37), and OPS [30] (L83-84).
    * **To the best of our knowledge, OV-PARTS achieved the highest performance in OVPS and there have been no follow-up studies** specifically addressing the part-level semantic segmentation task in an OV setting without relying on additional mask proposal models, aside from our work.


[30] OPS: Towards Open-World Segmentation of Parts (CVPR 2023)

[35] VLPart: Going Denser with Open-Vocabulary Part Segmentation (ICCV 2023)

[40] OV-PARTS: Towards Open-Vocabulary Part Segmentation (NeurIPS 2023 D&B)
## **2. Related to W3**

In this study, we identified three key challenges associated with existing OVPS methodologies (VLPart, OV-PARTS) and proposed an enhanced OVPS approach to address these issues: (a) Lack of generalization, (b) Ambiguous boundaries, and (c) Missing underrepresented parts. In the following, we clarify how the proposed approach addresses the key challenges.

##### **(a) Lack of generalization:**
To address generalization issues, we develop generalized part guidance and object guidance. (Section 3.2: L117-176)
Generalized part guidance is designed to learn the part class across (object-specific part) base classes in the training set so as to improve performance on novel classes in testing. Object guidance is used to resolve the ambiguity of the same part category name belonging to a different object class by using object-level supervision. Our results confirm that misclassification has reduced: qualitatively in Fig. 5 (row 3), Fig. R1 (a), Section 3.2.2 (suppl.), and quantitatively in Tables 1, 2, 3 (especially under the Pred-All setup), Table 4 (cross dataset), and Table R2 based on the m-IoU metric. An ablation study in Table R2 clearly justifies the roles of both guidances.

##### **(b) Ambiguous boundaries:**
To overcome the ambiguity in part-level supervision, we leverage the common understanding that these parts should be distinct and non-overlapping. To achieve this, we introduce a constraint $\mathcal{L}\_{sep}$, which ensures that attention activations from different object-specific parts remain disjoint within an object. (Section 3.3.1: L194-203). 
It is supported by qualitative results in Fig. 5 (row 1) where ambiguous boundaries are mitigated, Fig. R1(b), and Section 3.2.2 (suppl.), where non-fully covered part predictions occur in other baselines, particularly in the Pred-All setting. Quantitative results are presented in Table R3, based on the boundary IoU metric. An ablation study in Tables A5 and A6 further demonstrates the effectiveness of our proposed loss function in mitigating ambiguous boundaries.

##### **(c) Missing underrepresented parts:**
To ensure that underrepresented parts, such as small or less frequent ones, are not ignored, we propose $ \mathcal{L}\_{enh} $. This is achieved by maximizing the attention activation values for these parts, allowing them to be more effectively identified. (Section 3.3.2: L204-214)
Our method is supported by qualitative results shown in Fig. 4, Fig. 5 (row 2), Fig. R1 (c), and Section 3.2.1 (suppl.) where small part classes such as bird’s tail and beak are correctly segmented. Quantitative results are provided in Table 6, Fig. R4, Table A1, and Table A2, using the metrics m-IoU and recall. Ablation studies on m-IoU in Section 4.3, Table 5, and Recall in Table A3, and Table A4 further highlight the effectiveness of our proposed method.


**(Dataset / Evaluation Setting / Metric)**

**[Table A1] (Pascal-Part-116 / Oracle-Obj / Recall)**
|model|seen|unseen|harmonic Recall|
|:-:|:-:|:-:|:-:|
|ZSSeg+|65.47|32.13|43.10|
|CAT-Seg|56.00|43.20|48.77|
|CLIPSeg|55.71|43.35|48.76|
|PartCLIPSeg|58.46|47.93|52.67|


**[Table A2] (ADE20K-Part-234 / Oracle-Obj / Recall)**
|model|seen|unseen|harmonic Recall|
|:-:|:-:|:-:|:-:|
|ZSSeg+|55.78|40.71|47.07|
|CAT-Seg|43.48|39.87|41.60|
|CLIPSeg|49.59|48.11|48.84|
|PartCLIPSeg|53.31|51.52|52.40|


**[Table A3] (Pascal-Part-116 / Oracle-Obj / Recall)**
|model|seen|unseen|harmonic Recall|
|:-:|:-:|:-:|:-:|
|w/o $ \mathcal{L}\_{sep} + \mathcal{L}\_{enh} $ |58.97|46.47|51.98|
|w/ $ \mathcal{L}\_{sep} + \mathcal{L}\_{enh} $|58.46|47.93|52.67|


**[Table A4] (ADE20K-Part-234 / Oracle-Obj / Recall)**
|model|seen|unseen|harmonic Recall|
|:-:|:-:|:-:|:-:|
|w/o $ \mathcal{L}\_{sep} + \mathcal{L}\_{enh} $ |51.64|50.99|51.31|
|w/ $ \mathcal{L}\_{sep} + \mathcal{L}\_{enh} $|53.31|51.52|52.40|


**[Table A5] (Pascal-Part-116 / Oracle-Obj / B-IoU)**
|model|seen|unseen|harmonic B-IoU|
|:-:|:-:|:-:|:-:|
|w/o $ \mathcal{L}\_{sep} + \mathcal{L}\_{enh} $ |36.24|37.87|37.04|
|w/ $ \mathcal{L}\_{sep} + \mathcal{L}\_{enh} $|36.15|39.07|37.55|


**[Table A6] (ADE20K-Part-234 / Oracle-Obj / B-IoU)**
|model|seen|unseen|harmonic B-IoU|
|:-:|:-:|:-:|:-:|
|w/o $ \mathcal{L}\_{sep} + \mathcal{L}\_{enh} $ |24.99|22.41|23.63|
|w/ $ \mathcal{L}\_{sep} + \mathcal{L}\_{enh} $|25.67|22.46|23.96|
Thank you for providing the opportunity to clarify several points.

## **1. Related to W1**

* VLPart (ICCV 2023) approaches OVPS through instance segmentation, while OV-PARTS (NeurIPS 2023) utilizes semantic segmentation. We followed the OV-PARTS protocol, which is the more recent and higher-performing method, due to the following reasons.
    * We believe that the lack of text-image alignment is a significant issue in the current OVPS landscape.
    * Addressing semantic segmentation first allows us to focus on the core alignment issues within OVPS.
    * OV-PARTS has demonstrated the most notable performance among existing OVPS methodologies, and we have used it as a baseline for our task.
* In response to the reviewer’s suggestion that instance segmentation may be more suitable for part segmentation, we believe that further research is needed, particularly in open-vocabulary settings.
    * While it may be logical for part segmentation to evolve into instance or panoptic segmentation in the long term, it is critical first to develop models that can extend to various categories under semantic segmentation, especially in zero-shot environments like open-vocabulary. For this reason, semantic segmentation is actively researched in object-level open-vocabulary domains (e.g. OVSS), as seen in recent works like CAT-Seg (CVPR 2024), PnP-OVSS (CVPR 2024), and FC-CLIP (NeurIPS 2023).
    * **Methods based on instance segmentation have not yet demonstrated effective performance in fine-grained recognition tasks like part segmentation compared to semantic segmentation**.
        * As shown in Table 4 of VLPart, despite being tested in a simpler experimental setup (with only two novel classes, dog and bus), VLPart exhibits low performance, with a mAP of 4.2 and mAP_50 of 11.0 on novel classes. Similar trends were also confirmed under our experimental conditions, as shown in Table R1.
        * These results highlight the limitations of instance-based segmentation, which do not outweigh its purported advantages.
I first want to thank the detailed responses from the authors, esp. the added experiments, they are very helpful to understand the method. And I still have a few more comments:

1. Related to W1: "VLPart focuses on instance segmentation with masks, whereas our research compares semantic segmentation methods". Does it make sense to comparing semantic segmentation for this work in the first place? As far as I understand, this paper is targeting the part segmentation problem, then it definitely should treat it as at least an instance segmentation problem, or more appropriately, a panoptic segmentation problem. Especially given prior works VLPart already works on instance segmentation problem, I cannot understand the choice from the authors. I don't want to jump to negative guesses, so maybe the authors can give some more explanation on the chosen baselines and the reasoning behind it.

2. Related to W3: it's great to have more qualitative examples and additional quantitative studies on boundary IoU. But I still think it doesn't demonstrate clearly enough how the proposed method solves the key problems and why the method is able to. It's probably more about the paper presentation. At this point, it might even be helpful to rethink what's the key problems that the method has resolved and how to present them better. Then additional quantitative results would be useful to support the reasoning.

3. Final comments: I appreciate the authors adding many details like related works and training details. However, these should definitely have been present in the paper in the first place. These made me feel the paper was a bit hurried. To be frank, I am also a bit worried about whether a full comparison with other works is complete. While I did mention VLPart, that's because I only searched this paper. As a non-expert in this subfield, I am not sure whether there are many more or not. The lack of many details made me less confident of the overall paper. I will maintain my rating for now.
Thanks the authors for the rebuttal, and reviewers' for the comments. We are in the period of author-reviewer discussion.
- Reviewers, please read through all the reviews and rebuttal and see if authors' responses have addressed your and others' important questions.
- Authors, please follow up with any reviewers' further comments.

Cheers,

AC
We apologize for the oversight in our latest rebuttal. The response to Weakness 3 was accidentally omitted. We regret any confusion this may have caused.

The response to Weakness 3 is provided below.

----

## **Weakness 3**: Loss Functions Ablations Discussion

We will provide additional explanations to the initial explanation to address missing details.

The separation loss reduces the overlap between parts, while the enhancement loss strengthens the underrepresented parts. These two losses are not independent; true effectiveness can be achieved when used together.

Without the enhancement loss, the separation loss can lead to smaller parts being diminished when adjacent to larger parts. As illustrated in Fig. R2, applying only the separation loss results in missing small parts: minimizing the intersection may cause larger parts, such as sheep’s torso, head, and muzzle, to overshadow smaller parts, like sheep’s neck. Therefore, the enhancement loss is essential to ensure that the small parts, in the bottom row of Fig. R2, are accurately segmented and not overwhelmed by larger neighboring parts.

We hope this additional explanation clarifies the role of separation and enhancement losses.