Title: Open-Vocabulary Part Segmentation via Progressive and Boundary-Aware Strategy
Venue: NeurIPS 2025
Accepted: yes
Keywords: Open-Vocabulary Part Segmentation

Progressive Segmentation

Boundary-Aware Refinement

Hierarchical Part Connected Graph
Abstract: Open-vocabulary part segmentation (OVPS) struggles with structurally connected boundaries due to the inherent conflict between continuous image features and discrete classification mechanism. To address this, we propose PBAPS, a novel training-free framework specifically designed for OVPS. PBAPS leverages structural knowledge of object-part relationships to guide a progressive segmentation from objects to fine-grained parts. To further improve accuracy at challenging boundaries, we introduce a Boundary-Aware Refinement (BAR) module that identifies ambiguous boundary regions by quantifying classification uncertainty, enhances the discriminative features of these ambiguous regions using high-confidence context, and adaptively refines part prototypes to better align with the specific image. Experiments on Pascal-Part-116, ADE20K-Part-234, PartImageNet demonstrate that PBAPS significantly outperforms state-of-the-art methods, achieving 46.35\% mIoU and 34.46\% bIoU on Pascal-Part-116. Our code is available at https://github.com/TJU-IDVLab/PBAPS.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Progressive Boundary-Aware Part Segmentation (PBAPS), a training-free model for open-vocabulary part segmentation (OVPS). This model takes the target part class vocabulary, and uses Stable Diffusion, SAM and DINOv2 to get “prototype” embeddings for all classes. Then, it uses a class hierarchy graph (called HPCGraph) and the generated prototype embeddings to segment an image in a top-down manner: first the object-level classes, then the part-level classes, then the subparts, etc. When segmenting the parts and subparts, the model applies a Boundary-Aware Refinement (BAR) module to refine initial segmentation predictions at boundaries between parts that are structurally connected (e.g., “dog-head” and “dog-neck”), as the paper observes that this is where many existing OVPS models struggle. Through experiments, PBAPS is shown to be more effective than existing OVPS models, achieving new state-of-the-art results. Ablation experiments show that the use of the HPCGraph for hierarchical segmentation and the BAR module are essential in achieving this performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1.\tThe presented model, PBAPS, is shown to be effective. On various OVPS benchmarks, it significantly outperforms existing models. This demonstrates the value of the presented model.\n2.\tThe proposed HPCGraph and BAR module are shown to be effective and essential to achieve the state-of-the-art performance that PBAPS obtains, through ablation experiments in Tab. 3. This validates the design of the proposed components.\n3.\tThe qualitative results shown in Fig. 5 provide relevant insights into the actual quality of the predictions by the different models. They show that PBAPS is better able to identify the correct boundaries between parts of an object, which was one of the objectives of this work.\n\n**Weaknesses**\n\n1.\tThe overall method – i.e., the use of Stable Diffusion, SAM, and DINOv2 to obtain class prototypes that can be used for open-vocabulary segmentation – is largely based on existing method RIM [a], but the paper does not acknowledge this in the paper nor provides proper attribution to this work. The paper does cite RIM as one of several open-vocabulary segmentation models in the related work, and compares to it experimentally in Tab. 2, but it does not mention that the overall method is largely based on it. The submitted paper should mention that the overall operation of PBAPS is largely based on RIM, and explain how it differs from this existing work. Besides being good scientific practice, this enables readers to properly judge the significance of the contributions of the work, which is not possible in the current version.\n2.\tThe concept of top-down, hierarchical object-to-part segmentation – called “progressive” segmentation in this work – is not new, e.g., it is also done by ViRReq [b] and TAPPS [c]. The paper should acknowledge these works, and explain how the proposed top-down segmentation approach differs from these existing approaches, to clarify what the significance is of the presented contribution. This is currently not done.\n3.\tIn general, the paper does not clearly describe how the proposed method compares to existing OVPS methods. The paper does list several existing OVPS methods in L72-L87, and briefly summarizes how they work, but it does not contrast the overall operation of PBAPS to that of existing methods. As such, again, the significance of this work’s contributions cannot be judged fully.\n4.\tThe abstract (L4) claims that this work proposes the first training-free OVPS framework. However, in Tab. 2, there are two other methods (OVDiff and RIM) that are also training-free. Although these methods were not specifically designed for OVPS but for open-vocabulary semantic segmentation, they do work for the OVPS task. Therefore, the claim in L4 is not fully justified, and it would be better if it were removed or altered.\n5.\tSeveral sections of the paper are hard to understand and follow because they use (technical) terms that have not been defined or explained yet. This limits the readability and clarify of the paper. For example:\n\n   \n    -\tL6-L10 of the paper uses many technical terms to describe the method, e.g., “cross-hierarchy matching”, “matching cost divergence”, “deterministic-region context”. However, these are not commonly used terms, but terms that are introduced in this paper, in later sections. As a result, it is not clear what these terms mean, and the abstract is difficult to understand. The abstract would be much clearer if (a) the method were explained at a higher abstraction level or (b) if these terms were explained.\n\n   -\tSimilarly, L40-L46 of the introduction describes the BAR module, but this description refers to concepts from the overall method that have not been explained yet, e.g., “part prototypes”, “matching cost”, “original part masks”. As a result, this description cannot be properly understood yet. The clarity of the introduction could be improved by first describing the overall operation of the method (currently done in L47-L52) and introducing these concepts, before explaining the operation of the BAR module.\n\n   -\tSimilarly, L114-L118 refers to “deterministic regions” and “part prototypes”, concepts that have not been explained yet.\n\n6.\tAlthough the rough, overall operation of the method becomes clear when reading the paper, several aspects of the model are not explained in sufficient detail. This limits understandability and reproducibility of the work. For example:\n\n   -\tL130-L131 mentions that cross-attention maps are extracted. However, the paper does not explain how these are extracted. The submitted paper cites 3 other works in this sentence, but it is not clear what exact operation is used. \n   -\tL134 describes that DINOv2 extracts features from masked regions, but it is not clear if masked image is fed through DINOv2, or if the full image is fed, after which only the per-pixel DINOv2 features that belong to the part masks are used to generate the visual prototype.\n   -\tL135 makes it seem like there is only one visual prototype per class $c$, but L208 and L522 suggest that there are multiple subcategory prototypes per class. This is inconsistent. Are there multiple prototypes per class, or just a single one? This should be clarified. \n   -\tIn the cost divergence map $D_ {A,B}$ (L165-L166), pixels that have a low matching cost with *both* part $A$ and $B$ will also have a low divergence. This could happen at pixels outside of the mask of the parent class, where it is quite certain that both part classes are not present. In this case, there is clearly no semantic ambiguity, but there is still a low cost divergence. Hence, the statement that “low divergence values indicate regions where both parts exhibit similar matching costs, leading to semantic ambiguity” is not fully true. I assume, therefore, that there is some masking with the parent segmentation mask or initial part segmentation mask, but this is not mentioned or explained. This masking is only done for the deterministic regions $A_ {\\textrm{det}}$ and $B_ {\\textrm{det}}$ (Eq. 3), but not for the ambiguous regions $U$. This should be clarified.\n   -\tL204 mentions that “hundreds” of images are generated for each part class, but this is quite vague. Without knowing the exact number, the work is not reproducible.\n\n7.\tA limitation of the method, which is not acknowledged in the paper, is that the method does not scale well to a larger vocabulary. If the objective is to do segmentation with a truly 'open' vocabulary, then we would like to have a vocabulary of perhaps millions of concepts. CLIP-based (or other text-embedding-based) methods scale to this relatively well, as we would only have to feed the text description of these concepts through the text encoder, and we could use them directly for segmentation. In the presented PBAPS method, however, we would have to feed all these millions of concepts through Stable Diffusion, SAM, and DINOv2, generating hundreds of millions of images to obtain the visual prototypes, which is rather inefficient. Moreover, we would have to manually construct the hierarchical HPCGraph, which would become intractable at such a scale. I would encourage the authors to acknowledge and reflect on this limitation in the submitted work.\n\nThere are some minor additional weaknesses that do not significantly impact my rating:\n\n-\tL132 and L167: The symbol $A$ is used for two different purposes, which is confusing.\n-\tL217: the spaces should be removed, “7. 74” should be “7.74”, etc.\n-\tL238: “effectivenes” should be “effectiveness”\n\n[a] Wang et al., “Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation,” CVPR 2024.\n\n[b] Tang et al., “Visual Recognition by Request,” CVPR 2023.\n\n[c] De Geus et al., “Task-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations,” CVPR 2024.\n\nQUESTIONS:\nMy initial rating of this paper is “Borderline Reject”, but I am leaning towards “Reject”. Although the presented method obtains new state-of-the-art results for OVPS, I believe the paper contains too many flaws to be accepted in its current state. Importantly, the paper does not acknowledge the similarity of aspects of the presented method with existing work, making it difficult to judge how significant the contributions are. Moreover, the clarity of several parts of the paper is poor, and an important limitation is not addressed.\n\nIn the rebuttal, I would encourage the authors to attempt to address my concerns as stated in the “weaknesses” section, and specifically to (a) explain how the proposed method compares to existing work and (b) clarify the operation of the method.\n\nLIMITATIONS:\nThe paper lists some limitations of the work in the appendix, but does not mention the limitation of scalability. See weakness 7 in the 'Strengths and Weaknesses' part for more details.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a critical challenge in Open Vocabulary Part Segmentation (OVPS): the inaccurate delineation of boundaries between structurally connected parts. To solve this, the authors propose PBAPS, a novel training free framework capable of performing OVPS without any additional learning.\n\nPBAPS consists of three main components:\n\n- HPCGraph, which models the hierarchical relationships between objects and their parts\n\n- BAR (Boundary Aware Refinement), a module that detects ambiguous boundary regions and uses surrounding context to enhance feature representations and adapt visual prototypes\n\n- A training free framework that leverages pre-trained models such as Stable Diffusion, SAM, and DINOv2 to generate visual prototypes and perform segmentation purely through inference\n\nThe proposed method achieves state-of-the-art performance on three benchmarks: Pascal Part 116, ADE20K Part 234, and PartImageNet. It shows significant improvement over existing methods, especially in accurately segmenting structurally connected part boundaries.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper clearly identifies the fundamental cause of segmentation errors at structurally connected part boundaries, focusing on the intrinsic conflict between continuous visual features and discrete classification mechanisms, and proposes a solution to address this issue.\n\n- PBAPS is the first training-free framework in the OVPS field, enabling segmentation without any additional training. This approach demonstrates strong zero-shot generalization to unseen part classes.\n\n- The method achieves state-of-the-art mIoU and bIoU scores on three benchmarks (Pascal-Part-116, ADE20K-Part-234, and PartImageNet) surpassing both supervised and existing training-free methods by a significant margin.\n\nWeakness\n- Dependence on foundation models: Despite the advantage of being training-free, PBAPS heavily relies on large-scale pre-trained foundation models such as Stable Diffusion, SAM, and DINOv2 for visual prototype generation and feature extraction. As a result, its performance and applicability may be influenced by the quality, availability, and domain compatibility of these models.\n\n- Sensitivity to hyperparameters: The performance of the BAR module is sensitive to hyperparameters such as feature fusion weight, prototype adaptation coefficient, and number of sub-prototypes. Achieving optimal performance requires careful tuning, which can hinder the model’s generalizability and ease of deployment across diverse scenarios.\n\n- Manual construction of HPCGraph: As acknowledged by the authors, the construction of HPCGraph is not an automated process but rather a manual one, relying on predefined object-part hierarchies or external knowledge. This limits the direct applicability of PBAPS to new domains or complex object categories where such structural priors are not readily available.\n\n- Incomplete ablation study: The ablation study for the proposed method is conducted solely on the Pascal-Part-116 dataset. This presents a limitation in assessing the generalizability and robustness of individual components, as it remains unclear whether similar trends would hold across other datasets like ADE20K-Part-234 or PartImageNet.\n\nQUESTIONS:\n- In the Cross-Model Prototype Induction stage, the visual prototypes are entirely dependent on the localization capability of Stable Diffusion. However, Stable Diffusion does not always highlight regions that accurately correspond to the given input prompt. What is the likelihood that the resulting visual prototype actually represents the region corresponding to the prompt?\n- Additionally, the cross-attention maps in Stable Diffusion vary significantly depending on the timestep and the layer of the U-Net. From which timestep and which specific layer of the U-Net do you extract the attention maps for localization purposes?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of open-vocabulary part segmentation (OVPS), focusing on the challenge of segmenting structurally connected parts that lack clear visual boundaries. The authors argue that standard discrete classification struggles with these smooth transitions.\n\nTo mitigate this, they propose a Boundary-Aware Refinement (BAR) module that identifies ambiguous boundaries and refines them using nearby reliable regions and adaptive prototypes. They further design a training-free framework called PBAPS, which performs progressive refinement guided by a hierarchical graph of part relationships.\n\nThe method achieves strong performance on several benchmarks. While the use of multiple external models may introduce complexity, the approach is well-motivated and effectively tackles a core limitation in current OVPS systems.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The method is entirely training-free, avoiding the need for large-scale annotated data or fine-tuning.\n\n- The Boundary-Aware Refinement (BAR) module is a novel design that specifically targets the challenge of ambiguous part boundaries, a key limitation in existing OVPS methods.\n\n- The use of a hierarchical part graph to guide progressive refinement introduces a structured and interpretable approach to part segmentation.\n\n- The framework is modular and leverages existing vision-language models effectively without additional training overhead.\n\nWeaknesses:\n\n- The method relies on generating hundreds of prototype images per part class, which may introduce a risk of overfitting or inadvertently aligning too closely with the test set, especially on relatively small or simple datasets. It would be helpful to demonstrate the method on more complex or diverse scenarios\n\n- The paper lacks an analysis of how the similarity between generated prototypes and dataset images affects performance, which would be important to clarify whether the method generalizes or exploits dataset biases.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents PBAPS, a training-free framework for open-vocabulary part segmentation (OVPS) that addresses the inherent challenge of segmenting structurally connected part boundaries-regions where anatomically adjacent parts exhibit smooth, continuous visual transitions. The authors identify that the core difficulty arises from the mismatch between continuous image features and discrete classification mechanisms, leading to ambiguous boundaries. To resolve this issue, PBAPS introduces a progressive boundary-aware strategy built upon Cross-Model Prototype Induction, Hierarchical Part-Connected Graph and Boundary-Aware Refinement. Extensive experiments on Pascal-Part-116, ADE20K-Part-234, and PartImageNet demonstrate state-of-the-art performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nQuality: The paper presents a technically sound framework (PBAPS) for open-vocabulary part segmentation (OVPS). The methodology is well-structured, combining hierarchical reasoning (HPCGraph) with a novel Boundary-Aware Refinement (BAR) module. The experiments are comprehensive, covering three benchmarks (Pascal-Part-116, ADE20K-Part-234, PartImageNet) and demonstrating consistent outperformance over prior methods (e.g., +10% mIoU over PartCLIPSeg). The ablation studies validate the necessity of each component \nClarity: The paper is well-organized, with clear explanations of the BAR module’s four-step process (ambiguous region localization, feature optimization, prototype adaptation, reclassification). Figures effectively illustrate the boundary refinement pipeline. The distinction between structurally vs. non-structurally connected boundaries is rigorously analyzed via feature gradients, providing theoretical grounding.\n\nSignificance: The work addresses a critical gap in OVPS—structurally connected part boundaries—which prior methods struggle with due to smooth feature transitions.\nOriginality: While cross-model prototype induction resembles RIM’s pipeline, the novelty lies in adapting it for part-level segmentation and introducing BAR to resolve boundary ambiguities via cost-divergence maps and context-driven refinement. The HPCGraph’s hierarchical design (object → part → subpart) is tailored to OVPS, distinguishing it from prior OVSS methods.\n\n\nWeaknesses\nOver-smoothing in BAR \nBAR averages context from “certain” pixels to refine boundaries, but these pixels may themselves be ambiguous on gradual transitions, weakening discriminative cues.\n HPCGraph static dependency \nThe hand-crafted hierarchy (e.g., cat → head → eye) cannot generalize to objects with irregular or novel part topologies (e.g., modular furniture).\n\nMissing comparison to supervised open-vocabulary part detectors\nThe experimental baseline set omits recent strong supervised open-vocabulary detectors that can be adapted for part localization. Without this comparison, the paper cannot fully substantiate the added value of its training-free approach over competitive fine-tuned alternatives.\n\nInadequate ablation separating hierarchical reasoning and boundary refinement\nThe ablation study (Table 3) entangles the contributions of (i) the hierarchical HPCGraph and (ii) the BAR boundary-refinement module. A crucial baseline—BAR applied directly to flat DINOv2 features without the hierarchy—is absent, preventing an isolated assessment of each component’s impact.\n\nQUESTIONS:\n1. Contamination from blended-part features in the “deterministic” pixels of BAR  \nBAR refines ambiguous boundaries by aggregating context from “deterministic” regions, yet gradual transitions may still introduce mixed-part features into these pixels. Could the authors provide  qualitative visualizations  that demonstrate this blending does **not** materially degrade final segmentation accuracy?\n2. Dynamic extensibility of a static HPCGraph  \nHand-crafted hierarchies (e.g., cat → head → eye) struggle with irregular or novel topologies (e.g., modular furniture). Have the authors considered a lightweight mechanism that allows the hierarchy to adapt at inference time while preserving the training-free constraint?\n3. Additional supervised open-vocabulary part-detection baseline  \nRecent supervised open-vocabulary detectors can be minimally adapted for part-level segmentation. Please report the  mIoU/bIoU after fine-tuning at least one such model on the same part vocabulary; if PBAPS still significantly outperforms, this will strengthen its training-free advantage.\n4. Ablation decoupling hierarchical reasoning from boundary refinement  \nThe current ablation (Table 3) does not disentangle the contributions of the hierarchy and BAR. Please add a  flat baseline that applies BAR directly to DINOv2 pixel features  without HPCGraph.\n\nLIMITATIONS:\nPlease see weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel training-free Open-Vocabulary Part Segmentation (OVPS) framework called PBAPS, addressing the inaccuracies in segmenting structurally connected part boundaries caused by the conflict between continuous image features and discrete classification mechanisms. PBAPS employs a progressive segmentation strategy guided by a Hierarchical Part Connected Graph (HPCGraph) and introduces a Boundary-Aware Refinement (BAR) module. The BAR module locates ambiguous boundary regions, optimizes pixel features, and adaptively refines visual prototypes for precise reclassification. Experimental results demonstrate that PBAPS significantly outperforms state-of-the-art methods on multiple benchmark datasets, showcasing its superior performance, especially in handling fine-grained part boundaries.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1. This paper proposes a framework that successfully leverages pre-trained foundation models for a complex downstream task without any fine-tuning, demonstrating excellent zero-shot generalization and highlighting a promising research direction.\n2. The method achieves impressive SOTA results on three challenging benchmarks.\nWeakness:\n1. The paper only showcases successful examples. A discussion of failure cases would provide a more balanced perspective on the method's limitations and offer crucial insights for future work.\n2. The paper lacks a clear justification for how key hyperparameters were chosen without access to the test set. Furthermore, their robustness and generalizability across different datasets are not discussed.\n3. The method's reliance on a manually constructed Hierarchical Part Connected Graph (HPCGraph) is a major bottleneck, raising significant concerns about its scalability and applicability to novel object categories not defined in advance.\n\nQUESTIONS:\nsee weakness above\n\nLIMITATIONS:\nsee weakness above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-21"}