Title: PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding
Venue: NeurIPS 2025
Accepted: yes
Keywords: part segmentation

part detection

vision language

large multimodal models
Abstract: Real-world objects are composed of distinctive, object-specific parts. Identifying these parts is key to performing fine-grained, compositional reasoning—yet, large multimodal models (LMMs) struggle to perform this seemingly straightforward task. In this work, we introduce PARTONOMY, an LMM benchmark designed for pixel-level part grounding. We construct PARTONOMY from existing part datasets and our own rigorously annotated set of images, encompassing 862 parts and 5346
objects for evaluation. Unlike existing datasets that simply ask models to identify generic parts, PARTONOMY utilizes highly technical concepts and challenges models to compare objects’ parts, consider part-whole relationships, and justify textual predictions with visual segmentations. Our experiments demonstrate significant limitations in state-of-the-art LMMs (e.g., LISA-13B achieves only 5.9% gIoU), highlighting a critical gap in their part grounding abilities. We note that existing segmentation-enabled LMMs (segmenting LMMs) have two key architectural shortcomings: they use special [SEG] tokens not seen during pretraining which induce distribution shift, and they discard predicted segmentations instead of using past predictions to guide future ones. To address these deficiencies, we train several part-centric LMMs and propose PLUM, a novel segmenting LMM that utilizes span tagging instead of segmentation tokens and that conditions on prior predictions in a feedback loop. We find that pretrained PLUM dominates existing segmenting LMMs on reasoning segmentation, VQA, and visual hallucination benchmarks. In addition, PLUM finetuned on our proposed Explanatory Part Segmentation task is competitive with segmenting LMMs trained on significantly more segmentation data. Our work opens up new avenues towards enabling fine-grained, grounded visual understanding in LMMs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a key limitation of large multimodal models (LMMs): their inability to perform fine-grained part-level visual understanding. They propose a dataset/benchmark PARTONOMY  for \"Explanatory Part Segmentation,\" evaluating LMMs on identifying object parts, comparing parts across objects, and reasoning about part-whole relationships. It spans 787 objects and 2,768 part labels, with diverse coverage beyond prior datasets. A part-level segmenting LMM PLUM is proposed to improve part-level understanding by avoiding special [SEG] tokens (via span tagging) and using a feedback loop to refine segmentations with prior masks. PLUM outperforms state-of-the-art LMMs on part segmentation tasks while retaining strong performance on general vision-language tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The benchmark and model are rigorously designed. PARTONOMY-Core’s manual annotation ensures high-quality part labels, and PLUM’s architecture (span tagging, feedback loop) is theoretically motivated to mitigate distribution shift and improve segmentation consistency.\n2. Part-level understanding is a foundational yet underexplored challenge for LMMs. PARTONOMY fills a critical need for standardized evaluation, while PLUM provides a novel architectural paradigm to improve grounding without sacrificing general reasoning.\n3. The span tagging mechanism (replacing [SEG] tokens) and mask feedback loop are innovative solution.\n4. The paper is well-written and easy to follow.\n\n\nWeaknesses: \n1. The FiLM-based feedback loop’s implementation details are vague. For example, how prior masks are encoded into feature maps and pooled via \"patch-wise attention\" is not fully explained.\n2. PLUM’s feedback loop and span tagging may increase computational cost, limiting real-world deployment.\n\nQUESTIONS:\n1. PLUM’s feedback loop uses FiLM layers to condition on prior masks. How are multiple prior masks (e.g., for overlapping parts) weighted or disambiguated? Does the loop ever propagate errors (e.g., a poorly segmented first part worsening subsequent masks)?\n2. The authors use Llama 2-70B to generate part labels for PARTONOMY-Core, then manually prune \"non-visible\" parts. How were disagreements between annotators resolved? Was inter-annotator agreement measured?\n\nLIMITATIONS:\nYes; the authors partially addressed limitations in the paper. For additional limitations, please refer to the \"Weaknesses\" section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new benchmark called PARTONOMY, which requires Large Multimodal Models (LMMs) not only to identify the generic parts of an object but also to learn the part-whole relationships between entities. Experiments on baseline models such as LISA show that current methods fail on this benchmark, thereby highlighting the gap in their part-grounding abilities. Furthermore, the authors address two weaknesses of existing LMMs in segmentation tasks: the need for an additional special token and the auto-regressive paradigm. They propose a new LMM called PLUM that uses span tagging rather than a special token and conditions its predictions on prior ones in a feedback loop. Several experiments on different tasks show PLUM's outstanding performance.\n\nSTRENGTHS AND WEAKNESSES:\n## Stengths\n\n1. The proposed PARTONOMY benchmark provides a valuable research topic and resource for the community. It effectively demonstrates a key weakness of existing LMMs: their failure to identify part-whole relationships.\n\n2. The observation of the two weaknesses is insightful and correct. The proposed methods address these issues to a certain extent.\n\n3. Using span tagging instead of a `[SEG]` token avoids the need to train an additional special token, which can be beneficial for fine-tuning with limited data.\n\n4. Introducing past information as a condition is a good idea to enhance the segmentation ability of LMMs.\n\n5. Experiments show that PLUM achieves better performance on various tasks, even when using less data compared to its competitors.\n\n## Weaknesses\n\n1. A dataset of 1k images may seem insufficient for a benchmark. I am wondering if the authors can prove that 1k images are enough to robustly evaluate LMMs. Specifically, the authors could evaluate models using varying numbers of images from PARTONOMY and plot a performance curve to determine the point at which the results converge, thereby demonstrating the sufficiency of the benchmark size.\n\n2. The second problem addressed by the authors is very similar to that in paper [1], which also introduces entity relationships into LMMs. It would be better to discuss the relationship with this paper and include it as a baseline for comparison.\n\n3. PLUM designs a Mask Feedback Loop to use historical masks as conditions to enhance future predictions; however, this inevitably introduces error accumulation. Specifically, if the model outputs a wrong or inaccurate mask, subsequent predictions will be dynamically affected. The paper should discuss this phenomenon.\n\n4. The authors frame this as an entirely new task. However, it could essentially be considered a more challenging variant of \"Referring Expression Segmentation\" that involves complex reasoning, or a form of \"Multi-object Segmentation.\" The core objective remains the segmentation of specific image regions based on textual instructions. While the question formats (e.g., Part Comparison, Part-Whole Reasoning) are well-designed, defining this as a completely new and independent task category may be an overstatement.\n\n[1] SegLLM: Multi-round Reasoning Segmentation with Large Language Models\n\nQUESTIONS:\nPLUM generates part masks serially, one by one. It is questionable whether this is the most efficient approach. For a prompt like, \"Please segment the airplane's wings, engines, and tail,\" could the model first fully comprehend all target parts from the text and then generate all masks in parallel? Parallel decoding could potentially be faster and might avoid the error accumulation issue mentioned above. The authors should provide a stronger justification for adopting a serial generation approach.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates fine-grained part understanding with large multimodal models (LMMs). In doing so, the authors first introduce a pixel-level part grounding benchmark, PARTONOMY, which contains a training split and a test split, aiming to examine the capabilities of LMMs for part grounding, part comparison, and part-whole relationship reasoning. Moreover, they identify two issues with existing part grounding models: a distribution shift due to the reliance on a special segmentation token and a lack of modeling history part groundings. They further address the issues using an improved pre-training strategy and deliver a more performant LMM: PLUM. The experimental results demonstrate the superior performance of PLUM.\n\nSTRENGTHS AND WEAKNESSES:\n## strengths\n\n1. The paper addresses an important problem: pixel-level part understanding, which has been largely overlooked in LMM research.\n2. The authors introduce a benchmark and a set of evaluation metrics for part understanding.\n3. A new training paradigm is proposed to address the limitations of existing segmentation LMMs in part grounding.\n4. A new LMM, PLUM, which is trained using the proposed learning paradigm, is delivered and further demonstrates strong part-understanding performance.\n\n## weeknesses\n\n1. The chosen baseline open-vocabulary segmentation models seem outdated. Here are some more recent models: Grounded SAM/DINO: https://github.com/IDEA-Research/Grounded-SAM-2; https://arxiv.org/abs/2412.16334v1\n2. Lacking comparison with frontier models like GPT-4o. A potentially stronger baseline could be a combination of pre-trained VLMs for visual reasoning and Grounded SAM/DINO for part grounding. Including this baseline can better justify the superiority of the proposed unified model.\n3. Lacking an in-depth analysis of the span extractor. The incorporation of a specific span extraction module might complicate the model design. VLM should be able to identify segmentation-related text spans or can be trained to predict them directly.\n\nQUESTIONS:\nText evaluation in Table 3 shows marginal performance gaps between different models. Does that mean the answer choices construction is suboptimal? The authors argue that \"each answer choice shares extensive lexical overlap.\" Is it possible to quantify this, and are there any plans to resolve this issue?\n\nLIMITATIONS:\n1. \\# images and segmentation masks in Table 1 seem missing.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Partonomy, a new pixel-level part grounding benchmark designed to evaluate the ability of models to reason about part-whole relationships. The dataset covers a broad range of object categories (787 in total), including common items, biological entities, and specialized objects, and is annotated with 1,157 fine-grained part labels. A key feature of this benchmark is its requirement for models to not only recognize part-whole relationships via textual description, but also justify them through precise visual segmentations.\n\nBuilding upon this dataset, the authors propose PLUM, a segmentation-enabled large multimodal model (segmenting LMM) that employs span tagging to avoid distribution shift caused by [SEG] token. Moreover, it refines the segmentation prediction via a feedback loop. \n\nExperimental results demonstrate that existing state-of-the-art LMMs exhibit limited capabilities in fine-grained part grounding. In contrast, the proposed PLUM model shows compelling performance on the Partonomy benchmark as well as other evaluation tasks involving reasoning, segmentation, VQA, and visual hallucination.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe proposed explanatory part segmentation task and the corresponding Partonomy dataset are valuable contributions. Also, the taxonomy introduced for the explanatory part segmentation task is comprehensive and well-structured.\n\n**Weaknesses:**\n\nMy primary concerns span the areas of data, method, experiments, and clarity. Specific concerns are as follows:\n\n**Data:**\n\n- **W1. Lack of Demo Samples from Partonomy / Partonomy-Core.** Given that the explanatory part segmentation task and the Partonomy dataset are the core contributions of the paper, it is essential to provide representative samples in the supplementary materials to help readers better understand their details and annotation quality.\n\n**Method:**\n\n- **W2. Unverified Claim Regarding Bidirectional Attention (L220–221).** The paper claims that “bidirectional attention is critical for reliable BIO span tagging” but provides no theoretical justification or empirical evidence to support this claim.\n- **W3. Unclear Roles of the Components in Mask Feedback Loop.** The proposed mask feedback loop involves additional components to the SAM decoder, including Feature-wise Linear Modulation (FiLM) and Patch-wise Attention Pool. However, it is still unclear why these two specific techniques are involved and how they influence the overall performance. Can FiLM be replaced by alternatives, such as a standard cross-attention mechanism?\n- **W4. Use of Focal-Tversky Loss Lacks Justification.** As all baseline methods are evaluated using Dice loss while PLUM uses a different loss, it is unclear whether observed performance gains stem from architectural innovations or the loss function itself. A comparative study using Dice loss within PLUM is necessary to isolate the contribution of each component.\n\n**Ablation Study:** \n\n- **W5. Incomplete Ablation of Core Components.** The effectiveness of PLUM is attributed to three key components: Span Extractor, KL Constraint, and Mask Feedback Loop. However, a comprehensive ablation across all 8 combinations of these components is missing. The ablation study should be conducted not only on Partonomy-Core but also across the other evaluation tasks (e.g., reasoning segmentation, VQA, and hallucination detection).\n- **W6. Lack of Analysis on Loss Weights.** The paper explores the KL constraint weight ($\\lambda_2$) but does not examine the influence of other loss terms, particularly span extraction loss ($L_{span}$) and segmentation loss ($L_{segm}$). A more thorough loss weight sensitivity analysis is needed.\n\n**Other Experiments:**\n\n- **W7. No Evaluation of Span Extractor’s {B, I, O} Classification.** The quality of the segmentation largely depends on accurate BIO tagging from the Span Extractor. However, no explicit evaluation (e.g., accuracy, F1 score) is provided for this three-way classification task.\n- **W8. Misleading Conclusion in L268–273.** The authors claim that “Table 2 shows that predicting the object label first prior to generating the part segmentation masks leads to better mask prediction performance”, but this trend does not consistently hold across methods like PLUM (ft), GLaMM (ft), and PixelLM-13B (ft), which contradict the stated conclusion.\n- **W9. No Discussion of Results in Figure 4.** The experimental setup and the results shown in Figure 4 are not discussed, leaving the reader unclear about its purpose.\n- **W10. Lack of Insightful Analysis in Table 3.** The conclusion that \"the proposed experimental setting dampens model-to-model variance\" (L265–267) is vague and uninformative. Further analysis is required to understand whether this variance is caused by experimental setup, model architecture, or other confounding factors.\n\n**Clarity:**\n\n- **W11. Unsubstantiated Claim About Training Data Size (L22).** The paper claims that other segmentation-capable LMMs are trained on significantly more data, but no concrete evidence or statistics are provided to support this statement. Details should be included.\n- **W12. Missing Metric Descriptions in Table 3.** Table 3 presents several quantitative results without describing the specific calculation evaluation metrics.\n\n**Minor:**\n\n- **L227–228:** The functions $g(\\cdot)$ and $N^+$ are introduced without any definitions or explanations.\n- All figures should be rendered as vector graphics to enhance clarity and readability, particularly for publication purposes.\n\nQUESTIONS:\nPlease review the weaknesses outlined above. In particular, I would like to highlight the following four key questions:\n- Could you provide theoretical justification or empirical evidence to support the claim that “compared to causal attention, bidirectional attention is more critical for reliable BIO span tagging”?\n- Could you conduct a fair comparison using Dice loss within PLUM to isolate the impact of the proposed Focal-Tversky Loss?\n- Could you provide a comprehensive ablation study of the three core components in PLUM: Span Extractor, KL Constraint, and Mask Feedback Loop?\n- Could you include an evaluation of the Span Extractor’s classification performance for the {B, I, O} labels?\n\nLIMITATIONS:\nNo, the authors did not discuss the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-11"}