PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding
Authors: Ansel Blume
Jeonghwan Kim
Hyeonjeong Ha
Elen Chatikyan
Xiaomeng Jin
Khanh Duy Nguyen
Nanyun Peng
Kai-Wei Chang
Derek Hoiem
Heng Ji
Venue: NeurIPS 2025 spotlight ()
Keywords: part segmentation
part detection
vision language
large multimodal models
Primary area: applications
TL;DR: Introducing PARTONOMY and PLUM, a new benchmark and segmenting LMM that enable fine-grained, part-level visual reasoning by addressing architectural flaws in existing LMMs and setting a new standard for grounded multimodal understanding.

Abstract: Real-world objects are composed of distinctive, object-specific parts. Identifying these parts is key to performing fine-grained, compositional reasoning—yet, large multimodal models (LMMs) struggle to perform this seemingly straightforward task. In this work, we introduce PARTONOMY, an LMM benchmark designed for pixel-level part grounding. We construct PARTONOMY from existing part datasets and our own rigorously annotated set of images, encompassing 862 parts and 5346
objects for evaluation. Unlike existing datasets that simply ask models to identify generic parts, PARTONOMY utilizes highly technical concepts and challenges models to compare objects’ parts, consider part-whole relationships, and justify textual predictions with visual segmentations. Our experiments demonstrate significant limitations in state-of-the-art LMMs (e.g., LISA-13B achieves only 5.9% gIoU), highlighting a critical gap in their part grounding abilities. We note that existing segmentation-enabled LMMs (segmenting LMMs) have two key architectural shortcomings: they use special [SEG] tokens not seen during pretraining which induce distribution shift, and they discard predicted segmentations instead of using past predictions to guide future ones. To address these deficiencies, we train several part-centric LMMs and propose PLUM, a novel segmenting LMM that utilizes span tagging instead of segmentation tokens and that conditions on prior predictions in a feedback loop. We find that pretrained PLUM dominates existing segmenting LMMs on reasoning segmentation, VQA, and visual hallucination benchmarks. In addition, PLUM finetuned on our proposed Explanatory Part Segmentation task is competitive with segmenting LMMs trained on significantly more segmentation data. Our work opens up new avenues towards enabling fine-grained, grounded visual understanding in LMMs.

PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding
Ansel Blume Jeonghwan Kimls, Hyeonjeong Ha', Elen Chatikyan's Xiaomeng Jin', Khanh Duy Nguyen', Nanyun Peng", Kai-Wei Chang", Derek Hoiem', Heng Ji 'nnivesiity of Illinois Urbana- Chamma--haaaign, UUniversity of California Los Angeles {blume5, jk100, hengji)@illinois edu
Abstract
Real-world objects are composed of distinctive, object-specific parts. Identifying these parts is key to performing fine-grained, compositional reasoning- large multimodal models (LMMs) struggle perform this seemingly straightforward task. In this work, we introduce PARTONOMY, an LMM benchmark designed for pixel-level part grounding. We construct PARTONOMY from existing part datasets and our own rigorously annotated set of images, encompassing 862 part labels and 534 object labels for evaluation. Unlike existing datasets that simply ask models to identify generic parts, PARTONOMY uses specialized concepts (e.g.. agricultural airplane), and challenges models to compare objects' parts, consider part-whole relationships, and justify textual predictions with visual segmentations. Our experiments demonstrate significant limitations state-of-the-art LMMs (e.g.. LISA-13B achieves only 5.9% gloU), highlighting critical gap in their part grounding abilities. We note that existing segmentatioon-enablel LMMs (segment- ing LMMs) have two key architectural shorteomings: they use special [SEG] tokens not seen during pretraining which induce distribution shift, and they discard predieted segmentations instead of using past predictions to guide future ones. To address these deficiencies, we propose PLUM, a novel segmenting LMM that uses span tagging instead of segmentation tokens and that conditions on prior predictions in feedback loop. We find that pretrained PLUM outperforms existing segmenting LMMs on reasoning segmentation, VQA, and visual hallucination benchmarks. In addition, PLUM finetuned our proposed Explanatory Part Seg- mentation task is competitive with segmenting LMMs trained on significantly more segmentation data. Our work opens up new avenues towards enabling fine-graines grounded visual understanding in LMMs. The code and data are publicly available at: httpp::/github..oom/ http:////...oooee////e///rttoooo
1 Introduction
Real-world objects can be decomposed into distinctive parts. banana boat (Fig. 2). for instance, consists of a seating tube, handle. hull, and an inflation valve. Such parts characterize each concept, differentiating one object from another. The ability to recognize and distinguish between parts an important element of holistic object understanding, with applications ranging from explainable object recognition
6 g 58 to part-based novel concept design [10], and robotic manipulation Decomposing objects into their key building blocks allows models to reason about
[52 30 objects granular level [6 29), allowing for more complex and nuanced interactions.
[4]
Unfortunately, Large Multimodal Models (LMMs),, the backbones of today's 's multimodal systems, lack strong part recognition abilities [16,
29
33 While they perform well on visual reasoning
43
13
42
and visual hallucination tasks 23 we find that they are unable to accurately identify
Equal contribution.
39th Conference Neural Information Processing Systems (NeurIPS 2025).

---

Port identifiction
Partto-Whole
Wholeto-Part
arpiose is the image low!
What
agrizatural
What purts does this object lore?
nome object
fixed landing gear, propulaton
"The sirplns the
sproping rig... .... landing gear.
Thi is forpedo
comparent spraying rig and vings.
Okty, what parts torpedo?
Intersection
fvel guidence system
What visble parts does Has benone common farking
Ckay, parts, okjjctt
agricshrre aiplore
Parfonamy Instonce Construction through Quesfion-Asswer Mutation
Differance
Queafin Asswer
Matcled Acoweez
Matatice Opanations
hull tobe. has padele
What
prrts basasa the conce??
kove which fishing best dons notf
Ai inflation valve and poddle.
hes kandle infiatio valve sealing Kobe
(Gun)
Godyl
Flod landing
Seresing Fla
Wings
Figure I: The PARTONOMY dataset evaluates LMMs' part understanding through the Explanatory Purt Segmentation task. Given an input image, explanation and generates part segmentation masks which serve as textual and visual rationale for its answer choice. Our question-answer mutation framework generates challenging answer choices by predicting part co-occurrence and by selecting parts from confusable objects.
LMM selects textual
object parts in an image, occasionally regurgitating object parts memorized from text-only pre-training (e.g. "a fish must have a fin")) Worse, LMMs that can generate segmentation masks, segmenting LMMs [99 37 41 44]. lack the ability to ground these fine-grained regions despite being trained on part segmentation data. This severely limits LMMs' utility in real-world scenarios that require fine-grained, part-level understanding.
To quantify the part recognition abilities of LMMs, we propose Explanatory Part Segmentation, a task that assesses LMMs' ability to recognize object parts, associate objects with their distinctive parts, and use these grounded parts predict object labels. We then introduce PARTONOMY, comprehensive benchmark for the Explanatory Part Segmentation task. We construct PARTONOMY from existing part segmentation datasets [1
35 and our manually-annotated evaluation dataset of IK specialized object-centric images with complex part annotations. This subset, PARTONOMY-COM contains 862 distinct part labels-more than any existing part datasets (Tabll 1).
We then note two shortcomings of existing segmenting LMMs' architectures 19 37 41
First, they always rely on special [SEG] tokens not seen during pretraining, potentially hindering downstream performance by introducing distribution shift. Second, these segmenting LMMs discard their predic- tions after each output, missing the opportunity to incorporate prior information contained by the masks they predicted during the decoding process. This design is contrast to modern generative frameworks, which condition future predictions on past ones [48] 44 Based on these observations, propose PLUM,, Part-Level Understanding L.M. PLUM uses text span tagging module to avoid special segmentation tokens that induce distribution shift from the pre-trained LLM, and employs a mask feedback mechanism to condition past predictions (Section 4). Our results show that pretrained PLUM retains its general reasoning abilities far better than other segmenting LMMs. achieving stronger zero-shot segmentation performance and competitive finetuned performance to models trained on significantly more segmentation data.
2 Related Work
Reasoning in Large Multimodal Models Reasoning capabilities in Large Language Models (LLMMs) uncovered by prompting techniques such Chain-of-Thoughh (CoTT) [51 [8] have led to increased interest in their application LMMs [1
55 12]. Previous work shows that reasoning
abilities of L.LMs allow them to generate textual rationales given image inputs, allowing them to handle complex visual reasoning tasks such A-OKVQA (42) and ScienceQA [31]. Nonetheless, LLMS output space confined to text, limiting their spatial understanding and often leading to hallucinatory text outputs [26]. While recent efforts on visual compositional reasoning attempt to mitigate the gap between the text and image modalities in L.MSs [54 55
32 by using external
modules such as object detectors [6] or code interpreters [46], most attempts don't 't truly reflect the
innate visual reasoning capabilities of LMMs. Our proposed model and recent line of LMMs

---

Paddle
PLUM
PixeLM
LISA
GLaMM
Figure 2: An example of PLUM's part understanding compared to recent segmenting LMMs trained on part data.
19 37 try accomplish this by interleaving textual and visual rationale generated through
segmentations.
Segmentation-Enabbl Large Multimodal Models LMMs such as LISA 19 and GLaMM [37 have demonstrated the ability to generate text and grounded segmentation masks. Despite being trained on part-level segmentation datasets such as PACO [35 and Pascal-Part [9], they struggle to exhibit part-level understanding of visual concepts. While they demonstrate the ability to understand current LMMs fail relate concept- concept-indicattvy indicative parts to their wholes, as shown in Fig. 2 Frequently, they fail to generate the specialized segmentation token (e.g., [SEG]) added their vocabulary. leading
complex textual instructions [19 37 53
masks being generated for the parts. Even state-of-the-arr LMMs that seemingly "reason" struggle to establish attributive relationships between objects and parts, implying that current models and datasets lack the coverage and capacity to handle complex part understanding and grounding tasks. This observation motivates the proposal of new Explanatory Part Segmentation task, which requires LMMss to segment objects' parts (i.e. producing visual rationale) while generating the corresponding text rationale.
Part Semantic Segmentation Part segmentation is the task of decomposing objects into their 56
11 35) While this task has been studied open- vocabulary [24] 25 59 40) and multiple segmentation tasks (20], no existing work has evaluated parts-those those that help define the object category. In fact, most recent efforts on segmentation-enable LMMs focus on concept labels or referring expressions (37), ignoring part segmentation altogether. Our PARTONOMY dataset, which includes our manually annotated PARTONOMY -Crre evaluation set, integrates existing pert-level segmentation datasets such PACO 35 and PartImagenet [1] to further the part and object-level diversity of our benchmark.
constituent parts through segmentation 7
57
LMMs' ability to segment objects' "concept-indicatte
3 PARTONOMY: A Dataset for Explanatory Part Segmentation
3.1 Task Overview
We motivate our task definition by characterizing a model with part understanding. First, such a model should be able identify parts. Given image of object, the model can list visible parts and ground them in the image. Second, this model should be able to compare and associate object parts. It recognizes that both dogs and tables have legs, despite their difference in form. On the other hand, understands that a passenger plane and biplane both have wings. but that the biplane's double wings are feature that distinguishes it from other aircraft. Finally, this model should be able to use its part knowledge to predict object labels based their parts. Identifying key feature, like large scope, suggests to the model that rifle is likely a sniper rather than assault rifle.
To evaluate these elements part understanding, we define the Explanatory Part Segmentation task. In this task, model is provided with image and question about an object's parts (c.g.

---

Table 1: Comparison between part segmentation datasets. indicates usage in PARTONOMY. "C" refers to common objects (e.g. chair, airplane), "O" organisms (e.g. dogs, snake), and "S" to specialized objects (c.g, intersecting lines, highway map. fighter jet). PARTONOMY- Core has over three times many object labels and four times many part labels the widely used PACO dataset, has more part labels than PartImageNet++ (which has twice as many object labels), and contains specialized object parts annotated on object-centric images.
Object-Centric Images
Object Domain
Datasets
Object Labeks
Part Labels
Object-Paart Labek
Images
Seg. Masks
PASCAL-PPan
10,103
111,960
C,
PartImageNet
158
14
14
24,000
112,000
C,O
PurlmageNeree+
1000
818
3,308
100K
406.4K
/
C,O
PACOT
75
84.027
641,000
PARTONOMY-COO Core!
534
862
1,976
1,068
4,968
C. O.S
PARTONOMY
606
975
2.507
74,500
407.101
O.
"What visible parts does the agricultural plane in the image have?" Fig. 1p. The model must then select the best response and generate segmentation masks for the corresponding parts explain its selection (e.g. "The agricultural plane has wings, a propulsion component, and spraying rig" Motivated by characterization of part understanding, define three classes of questions (Fig. I):
Part Identification questions ask the LMM identify then segment object's visible parts. These questions test LMMs' ability to recognize and ground parts without hallucination.
Part Comparison questions ask the LMM to identify an object's 's visible parts and compare or contrast them to the parts of another object. These questions test models' knowledge of objects' common parts. Concretely, let Pi and Pc be the parts of an object in the image and the parts of a separate comparative concept. We define two subtasks:
Part Intersection. The model is asked which visible parts the object in the image has in common with specified query concept, Pr n Pc, then segments them.
Part Difference. The model asked which visible parts the object in the image has which the query concept does not, P1 Pc. then segments them.
Part-Whole Reasoning asks the LMM to identify an object or its parts as a consequence of the other. These questions assess whether the model can apply its part knowledge to identify objects, or use an object identify its parts. Subtasks include:
Whole: The model is asked to identify and segment object's visible parts, and based on the predicted parts, determine the object label.
Whole-to-Part The model is asked identify the object in the image, and based on the predicted object, identify and segment its visible parts.
The subtasks assess decomposable object recognition, where object and its parts each provide evidence for the other's identity.
3.2 Dataset Construction
We introduce the PARTONOMY dataset to facilitate training and evaluation on Explanatory Part Segmentation. PARTONOMY consists of three training and evaluation PARRTots-P CATTNOMMY-Partmageeht
and PA PARTONOMY-PASE PASCAL Part-which are constructed from their respective datasets' part annotations [1 We further contribute evaluation-only subset of 1K images of domain-specine specific objects, which we term PARTONOMY-Core
PARTONOMY-Core Construction. To construct the PARTONOMY-Core ontology, we start from broad object categories containing decomposable objects for example, airplanes, garden tools, weapons, and boats (details on dataset construction in the Appendix). We then manually select objects which provide category coverage and which have readily identifiable parts.
With object classes selected, we the Bing search API to download preliminary set of object images, and prompt an LLM (Llama 2-70B (47]) to generate part names for each object which are visible and specific to that object category. We manually review each object's assigned parts,

---

removing those which are not outwardly visible or are not commonly found on the object. Part annotation proceeds using combination of CV and mask annotation interface we developed to streamline the annotation process from multiple annotators" Parts are further refined and pruned during the annotation process depending on their visibility and frequency.
Explanatory Part Segmentation Data Generation Pipeline. Our Explanatory Part Segmentation data generation pipeline is applicable to any part dataset containing object names, part segmentations, and part names. We start with the ground truth set of object parts for each image and format these in natural language as the parts the model must identify and ground. For Part Comparison questions, we sample a separate object class with parts in common with those in the image, then intersect (for Part Intersection) or subtract (for Part Difference) its parts to form the ground truth set of parts.
After constructing the answer choice with the ground-truth parts, we create incorrect answer choices for each question. We adopt an answer mutation framework to generate plausible, challenging wrong answers. For set of ground truth parts, we repeatedly apply mutation operations which add, remove, or replace an existing part. This process keeps wrong answers close to the original require deep part understanding of the evaluated model. Instead of randomly sampling parts for mutation which could result in unrelated part additions (c.g. "The airplane in the image has wings, a row of windows, and an ice cream cone")-we select those most related to the existing parts and object. We train logistic regressors on part co-occurrence to predict likely parts given the current set of parts, and restrict wrong answer parts those from the same object category, if available (e.g., wrong parts for an airplane come from other airplanes' parts). Challenging wrong object answers for Part- Whole Reasoning questions are sampled in similar way, selecting objects with high semantic similarity to the ground truth object as measured by word embeddings (we use Sentence Transformers 33 to measure similarity).
Differences from Existing Datasets. PARTONOMY is the only part segmentation dataset designed for use with VLMs, testing not only part identification but also part reasoning and grounding. The data generation pipeline's 's extensibility and capacity to generate challenging questions serve as an important asset for future part-based pretraining and evaluation.
LATTNNOM-Cor has, to our knowledge, the most part classes of any part segmentation dataset, with four times the number of part labels of the widely used PACO dataset (35) and more part labels than PartlmggeNet+ [22] which has twice as many object classes. It has object-centric images for consistent evaluation, unlike datasets like PACO, which frequently have partially occluded objects. PARTONOMY- -Cree lightweight evaluate on, with only 1K images, but covers wide range of concepts with balanced instances (2 images per object), more than any dataset other than PartImageNet+++. These qualities, coupled with PARTONOMY-COre 's use of technical domains with highly-specific objects (c.g., electric coffee grinder, city map, and combat drone), make a unique contribution for part segmentation evaluations.
3.3 Evaluation
Explanatory Part Segmentation requires models to choose the correct textual response and segment parts in the image.
To evaluate textual part predictions, we prefer a multiple choice over a generative setting to avoid ambiguities in phrasing bhr., where the model identifies clip but the annotations list the part as a clamp- and incomplete part annotations, e.g., where the model identifies a valid part that isn't 't annotated. We provide the model with one correct and four incorrect answer choices. Incorrect choices either include non-visible parts or lack visible parts present in the correct response. We select the predicted answer choice via language modeling probability, as is common in VQA 21 For Part-Whole Reasoning questions, we select an answer choice twice in sequence: once to predict the set of parts, and once to predict the object.
Text Evaluation
We evaluate answer selection via accuracy (random = 20% for 5 answer choices) for both part prediction (all questions) and object prediction (Part-Whole Reasoning questions). However, some wrong answer choices are better than others. Precision and recall capture the similarity of the predicted parts to the ground truth set, and we adopt these as more fine-grained measurements of part recognition by the LMMs.
"This interface and the dataset generation pipeline will be released to facilitate future dataset construction.
5

---

Poodback
Downsompied
Neak Enceder + FILM
Wask Enceder PILM
FILM
RUl
Ridoorieed Sall Alteeatte Klack Rl Porroded
Layerierm Convolation
Layerierm
wade
Convelation
Legasargoge ModHILLN
Focal-Tversky Lose
OKUy
DkU
FILM
FIll
Layeriern
Layerferm
CUP Encoser
Coneelution
Convelation
Wharpers coouthi Dassasa ho COnNECT (fonenghaaa
Hand
Eas
Hard
Span
Erbeaking
Span
Viek
Makk
p-1)
Figure 3: Overview of PLUM. PLUM is not dependent on special tokens (e.g., <SEG>) added during finetuning to generate segmentation masks. PLUM uses bidirectional span extractor that automati- cally determines which tokens should be passed to the mask decoder to generate segmentations. A feedback loop based SAM's mask decoder enables PLUM to condition future segmentations those past.
Segmentation Evaluation We evaluate part segmentations via gloU (global IoU), which measures the average IoU over part annotations [19] 37.. micro-gloU averages part IoUs over all masks in the dataset, measuring how well the model segments the most common parts. macro-glot averages part loUs for each image, then averages these image loUs over the entire dataset. This metric is less affected by common parts, measuring how well a model segments parts in general.
PLUM: Part-Level Understanding LMM
Shortcomings of LMMs on Part Understanding We find that existing LMMs are unable to accu- rately identify parts in an image. Even segmenting LMMs 19 37 trained on part segmentation datasets such as PACO [35] and Pascal-Part (7] exhibit poor performance on part-level segmentation Fig. 4 and Table 2). We identify two key architectural deficiencies of segmenting LMMs: (I) They rely on special tokens for segmentation (e.g. [SEG] or <p></>>). These tokens are not seen during pretraining, so we hypothesize that their addition to the vocabulary and subsequent finetuming perturbs models' original token distributions Table 5}. (2) They discard prior mask predictions when segmenting in sequence, conditioning only on past text during generation. Incorporating prior mask predictions would likely help maintain consistency and better localize future predictions [Fg.. 5a).
Proposed Method Based on these observations we propose PLUM, segmenting LMM with part-level understanding. PI.M consists of vision-language model (initialized from LLaVA (8 which takes image and text inputs, along with a mask decoder (initialized from SAM's decoder that generates segmentation masks.
Let h E be the VLM's last-layer embedding of token (i= LI.., N) of the output sequence. We process these embeddings along two complementary pathways: (i) the Span Extractor, bidirectional self-attention block that tags beginning (B), inside (f), and outside (O) (36) positions of tokens to segment, and (ii) a projection head that maps B/I embeddings into "mask queries," regularized by KL divergence, An overview is given in Fig.
Token-level Query Selection (Span Extractor) The Span Extructor enables the selection of segmentation--elewe text spans to pass to the mask decoder without the use of dedicated seg- mentation token. Given the last-layer token embeddings {h. J apply two-layer token-wing wise MI.P before infusing global context by passing the embeddings through a bidirectional Transformer encoder block. This bidirectional attention is critical for reliable BIO span tagging, as otherwise the L.M's causal masking prevents embeddings from seeing future context. A final projection layer maps these contextualized embeddings to {B, I, O} logits. We train the span extraction module using cross entropy loss Capan where B, I tags correspond to part names to segment.

---

space and During inference, contiguous B / chains are greedily merged to form text spans that are projected into segmentation queries. Note, we also enable users of PLUM to override the automatic tags with manual span selection, enabling interactive, interpretable "highlightto-ssegmenn to-smmment" behavior.
Query Projection with KL Constraint Let S = {(i.. j=) be the set of contiguous B 3- I spans produced by the span extractor, where i, and Js are the start and end token indices of span and N = [S] is the total number of such spans in the sequence. For every span token & E je] we obtain "mask-query" vector qs = =((h)) Rm, with g(-) learned MLP projection. To keep the span representations close to the pre-trained backbone VLM's manifold, we pool the last-layer embeddings of each spart) and impose a Gaussian KL penalty against the corresponding frozen
1 -o || - 11 This term is applied only to B/I spans, 20
teacher embedding .4,, LxL. = N, Σm
preventing their hidden states from drifting away from the original thereby preserving the VLM's textual reasoning ability.
To incorporate previously predicted masks into the mask decoding process, we inject feature-wise linear modulation (FiLM) (34) layers into the SAM decoder's mask encoder (Fig. 3). These layers allow us to encode the mask while conditioning on prior text spans, providing semantics beyond raw binary mask. We use this modified mask encoder to encode each prior mask into a stack of text-enhanced feature maps. The stack of feature maps pooled into single feature map via patch-wise attention pooling (over the stack dimension), with a learned feature map providing an attentional query for each patch. This pooled feature map representing all prior predicted masks is fed into the mask decoder (along with the pooled text embeddings) to generate the next mask.
Mask Feedback Loop
Segmentation Mask Generation Tagged token embeddings qk are average pooled and passed to the mask decoder, generating a mask M, With ground-trun -ruth mask M, we adopt the Focal-Tversky loss 1. Lacg = N I #0 (Fr(M,, M,). Versly loss is a generalized version of the DICE loss 45 This gives the overall objective equal to L LLM + λ Capan + XaCKI. + AsCseg + A&CRCE, where LiM is the standard language- geneegg-gn loss and LECE is per-pixel binary cross-entropy as adopted from [9]. The BIO head precisely extracts segmentation spans, while the Focal-Tversky loss, biased toward recall 0.7) and precision (3 (033), encourages sharper, high-IoU masks at fine-grained image regions. For additional details on the hyperparameter setting, refer to AA.I
5 Experiments
Implementation Details We use pre-trained LMM, LLaVA-7B, and LLa LLaVA-Ilama2-133 (28]
as backbones for PLUM (Sec. 4 PLUM follows the consecutive two-stage finetuning process: (1)
PLUM is first finetuned with randomly sampled mixture of PACO-LVIS (35), Pascal Parts (7].
PartImageNet [T]. COCO-Stuff [5). ADE20k [56), the RefCOCO line of datasets [5]. a VQA
dataset from LLaVA (1lava_ Allavainnstruc instruct 150k), and a Reasoning Segmentation [19] dataset. This setting is similar to the previous line of coteeenttttionennalle LMMs [19 37741 and we refer to the stage-1 checkpoint of PLUM as the zero-shot (or pretrained) baseline throughout this paper. (2) To further finetune PLUM on our PARTONOMY training dataset, we take PARTONOMY PACCO, -PartlmageNett and PartcllParss construct training split and validation split. Note, do not use the PARTONOMY-CON split as training data and use it only as evaluation data. We refer to the Appendix for additional details on the hyperparameter settings and training details.
Baselines To evaluate PLUM's proposed changes, we use LISA 19), GLaMM [77, and PixeILM [1 as our primary baselines. All of these models use LLaVA as the base LMM and use SAM-style decoders to generate segmentation masks 28
For segmentation, we also evaluate X-Decoder, SEEM, and Grounded SAM 2 as general open- vocabulary segmentation models [59 60 40 38 As they do not understand question-based prompts, we provide them with the ground truth (gt) parts to segment individually. We also include SegLLM,
segmenting LMM based on LLa .5 and HIPIE (2 [5). HIPIE is a decoder built for multi-scale and part segmentation, and SegLLM is trained to perform multi-round segmentation on parts, allowing it to refer back to previously predicted masks. The similarity of this mechanism to our feedback loop motivates us to include SegLLM as a baseline, despite the imperfect comparison due to its use of newer LMM and different mask decoder.
We use mean pooling: any differentiable pooling operator is admissible.
7

---

idwndl' kation micro diantfication macro
Paokive micro
Negative maro Negative macro
Poukiive macro
PAFO LVIS
Partimagene
Pasce
Cone
80
N.giod
ink nesters
Lite ink
ins
postlete
Loe
Figure 4: Performance (micro/macro gloU) on PARTONOMY validation splits.
For evaluations, we include a random baseline (which randomly selects answers) situate the models' part precision and recall. GPT-4g frontier model, provides upper bound on performance. GPT-4o has an advantage allowing it to take advantage of shortcuts not available to the other models (like identifying the base answer from which the wrong answer choices generated).
must be provided with all four answer choices at once,
5.1 Explanatory Part Segmentation
Part Identification and Comparison Questions In Table PLUM outperforms LISA and GLaMM on all three part-segmentation question types in the zero-shot setting. We attribute this gain (i) span-level constraints that keep pre-trained textual semantics intact and (ii) our mask-feedback loop, which refines each mask using its visual history. By contrast, Table shows only marginal gaps in text-only metrics (P, R, Acc.). PARTONOMY's answer choices intentionally contain extensive lexical overlap, demonstrating the language models' difficulties in comparing similar answer choices.
Table 2: Explanatory Part Segmentation's segmentation performance (gloU) on PARTONOMY-
Core. "ft" = fine-tuned on Partonomy; "gt" = OV segmentation models given ground-truth answers. Part2Whole and Whole2Part scores are reported only for part prediction.
Identificaion mauno
Intersection microo
Difference mocro
Part2Whole milero
Whols2Part
Extra Seg Data
micooo
macro
micro
macro
micro
macro
Open. Vacabulayy Segmenttation
X-Decodrr tQU
39
11.5
13.4
13.5
14.0
11.1
12.4
SEEM
60
13.5
15.5
17.1
18.2
12.5
13.8
Grounded (g)
LLAVAUL. INPIE 27
S
29.6
28.5
32.3
29.3
LLOVA MAM-sryle Musk Decoder [28
LISA-13B 19
X
5.9
7.0
7.1
7.5
6.1
7.1
5.7
6.6
6.0
6.8
PixelL 33 41
6.8
8.4
4.4
4.8
4.2
4,8
4.6
5.4
6.3
7.8
GLaMM
6.2
5.2
5.6
4.9
PLUM-13B
X
14.5
27.4
23.7
29.9
14.9
24.8
14.3
26.8
15.4
27.5
LISA-13B (ft)
33.6
35.4
37.0
38.4
30.4
31.6
32.6
34.7
34.3
36.2
PixeILM-133 (f)
36,8
35.4
34.7
38.5
11.2
12.1
34.9
33.6
32.9
34.2
GLaMM (f)
36.6
38.8
40.3
42.1
33.6
34,8
36.
38.5
35.7
38.0
PLUM-138 (ft)
X
36.2
41.6
45.9
33.0
39.4
36.7
40.8
36.2
39.8
Part- Whole Reasoning Questions The Whole Reasoning results Table showow that knowing the object label prior to part segmentation leads to better mask prediction performance the pretrained models obtain higher Whole2Part than Part2Whole scores. This suggests that part mask prediction benefits from object label conditioning. The advantage obtained by object conditioning evaporates once the models have been trained on sufficient part data, however, as shown by the finetuned models. Similarly, in Table 3 the models' increase in object accuracy after conditioning on the object's parts, and their increase in part accuracy after conditioning the object label, underscores the utility of jointly predicting object labels and parts.
'Specifically, gpt-40- 2000-44-224408---0

---

Table 3: Explanatory Part Segmentation text performance PARTONOMY -ore. "ft" = fine- tuned on the Partonomy training sets. and R denote Precision and Recall; is multiple-choiee accuracy. OA and PA refer to object and part accuracy for Parl2Whole and Whole2Part questions.
Methods
Identilication
Intersection R
Difference R
Phol2Whole
Whole2Part
R
P
PA
CA
R
PM
(M
LISA-13B
21.9
57.8
87.2
65.4
Pis +1
90.4
73.6
35.0
66.1
57.0
46.1
83.3
71.1
35.8
87.6
73.6
33.2
60.4
94.5
84.3
57.9
41.2
GLaMM (37
83.8
89.5
-8.6
51.5
69.1
52.3
74.0
81.8
32.9
72.8
95.0
20:1
62.6
74.
93.2
24.1
50.7
PLUM
86.6
95.6
60.2
49.1
72.7
26.6
73.9
88.8
35.0
85.6
94.5
58.6
71.5
90.6
95.9
70.9
59.3
(ft)
30.7
98.2
96.5
73.1
68.8
97.3
28.0
PixeILM-13B (ft)
87.2
64.3
16.5
58.3
49.6
36.9
83.3
60.8
22:7
84.8
64.2
16.1
49.9
87.2
66.3
22.1
36.0
GLaMM (f)
75.9
87.2
30.1
48.1
68.
26.0
68.6
81.1
21.9
80.4
78.5
31.8
56.3
79.6
78.3
32.0
43.6
PLUM (It)
82.9
85.0
42.1
53.6
69.9
34,7
73.3
77.8
30.2
81.7
$4.0
40.9
60.0
84.8
88.1
46.3
51.2
Random
20.0
70.9
73.3
75.3
76.5
20.0
LM
90.6
86.4
51.9
61.7
70.2
47.0
77.1
79.8
73.3
92.3
81.1
48.4
69.7
94.2
89.3
65.3
62.8
GPT-40
95.6
96.5
81.7
78.9
84.9
70.7
92.1
92.4
72.3
97.7
97.9
89.0
96.5
97.7
97.9
89.2
96.3
5.2
Additional Downstream Tasks and Ablation Study
We further evaluate PLUM on non part-centric downstream tasks to evaluate its generalizability and whether our choice to omit special [SEG] tokens preserves PlUM's pretraining knowledge. For segmentation, we choose the Reasoning Segmentation task [19], which requires the model to reason about the referenced object before segmenting it. To evaluate PI.M's general visual reasoning,
select the Visual Question Answering (VQA) tasks TextVQA 43 and GQA [13, and a visual
hallucination task, POPE 23
Reasoning Segmentation PLUM demonstrates strong generalization to the Reasoning Segmen- tation task proposed in [19]. As shown in Table 4 PLUM outperforms existing open-vocabulary segmentation models such X-Decoder and OVSeg, and also generates more accurate segmentations than LISA, the model proposed for the task.
Table 4: Performance on the ReasonSeg benchmark. PLUM outperforms open-vocabulary seg- mentation models and LISA, the original model trained for reasoning segmentation.
Method
gloU
Method
gloU
OVSeg
28.5
LISA-7B 19
44.4
X-Decoder
59
22.6
LISA-7B (ft)
52.9
SEEM 60
25.5
LISA-13B
48.9
Grounded-SAM -SAM 400
26.0
LIB (ft)
56.2
LLaVA 1B5-7B + OVSeg
38.2
PLUM-7B (ft)
53.5
LLaVA1.5 1..5 13B OVSeg
37.9
PLUM-13B (ft)
57.3
Table 5: Accuracy (%) on VQA datasets and the POPE hallucination benchmark. Numbers in parentheses show percentage change relative to the LLaVA-I3B backbone. LISA and PixeILM lose most of their lasion-language reasoning abilities upon finetuning for segmentation, whereas PLUM's performance increases.
Method
TextVQA
GQA
POPE
LISA-13B 19
1.58
(-93.1%)
1.33 (-97.4%)
1.87 (-94.1%)
PixeILM- M-13B
41
8.37
(-6.44%)
21.72 (-57.6%)
15.29 (-51.9%)
LLaVA- 13B
28
22.84
51.27
31.80
PLUM-13B (ours)
30.11 (+31.%%)
39.18 (-23.6%))
34.65 (+-8.9%)
Distribution Shift Induced by Special Tokens To investigate whether Plum's removal of special tokens mitigates distribution shift from LLaVA's pre-trained representations, we compare PLUM'S performance VQA [3] (23) tasks to those of models which use [SEG] tokens and to that of the base LLaVA model Table 5] To our surprise, the performance of state-of-the-art segmenting LMMs deteriorates significantly on VQA and visual hallucination [23] tasks. Through inspection, we find that LISA tends only generates irrelevant [SEG] tokens. While PixeILM performs somewhat better, it still suffers due to its use of multiple specialized tokens AT]. In contrast, PLUM outperforms the other segmenting LMMs, even outperforming the LLaVA backbone on 2/3 tasks. This strong

---

performance demonstrates that our paradigm of segmentation through span extraction preserves the visual reasoning capacity of the LMM.
Figure 5: Ablations (a) The feedback loop and tagging mechanism improve part segmentation on PartImageNet; (b) Varying the KL-constraint weight AKL trades off segmentation gloU and TextVQA accuracy.
(a) Ablating key components of PLUMM. F stands for the feed- back loop. LISA has neither feedback loop nor span extractor.
Ace Partimegellet and Textvon Accurscy nugebbt Teevis (woc)
70
Model(s)
micro-gloU
macro-gloU
60
LISA-13B
65.6
67.7
LISA-13B (+F)
66.7
68.6
PLUM-13B (-F)
61.4
73.9
30-
PLUM-13B
67.9
80.3
(b) Effect of AxI. segmentation (PartIma- geNet) and visual reasoning (TextVQA).
Ablating Key PLUM Components Removing the iterative mask-feedback loop lowers micro-gloU by 9.6% and macro-gloU by 8%, though the span-based tag extractor alone still beats the LISA-13B baseline by 8.4% macro-gloU. When both components are active, PL.M tops LISA by 3.5% (micro) and 20% (macro), showing that tag extraction broadens long-tail coverage while feedback refines segmentation accuracy.
Effect of KL Divergence Increasing the KL-alignment weight Aæz from 0 to 1.0 steadily trades segmentation for reasoning: PartImageNet micro-gloU falls by nearly 20% whereas TextVQA accuracy sees a 75% performance improvement. We set AKL = 0.1 in this work.
6 Limitations and Broader Impacts
Limitations Our work advances fine- greeggaiied, part-level understanding but still faces several con- straints Although PARTONOMY-COM includes the largest number of part labels to date, with 862 categories across 534 objects, it omits some rare or domain -omainn-pecific parts and lacks the object diversity of [1]. Expanding its coverage would further enhance LMM comprehension. While PLUM mitigates major architectural limitations of prior segmenting LMMs via BIO tagging and a FiL.M-asse feedback loop, it still struggles with small or ambiguous parts and may not scale efficiently to high-resolution imagery.
Broader Impacts Part grounding is crucial for compositional visual reasoning and safety-critical domains such as robotic manipulation, assistive systems, and automated inspection. Such scenarios require accurate and interpretable part understanding, with mistakes in identifying and the grounding of parts leading catastrophic consequences. By introducing PARTONOMY and PI.M, aim to foster more reliable and interpretable LMMss The high computational cost of large segmenting LMMs also underscores the need for more efficient, sustainable architectures. We hope our benchmark and analyses inspire continued research toward robust, efficient, and responsible part-grounding methods.
7 Conclusion
We introduce Explanatory Part Segmentation with the PARTONOMY benchmark evaluate part-level visual reasoning and segmentation at scale. PARTONOMY spans 606 object labels and 2,507 part labels; its evaluation split, PARTONOMY-Core alone contributes 534 objects, 862 unique parts, 1,068 images and 4.968 pixel masks masks-tripling PACO's object diversity and quadrupling its part vocabulary. By analysing current segmentation-enabled L.Mss, we pinpoint two systemic flaws-(i) distribution-shiitt from pre-trained weights and (ii) discarded visual context- and address them with PLUM, a span-tagging, mask-feedback L.MM that interleaves textual and visual reasoning without extra tokens. PLUM outperforms fine-tuned LISA-130 on ReasonSeg and adds 31.8% relative performance improvement TextVQA compared the I.L.VA-13B backbone. Together, PARTONOMY and PLUM lay a quantitative and methodological foundation for future research on fine-gun -in-ggraied, compositional, and interpretable multimodal models.
10

---

8 Acknowledgements
This research is based upon work supported by U.S. DARPA ECOLE Program No. #HR001 12390060. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
implied, of DARPA, or the
This work used Delta at the National Center for Supercomputing Applications through allocation #250183 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program 44. which is supported by U.S. National Science Foundation grants #2138259, #2138286, #21 #2138307, #2137603, and #2138296.
The authors thank Steven Gomez and MIT Lincoln Laboratory for proposing an initial set of part- centric concepts as part of the DARPA ECOLE Program. We also thank the numerous annotators who contributed to the Partonomy dataset.
References
[1] Nabila Ahraham and Naimul Mefraz Khan. A novel focal tversky loss function with improved attention u-net for lesion segmentation. In 2019 IEEE 16th international symposium on biomedical imaging (ISBI 2019). pages 683-687. IEEE, 2019.
[2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. 2015 IEEE International Conference on Computer Vision (ICCV), pages 2425-2433. 2233,243322222 2015.
[3] Ansel Blume, Khanh Duy Nguyen. Zhenhailong Wang. Yangyi Chen, Michal Shlapentokh-Rothnen Xiaomeng Jin, Jeonghwan Kim, Zhen Zhu, Jiateng Liu, Kuan-Hao Huang, et al. Miracle: An online, explainable multimodal interactive concept learning system. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 11252-11254, 2024.
[4] Timothy J. Boemne,, Stephen Deems, Thomas R. Furlani, Shelley L. Knuth, and John Towns. ACCESS: Advancing Innovation: NSF's Advanced Cyberinfrastructuee Coordination Ecosystem: Services & Support. In Practice and Experience in Advanced Research Computing (PEARC '23), page 4, New York, NY, USA, July 2023, ACM.
[5] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1209-1218, 2018.
[6] Shi Chen and Qi Zhao. Divide and conquer: Answering questions with object factorization and composi- tional reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6736667444 June 2023.
[7] Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1971-1978, 2014.
[8] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Lii, Pascale Fung, and Steven Hoi. InstructBI.IP: Towards purppse rision-languag models with instruction tuning. In seveth- Conference on Neural Information Processing Systems, 2023.
[9] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http:/www.pascall sewwork.rg//halllnnees hoc.:eeeeeeeco220/////sssoneeee
[10] Hyeonjeong Ha. Xiaomeng Jin, Jeonghwan Kim, Jiateng Liu. Zhenhailong Wang, Khanh Duy Nguyen. Ansel Blume, Nanyun Peng, Kai-Wei Chang, and Hong Ji. Synthia: Novel concept design with affordance composition. arxiv preprint arxv:2200 17793, 2025.
[11] Ju He, Shuo Yang, Shaokang Yang, Adam Kortleeski, Xiaoding Yuan, Jie-Neng Chen, Shuai Liu, Cheng Yang, Qihang Yu, and Alan Yuille. Partimagenet: A large, high-quality dataset of parts. In European Conference on Computer Vision, pages 128-145. Springer, 2022.
[12] Liqi He, Zuchao Li, Xiantao Cai, and Ping Wang. multi-modal latent space learning for chain-or of thought reasoning in language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18180-158 18--18187, 2024.
11

---

[13] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6700-677 6709, 2019.
[14] Chen Jiang, Allie Luo, and Martin Jagersand. Robot manipulation in salient vision through referring image segmentation and geometric constraints. arXiv preprint arXiv:2409. 11588, 2024.
[15] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg, ReferltGame: Referring to objects in photographs of natural scenes. Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 787-798, Doha, Qatar, October 2014. Association for Computational Linguistics.
[16] Jeonghwan Kim and Heng Ji. Finer: Investigating and enhancing fine-grained visual concept recognition in large vision language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pages 6187-6207, Miami, Florida, USA, November 2024, Association for Computational Linguistics.
(17] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IÉEE/CVF International Conference Computer Vision, pages 4015-4026, 2023.
[18] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advunces in neural information processing systems, 11222199-222111 2022.
[19] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Pruceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579-9589, 2024.
[20] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recognize anything any gramularity. arXiv preprini arXiv:2307.04767 2023.
[21] Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language -mmage pre-training for unified pages 12888- 12900. PMLR, 2022.
understanding and generation. In International conference on machine learning,
[22] Xiao Li, Yining Liu, Na Dong. Sitian Qin, and Xiaolin Hu. Partimagenet++ dataset: Scaling part-based models for robust recognition. In European Conference on Computer Vision, pages 396-414. Springer, 2024.
[23] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wea. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292-305, 2023.
[24] Feng Liang. Bichen Wu, Xiaoliang Dai, Kunpeng Li. Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061-7070, 2023.
[25] Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmentation. In Proceedings of the (enfeececonferencc on computer vision and pattern recognition, pages 23592-23601, 2023.
[26] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations.
I Lijuan Wang. Mitigating hallucination
[27] Haotian Liu, Chumyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 26296-26306, 2024.
[28] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee, Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916 2023.
[29] Mingxuan Liu, Subhankar Roy, Wenjing Li, Zhun Zhong, Nicu Sehe, and Elisa Ricci. Democratizing fine-grained visual recognition with large language models. In The Twelfft International Conference on Learning Reprexentations, 2024.
[30] Weiyu Liu, Jiayuan Mao, Joy Hsu, Tucker Hermans. Animesh Garg, and Jiajun Wu. Composable part-based manipulation. arXiv preprint 240.20505666666. 2024.
12

---

[31] Pan Lu, Swaroop Mishra. Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan, Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances Neural Information Processing Systems, 35:2507-255 2521, 2022.
[32] Chancharik Mitra, Brandon Huang. Trevor Darrell, and Roei Herzig. Compositional chain-of-thougle prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14420-14431 2024.
[33] Wujian Peng. Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu. Synthesize diagnose and optimize: Towards fine-grained iisoo-languaae understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
[34] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings the AAAI volume 32, 2018.
conforreencc conference artificial intelligence,
[35] Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang. Aaron Marquez. Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7141-7151 2023.
[36] Lance A Ramshaw and Mitchell P Marcus. Text chunking using transformation-base learning. In Natural language processing using very large corpora, pages 157-176. Springer, 1999.
[37] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing. Ming-Ilsuan Hsuan Yang, and Fahad Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 55009-13016 2024.
[38] Nikhila Ravi, Valentin Gabeur, Than-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, al. Sam 2: Segment anything in images and videos. arXiv preprint (xXi::2408.00//1 2024.
[39] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ( (EMNLP-IJCNLP), pages 3982-3992, 2019.
[40] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint M4XI::2401.115555 2024.
[41] Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26374-26383, 2024.
[42] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146-162. Springer, 2022.
[43] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlci Chen, Dhruy Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
[44] Jascha Dick-Diisttin, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibriu thermodynamics. In International conference on machine learning, pages 2256-2265. pmlr, 2015.
[45] Carole H Sudre, Wengi Li, Tom Vercauteren, Sebastien Ourselin, and M Jorge Cardoso. Generalised dicc overlap as deep learning loss function for highly unbalanced segmentations. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: Third International Workshop, DLMIA 2017, and 7th International Workshop. ML-CDS 2017, Held in Conjunction with MICCAI 2017, Québec City, QC, Canada, September 14, Proceedings 3, pages 248. Springer, 2017.
[46] Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages l 11888- 11898, 2023.
[47] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tunel chat models. arXiv preprint arXiv:2307.09288, 2023.
13

---

[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
[49] Junchi Wang and Lei Ke. Llm-seg: Bridging image segmentation and large language model reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1765-1774 2024.
[50] Xudong Wang. Shufan Li. Konstantinos Kallidromitis, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. Hierarchical open- vocabulayy universal image segmentation. Advances in Neural Information Processing Systems, 36:21429 21453,,,,,,,- 2023.
[51] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2485 2022.
[52] Mengdi Xu, Peide Huang. Wenhao Yu, Shiqi Liu, Xilun Zhang, Yaru Niu, Tingnan Zhang, Fei Xia, Tan, and Ding Zhao. Creative robot tool use with large language models, 2023.
[53] Yuqian Yuan, Wentong Li, Jian Liu, Donggi Tang. Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28202-2821 June 2024.
[54] Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, and Thomas Serre. A benchmark for compositional visual reasoning. Advances in neural information processing systems, 35:29776-29788, 2022.
[55] Zhuosheng Zhang, Aston Zhang, Mu Li, George Karypis, Alex Smola, et al. Multimodal chain-of-thought reasoning in language models. Transactions on Machine Learning Research, 2023.
[56] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through adc20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[57] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler. Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):302- 321, 2019.
[58] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[59] Xueyan Zon, 7i-Yi Dou, Jianwei Yang. Zhe Gan, Linjic Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel. image, and language. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15116-15127, 2023.
[60] Xueyan Zou, Jianwei Yang. Hao Zhang. Feng Li, Linjie Li. Jianfeng Wang. Lijuan Wang. Jianfeng Gao. and Yong Jae Lec. Segment everything everywhere all at once. Advances in neural information processing systems, 36: 1971999979222 19782, 2023.
14

---

A Appendix
A.1 Model Setup
Hyperparameter
PLUM
LISA
PixelLM
GLaMM
Backbone
Language model
LLaMA-2-130 13B CLIP VIT-L/14
LLaMA-2-133 13B
LLaMA-2-13B CLIP VIT-L/14 Conv-U-Net
LLaMA-2-13B
Vision tower
CLIP VIT-L/14
CLIP VIT-L/14
Mask decoder
SAM VIT-H
SAM VIT-H
SAM ViT-H
1/O and training schedule
Input resolution (px")
1024"
1024
1024
10242
Max text length
512 bfl6
512
512
512
Precision
bfl6
bf16
bf16
Epochs
25 (0-shot) 14 (ft)
/1 (ft)
/14 (ft)
/14 (fi)
Batch size
6
6
6
Grad. accumulation
10
10
10
Effective batch
10x bsz GPU
10x bsz
GPU
10x baz GPU
bsz GPU
Optimizer
Optimizer
Adam 3x 100
AdamW
AdamW 3x10-*
Adam W
Learning rate
3x100=
3x 1x110%
Betas
(0.9, 0.95)
id.
id.
Weight decay
0 1.0
0
0
0 1.0
Gradient clip
1.0
1.0
Loss weights
Ace: (LMM)
1.0
1.0 0.5 (Dice) 2.0
1.0 0.5 (Dice) 2.0
1.0 0.5 (Dice) 2.0
Aueg (Dice/FTL)
8.0 (FTL')
ABEE (mask)
2.0
AKL
0.1
Aata (BIO)
2.0
Dice scale factor
10a
10a
10a
10a
FTL (c.
(0.7. 0.3)
Additional Modules
BIO span tagger
v 2048
Bidircctional encoder
Feedhack Loop (Temporal Mask Pooler))
v
Trainable SAM parts
decoder + prompt-enc. 16, p 0.05
decoder
decoder
LoRA LM (q.x)
id.
id.
Table 6: Hyperparameters used for all experiments. We juxtapose four segmenting LMMs, including PLUM, against each other to illustrate the hyperparameter differences among the models. "id." indicates "identical" to the other model's setting. indicates "not applicable", and "bsz" indicates the batch size the models are trained on. tPLUM defaults to Feeal-Tversky loss; when we ablate we fall back to standard Dice. *Cross-attention is enabled only in ablation runs when explicitly specified.
Model Training and Evaluation PLUM is optimized in two stages: Stage-O ("O-shot") mixes four
mublily--vaillabe multi-tast corpora- semantic segmentation [35 [1 referring segmentation
15], visual-question answering and image captioning (88 (9:5:5:1 sampling ratio)-frr 25 epochs. Stage-/ then optionally finetunes on PARTONOMY-PAP PACOO PARTONOMY-ParTIng PartImageNet, PARTON- OMY-PascalPart training splits for an additional 4 epochs, and call this PLUM checkpoint PLUM
(ft) as shown in Tables and We resize every image to 1024 pixels and truncate text to 512
tokens.
Training uses DeepSpeed ZeRO-2 with bf16 precision, gradient_ane awccuullatioontttteeeeoeeeeeeeee
per-GPU batch of 6, and 10 (effective batch 10 x bsz x NoPU). Weights are updated by AdamW W (0=(0.9, 0.95), no weight-decay) with a peak learning-rate of 3 x 10 linearly warmed up for the first 100 optimization steps and clipped global norm of 1.0 thereafter.
We freeze the CLIP vision tower and the MM projection layer; all other components are trainable. The LLaMA decoder receives LoRA adapters on q-proj, v_proj (r=8, a=16, dropout 0.05). On the vision side we finetune the SAM mask-decoder and, when specified, the prompt-encoder. PLUM's additional heads, the bidircctional BIO encoder and the Temporal Mask Pooler for mask feedback loop, are always optimized.
15

---

Concept-Part Ontelogy
Input Imege
Airplane
Agriculture
Knife
Refrigerator
Landing
Spraying
Drip Tray
Question-Answer Mutator Framework
(partX))
Mutation Openations
Deletion
pint 7A... pi
Segmenting LMM
Dent %
joat joinr
Subatitution
piert 7h. fooe A0 (owt, 1) Boer
Explanatory Segmentation Tasks
rxpert bor hiinaing parte oxemen (grann--trdi-sdd
parn doos tri jprcundieuthteass correce with joantrastive labal Spert//g
Segmentation
Negative Segmentation
What dows tra (pround-out conmron with fcontrastive Spert//)
Difference
A: has
(gronnd-doutb-ubbeg (part
jpastsstiv
Whole2Part
02: Onay the unue pats maie but toot Ne (predctee-sasg?
a fpredicted-tutt heag
(wwC. (PaL
repert hiphight mod atowahow
Part2Whole
AF: Kia (owrt 1) (earL.)
you choonesiiggiittttt
A2: Vi (predkene--muth
Figure 6: Illustration of the hierarchical object-part ontology construction in PARTONOMY-Core We manually collect 562 object-level concepts and generate part-level concepts using LLM and manually filter out overlapping parts. Our question-answer mutator then generates challenging answer choices based the part set overlap between object-level concepts.
The total loss is weighted sum of (i) language modeling cross-entropy, (ii) BIO span classification loss, (iii) Focal-Tvers Tversky and pixel-wise BCE for masks. (iv) KL divergence to a frozen teacher snapshot of LLaMA. Loss weights follow Table 6 random seed is fixed to 42.
For model evaluation, we first divide the performance evaluation to two facets: (i) pixel-level mask prediction evaluation as in Table and (ii) multiple choice answer selection evaluation. Note, for multiple choice answer selection, we take the argmin over the entropy of each answer choice (i.e., the argmax over the sequence-lewe choice with the lowest entropy. For pixel-level mask prediction, we provide the ground-truth answer choice and their part text (or [SEG] for LISA, PixelLM and GLaMM) so that the models can be evaluated solely on their mask prediction performance per part text.
probability of each answer choice), and greedily select the answer
B PARTONOMY Construction
Ontology construction For PARTONOMY-COrf we first construct the object-part hierarchy as illustrated in Figure The hierarchy starts with set of root-level, superordinate object categories (e.g., airplane, kitchen tools), where each category contains set intermediate object- level concepts (c.g., agricultural airplane. coffee maker) that fall under each superordinate category, and part-level concepts which compose the leaf nodes of each object-level concept. We manually define 10 distinct superordinate concept categories, ranging from generic concepts, e.g., vehicle, office supplies, to more complex concepts such as geography") There are total of 534 object-level concepts (c.g., banana boat under the boats category) These concepts contain 1976 concept-specific parts (e.g. biplane-wii -wing, 862 unique part types, each appearing in 1.6 object categories on average. Table 7 shows the full list of object categories with example object labels.
$ 'Dice loss when focal, tvereky=False. "We thank MIT Lincoln Labl and DARPA for developing an initial category and concept ontology as part of the DARPA ECOLE effort, which refine and expand to construct part-centric ontology.
16

---

Object Category
Object Label Examples
airplanes
agriculuuall, fighter jet, ultralight amphibious, barge, submarine firefighting, combat, recreational hose, hand rake, hedge shears airport, hot spring, roadmap angle, intersecting lines, ray attack, medevac, law enforcement air fryer, dish brush, soup spoon shredder, staple remover, legal envelope aircraft carrier, corvette, oil tanker adjustable wrench, level, vise bulldozer, racing car, tricycle anti- mnti-ballistic missile, handgun, sword
boats
drones
garden
geography
geometry
helicopter
kitchen
office supplies
ships
tools
vehicles
weapons
Table List of object categories with example object labels from each.
Table 8: Full PARTONOMY statisties after normalizing object and part names across subsets. For PartImageNet we define an "object" as its category (containing multiple classes), as the parts the same across each category. There more object and part categories in the validation set as PARTONOMY- evaluation only.
Datasets
Object Labels
# Part Labels
# OOjject-Part Labels
# Images
Seg. Masks
PARTONOMY (train)
89
216
563
58,706
321,751
PARTONOMY (val)
606
975
2,493
15,794
85,350
PARTONOMY
606
975
2507
74,500
407,101
Image Annotation Our image annotation consists of two stages: (i) Mask annotation with the first group of annotators; (ii) Mask re-anmotation and revision with the second group of annotators (the authors of this paper) ensure the quality and correctness of the mask annotation and the parts associate with each coarse-level concept category. For the first stage of our mask annotation, we first crawl the images using the object-level and part-level concept names in our ontology hierarchy as queries and manually annotate masks for each image with human annotators. We assign each annotator with at least 10 coarse-level concept categories and provide instructions to use an external image annotation tool"] for the segmentation mask annotation. For the parts with fewer than annotations across the dataset, we remove the parts associated with concept; we set hard threshold at m 5.
Building off' of Existing Part Segmentation Datasets Our pipeline allows us to generate questions- the Pascal-Part (7]. PartImageNet [[]. and PACO (35) datasets to expand the diversity our task data. We take the training splits from each, divide them into training and evaluation sets by images in an 80/20 ratio, then generate at most one question of each type for each image. Depending on the question type, number of objects the image, and the objects' parts, generating multiple questions for a single image is messibee-howeee,, type for each image to one.
answer pairs from any set of existing part annotations. Therefore,
maintain dataset balance we cap the number of questions per question
The PACO dataset frequently has multiple instances the same object class in an image. To disambiguate the referenced object when asked to ground parts of "the object", we annotate the images with bounding boxes for those with multiple object instances. We select the object instance which is largest and which has the most annotated parts for use in our dataset.
Dataset statistics for our merged PARTONOMY dataset, containing all subsets (Pascal Part, PartIma- geNet, PACO, and PARTONOMY-Core) can be found in Table 8
'httss//www.cvaa.a//
17

---

C
Additional Experiments
To examine how Explanatory Part Segmentation models behave beyond the Partonomy Core split, we evaluate the same set of systems on three public, large-scale nart-segmentatinn benchmarks that vary in object vocabulary size and annotation granularity: PACO LVIS 33], ParllmageNet [IT], and PascalParts 7 The quantitative results summarized in Tables 11 findings.
Below we highlight the key
C.1 Zero-shot generalization
In Tables 9 10 and I1 we show that PLUM-13B generalizes best without extra segmentation data. Despite using no part masks during pre-training, PLUM-13B outperforms every other zero-shot model on all three datasets (e.g.. Identification macro-gioU 16.0 on PACO LVIS Difference macro-glov carro-gloU== = 14.3 vs. & 0.8). The gains are most pronounced on PACO LVIS an open- vocabulary benchmark with 406 object Serggorres-uuggesteee
1.1 for baselines;
that PLUM's part mask-language aligmment carries over to out-of-distribution objects with minimal degradation.
helps but not sufficient, PixeILM-13B and GLaMM leverage large-scale mask supervision during pre-training (vin the Extra Seg Data column) and indeed achieve higher zero-shot scores than LISA-13B PartImageNNe and PascalParts. However, they still fall far short of PLUM in every metric, indicating that PLUM's special token-agnostic approach is more sample-efficient than the additional incorporationg of specialized tokens during segmenting LMM training.
Additional segmentation-superviis
C.2 Effect of fine-tuning Partonomy
Fine-tuning on only the Partonomy training split yields double-digit improvements across our baselines, with three consistent patterns emerging. PL.UM (ft) attains the strongest macro-gloU scores. It tops all three datasets in Identification and Intersection (c.g.. 80.3 and 77.4 7.4 on PartImageNet), and remains within < 2.0 micro-gIoU of the best performer, demonstrating excellent part recall on rare classes. GLaMM (ft) is highly competitive on micro metrics. GLaMM (ft) slightly edges out PLUM on Difference-micre for PartlmageNet and PascalParts. Nevertheless, the gap macro-glol -aauo (z 4.0 gloU) shows that GLaMM still under-segments uncommon parts. PixeILM-133 (ft) saturates early without Intersection gains.
C.3 Dataset difficulty and domain shift
PACO LVIS is the hardest split. Even after fine-tuning macro-gloU scores on PACO LVIS are 20.0 30.0 points lower than on PartImageNNt,, reflecting its long-tailed distribution and heavy occlusions. PLUM's lead here (e.g., 9.0 macro-gloU over GLaMM in Difference) underscores its robustness to veecbbocyyylary shift. PartImageNet rewards holistic part coverage. High Identification and Intersection numbers (e.g., 70.0 73.0 macro-gloU for GLAMM/PLMM) reveal that most methods can capture coarse part extents when objects are canonical and well-cropped. Nonetheless, PLUM's advantage ( 9.0 macro-gloU) suggests better treatment of fine-graing tails (e.g.. bird beaks, insect legs). PascalParts exhibits the largest fine-tuning boost. All models gain > 38.0 macro- gloU in Identification after fine-tuning- potential consequence of its limited category set and high annotation quality. Here, PLUM (ft) again leads the macro-gloU metrics, validating that its robustness.
Table 9: Explanatory Part Segmentation's segmentation performance (gloU) on PARTONOMY- ANCO_VVIS "ft" fine-tuned on Partonomy; Part2Whole and Phrle2Part are not reported for this dataset.
Methods
Extra Seg Data
Identification micro
Intersection micro
Difference
Phole
Whole2Part micm
micro
LISA-13B
0.8
1.1
1.0
1.2
0.7
0.8
0.8
1,0
0.8
1.1
PixelLM-138
1.1
1.3
1.3
1.5
1.2
1.5
1.1
1.2
GLaMM 37
0.6
0.8
0.6
0.8
0.5
0.6
0.5
0.6
0.5
0.7
PLUM-13B
16.0
8.2
17.3
5.2
14.3
LISA-13B
29.0
30.7
30.6
26.8
28.0
28.4
PixcILM-133 (ft)
37.8
40.5
34.8
42.7
36.0
36.5
6.8
8.5
6.8
8.4
GLaMM (fU 37
36.8
39.3
38.9
41.2
34.7
36.7
17.4
16.0
31.0
33.3
PLUM-13B (ft)
34.5
49.4
35.6
50.8
32.2
46.1
18

---

Table 10: Explanatory Part Segmentation's segmentation performance (gloU) on PARTONOMY- PartImageNet. "ft" fine-tuned Partonomy; Part2Whole and Parte2Part are not yet reported for this dataset.
Methods
Identification macro
Intersection micro
Difference macro
Part2Whole macto
Whole2Part macio
Extra Seg Data
micro
macvo
micro
micro
micro
8.7
5.7
7.1
PixcILM-13B
8.0
9.4
10.3
12.8
1.5
1,4
1.7
1,8
8.4
10.1
GLaMM
5.3
4.6
4.4
PLUM-13B
25.6
11.6
26.9
12.2
21.7
14,0
LISA-13B (ft)
65.6
67.7
68.4
70.4
60.8
61.5
62.9
65.0
65.3
67.3
PixelI.M-1381 (I))
35.7
61.2
GLaMM (ft) 37
68.9
70.8
71.2
73.1
64.7
65.4
61.1
64.0
62.9
65.9
PLUM-13B (ft)
67.9
80.3
70.2
77.4
63.3
60.7
Table 11: Explanatory Part Segmentation's segmentation performance (gloU) on PARTONOMY- PascalParts. "ft" = fine-tuned on Partonomy: Part2Whole and Whole2Part are not yet reported for this dataset.
Methods
Identification micro
Intersection micro
Difference
Part2Whole
Whole2Part marm
Extra Seg Data
macro
macm
miieno
macro
micro
macro
mucm
LISA-13B
2.2
2.4
2.4
2.6
1.7
1.8
2.1
2.3
2.1
2.4
PixcILM-13B
3.7
3.7
4.1
3.0
3.2
GLaMM 37
V
2,0
2.1
2.4
2.4
1.2
1.4
1.7
1.8
1.6
1.8
PIUMM-13
LISA-13B (ft)
42.9
44.6
44.2
46.4
38.2
39.8
37.4
38.7
42.5
44.3
PixcILM-13B (ft)
39.9
$4.1
38.8
55.0
38.8
55.0
37.6
51.5
40.9
42.5
GLaMM (fU (3
48.8
50.6
49.8
52.0
44.6
45.8
39.9
42.2
41.7
PLUM-13B (f))
48.4
64.0
49.3
65.5
44.4
54.0
C.4 Additional Ablation Studies
In addition to the main experiments, we conducted several ablations and follow-up analyses. These studies validate PLUM's architectural choices, loss formulation, robustness, and dataset design.
Bidirectional attention for BIO tagging. We confirmed that bidirectional attention is critical for accurate span extraction. Removing it severely degraded "T" tag accuracy both Parton- Pat-PartlmageNNt and RefCOCO.
Table 12: Bidirectional atfention ablation. Removing bidirectionality collapses span tagging performance.
Model
B-Acc
I-Acc
O-Acc
PLUM (bi)
98.59
87.32
99.98
PLUM (no bi)
100.00
15.86
99.78
PLUM (bi, RefCOCO)
99.98
99.87
100.00
PLUM (no bi, RefCOCO)
6.68
4.92
99.98
Loss function comparison. To isolate the impact of the Focal-Tversky loss (FTL), we retrained PLUM with Dice loss under identical conditions.
Table 13: DICE vs. Focal-Tversky. PLUM's gains stem from its architecture choices; FTL provides marginal improvement over DICE.
Loss
micro-gIoU
macro-gToU
B-Ace
I-Ace
O-Acc
DICE
66.47
79.86 80.30
92.99
99.99 87.32
86.48
Focal- -vers-yy
67.90
98.59
99.98
We ablated PLUM's span extractor (SE) and feedback loop (F). Both contribute independently and jointly yicld the best results.
Component ablations.
19

---

Feedback loop robustness. While conditioning on past inputs may theoretically introduce enror accumulation, manual inspection showed that even when early masks were incorrect, PLUM's feedback loop still produced correct later masks by relying on text embeddings. The loop also reduced duplicate or overlapping predictions. For example, the model without feedback repeated the same region for a lizard's foot and tail, while PLUU with feedback predicted them distinctly.
Computational cost of feedback loop. Timing experiments show that mask prediction is dominated by the LLM''s forward pass: averaged over 1000 examples, the LLM forward pass took 2.47 seconds, decoding without a feedback loop took .0097 seconds, and decoding with a feedback loop took .0162 seconds, a .2% increase compared to the language model's ' time for its forward pass.
Loss weight sensitivity. We varied the loss weights verify stability with respect ÅKT. and Aseg. Performance changed modestly, confirming robustness.
Table 14: Loss weight sensitivity, PLUM is stable; slightly higher AKL improves performance.
AKT.
Aseg
ABEE
LR
gloU
cloU
2.0
2.0
0.1
3e-4
0.385
0.331
4.0
2.0
0.1
3e-4
0,489
0.506
5.0
2.0
0.1
3e-4
0.517
0.494
6.0
2.0
0.1
3e-4
0.527
0.501
8.0
0.1
3e-4
0.573 0.546
Overall, these ablations confirm that PLUM's architectural innovations iosooaatioon-biiiretttoo span extraction, mask feedback conditioning, and Focal-Tversky optimization-joinnl account for its strong fine- grained segmentation and reasoning capabilities with minimal computational overhead.
C.5
PARTONOMY-Core Samples
Fig. provides an example of each question type for the Explanatory Part Segmentation task from PARTONOMY-CORE
20

---

Cooking Surface
Spout
Handle
Identification
Q1 What visible parts does the cast iron skillet the image have?
A1 The cast iron skillet the image has the following visible parts: cooking surface, handle, and spout.
iron skillet image has following visible parts: cooking
A3 The cast iron skillet in the image has the following visible parts: cooking surface, and handle.
following A5 The cast iron skillet in the image has the following visible parts: cooking surface, handle, isolated hill, and spout.
iron skillet image
parts: cooking
spout.
(a) An Identification question.
Blade
Handle
Intersection
Q1 What visible parts does the dinner knife in the image have common with utility knife?
A1 The dinner knife in the image has the following visible parts: blade, and handle.
knife image
following visible parts: blade.
A3 The dinner knife in the image has the following visible parts: cockpit.
knife image
following visible blade, locking
A5 The dinner knife in the image has the following visible parts: blade, panels, and tip.
(b) An Intersection question.
21

---

Handle
Foot Pedal
Tines
Difference
Q1 What visible parts does acrating fook the image lave which a hole puncher does not?
A1 The aeruting fork in the image has the following visible parts which hole puncher does not: foot pedal. tines.
The acrating fork the image has the following visible which hole puncher does foot podal. handle, and neck.
A3 The acrnting fork the image has the following visible parts which hole puncher does not: punch head, and tines.
The aerating fork the image has the following visible which hole puncher does foot podal, and punch head.
A5 The acrnting fork the image has the following visible parts which hole puncher does not tincs.
(c) An Difference question.
Bowl Body
Handles
Holes
Whole-to-Part
QI What is the object the image?
A1 is a colander.
wooden spoon.
A3 cheese
cake stand.
AS
peeler.
Q2 What visible does colander in image have?
The colander image has following visible
bowl body, handles, and boles.
A2 The colander in the image has the following visible posts: bowl body, hundles, holes, and plate.
A3 The colander in image
following visible
bowl body, point, handles, and saddle.
A4 The colander in the image his the following visible posts: handles, and holes.
The colander in image following visible parts: bowl body, handles, holes, island, precipice.
A Whole-to to Part question.
22

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and socictal impact. Do not remove the ciecklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA]-
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificattion right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked While "[Yes] " is generally preferable to "[No] ", it perfectly acceptable to answer "[No] providedd a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes]] to a question, in the justification please point to the section(s) where related material for the question can be found.
the checklist one of the factors in their evaluation.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
DO NOT REMOVE THIS
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:The abstract and introduction clearly outline our contributions, establishing PARTONOMY benchmark, diagnosing key limitations in existing segmentation-enan LMMs, and proposing the PLUM architecture for improving part-level understanding in LMMs.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
23

---

Answer: [Yes]
Justification: We discuss the limitations of the work in Section 6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only lested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
Justification: This work mainly focuses on empirical methods, evaluations, and analyses, and does not present any theoretical results, assumptions, or formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe the implementation details including training hyperparameters in Section 5 and the Appendix.
Guidelines:
24

---

reproducible, they The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis-
sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: For data, we elaborate the details on dataset in Section 3 while we will release the data once our paper officially get accepted as it is our one of the novel contribution. We will include the source code in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ Tinips.ccpubbll cuublic/guides... httpggudddeeeoeeeee
iobbiissionn iisinnicccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
25

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the statistics of the proposed data in Table 1, and hyperparameters for training in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do perform statistical significance tests or include error bars due to computational constraints. Instead, we train and provide comparisons against several baselines on various datasets to support the robustness of our claims.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma eтror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the computational resources in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
26

---

The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://neuriip...e httppppcccccpbccc/cc/ccccuiiuaae
Answer: [Yes]
Justification: Our study uses only publicly available models and datasets, involves no human or animal subjects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We describe broader impacts including positive societal impacts in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeent of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification:
We do not handle the safety models, so it is not applicable to our paper.
Guidelines:
The answer NA means that the paper poses such risks.
27

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The authors cite and acknowledge all utilized tools, models, datasets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
We elaborate the details on the proposed dataset in Section and Appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
We do not employ crowdsourcing baasee based experiments research with human subjects.
Justification:
Guidelines:
28

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not utilize human participants for experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29

---

Propulsion Component
Refueling Boom
Wings
Q1 What visible parts does the object the image have?
A1
The object the image has the following visible parts: propulsion component, refueling boom, and wings. The object in the image has following visible purts: refueling boom.
A3 The object in the image hos following visible parts: long propulsion component, and retueling boom.
The object in image has following visible purts: brake lever, and refoeling boum.
The object image has following visible degree scale, and wings.
Q2 What the
the image?
It aerial refueling tunker.
It is reconnaisaacc uiruraft.
is business
A4 It is murrowbbddy singlo-aisle nireralt.
regional
(e) A Part-to- Whole question.
Figure 7: Examples of the different question types from the Partonomy-Core dataset.
30


Reviews:
[{"review_id": "GcwOXArwIu", "reviewer": "Reviewer_gK9E", "summary": "This paper introduces Partonomy, a new pixel-level part grounding benchmark designed to evaluate the ability of models to reason about part-whole relationships. The dataset covers a broad range of object categories (787 in total), including common items, biological entities, and specialized objects, and is annotated with 1,157 fine-grained part labels. A key feature of this benchmark is its requirement for models to not only recognize part-whole relationships via textual description, but also justify them through precise visual segmentations.\n\nBuilding upon this dataset, the authors propose PLUM, a segmentation-enabled large multimodal model (segmenting LMM) that employs span tagging to avoid distribution shift caused by [SEG] token. Moreover, it refines the segmentation prediction via a feedback loop. \n\nExperimental results demonstrate that existing state-of-the-art LMMs exhibit limited capabilities in fine-grained part grounding. In contrast, the proposed PLUM model shows compelling performance on the Partonomy benchmark as well as other evaluation tasks involving reasoning, segmentation, VQA, and visual hallucination.", "questions": "Please review the weaknesses outlined above. In particular, I would like to highlight the following four key questions:\n- Could you provide theoretical justification or empirical evidence to support the claim that “compared to causal attention, bidirectional attention is more critical for reliable BIO span tagging”?\n- Could you conduct a fair comparison using Dice loss within PLUM to isolate the impact of the proposed Focal-Tversky Loss?\n- Could you provide a comprehensive ablation study of the three core components in PLUM: Span Extractor, KL Constraint, and Mask Feedback Loop?\n- Could you include an evaluation of the Span Extractor’s classification performance for the {B, I, O} labels?", "limitations": "No, the authors did not discuss the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\nThe proposed explanatory part segmentation task and the corresponding Partonomy dataset are valuable contributions. Also, the taxonomy introduced for the explanatory part segmentation task is comprehensive and well-structured.\n\n**Weaknesses:**\n\nMy primary concerns span the areas of data, method, experiments, and clarity. Specific concerns are as follows:\n\n**Data:**\n\n- **W1. Lack of Demo Samples from Partonomy / Partonomy-Core.** Given that the explanatory part segmentation task and the Partonomy dataset are the core contributions of the paper, it is essential to provide representative samples in the supplementary materials to help readers better understand their details and annotation quality.\n\n**Method:**\n\n- **W2. Unverified Claim Regarding Bidirectional Attention (L220–221).** The paper claims that “bidirectional attention is critical for reliable BIO span tagging” but provides no theoretical justification or empirical evidence to support this claim.\n- **W3. Unclear Roles of the Components in Mask Feedback Loop.** The proposed mask feedback loop involves additional components to the SAM decoder, including Feature-wise Linear Modulation (FiLM) and Patch-wise Attention Pool. However, it is still unclear why these two specific techniques are involved and how they influence the overall performance. Can FiLM be replaced by alternatives, such as a standard cross-attention mechanism?\n- **W4. Use of Focal-Tversky Loss Lacks Justification.** As all baseline methods are evaluated using Dice loss while PLUM uses a different loss, it is unclear whether observed performance gains stem from architectural innovations or the loss function itself. A comparative study using Dice loss within PLUM is necessary to isolate the contribution of each component.\n\n**Ablation Study:** \n\n- **W5. Incomplete Ablation of Core Components.** The effectiveness of PLUM is attributed to three key components: Span Extractor, KL Constraint, and Mask Feedback Loop. However, a comprehensive ablation across all 8 combinations of these components is missing. The ablation study should be conducted not only on Partonomy-Core but also across the other evaluation tasks (e.g., reasoning segmentation, VQA, and hallucination detection).\n- **W6. Lack of Analysis on Loss Weights.** The paper explores the KL constraint weight ($\\lambda_2$) but does not examine the influence of other loss terms, particularly span extraction loss ($L_{span}$) and segmentation loss ($L_{segm}$). A more thorough loss weight sensitivity analysis is needed.\n\n**Other Experiments:**\n\n- **W7. No Evaluation of Span Extractor’s {B, I, O} Classification.** The quality of the segmentation largely depends on accurate BIO tagging from the Span Extractor. However, no explicit evaluation (e.g., accuracy, F1 score) is provided for this three-way classification task.\n- **W8. Misleading Conclusion in L268–273.** The authors claim that “Table 2 shows that predicting the object label first prior to generating the part segmentation masks leads to better mask prediction performance”, but this trend does not consistently hold across methods like PLUM (ft), GLaMM (ft), and PixelLM-13B (ft), which contradict the stated conclusion.\n- **W9. No Discussion of Results in Figure 4.** The experimental setup and the results shown in Figure 4 are not discussed, leaving the reader unclear about its purpose.\n- **W10. Lack of Insightful Analysis in Table 3.** The conclusion that \"the proposed experimental setting dampens model-to-model variance\" (L265–267) is vague and uninformative. Further analysis is required to understand whether this variance is caused by experimental setup, model architecture, or other confounding factors.\n\n**Clarity:**\n\n- **W11. Unsubstantiated Claim About Training Data Size (L22).** The paper claims that other segmentation-capable LMMs are trained on significantly more data, but no concrete evidence or statistics are provided to support this statement. Details should be included.\n- **W12. Missing Metric Descriptions in Table 3.** Table 3 presents several quantitative results without describing the specific calculation evaluation metrics.\n\n**Minor:**\n\n- **L227–228:** The functions $g(\\cdot)$ and $N^+$ are introduced without any definitions or explanations.\n- All figures should be rendered as vector graphics to enhance clarity and readability, particularly for publication purposes.", "was_revised": true, "final_justification": "The authors have addressed my primary concerns. Although I remain unconvinced by their explanation of **W4** (motivation and fairness of using the Focal-Tversky loss), this is not a major issue. Overall, I am inclined to accept the paper and have increased my rating accordingly.", "rebuttal": "We sincerely appreciate the time and effort you have dedicated to reviewing our work. In the following, we have provided supplemental details and additional experiment results you requested to address your concerns.\n\n> **W1: Lack of Demo Samples from Partonomy / Partonomy-Core**\n\nWe provide Partonomy-Core samples in Figures 1 and 6 and will add more with model predictions.\n\n---\n\n> **W2: Unverified Claim Regarding Bidirectional Attention**\n\nPrior work shows that bidirectional models (e.g., BERT, BiLSTM-CRF [1-4]) outperform causal ones for token-level tasks, as causal masking is suboptimal [5,6]. Thus, we apply a bidirectional layer for BIO tagging. For example:\n\n1. Watering hoses are on the ground.\n2. Watering the grass is important.\n\nWithout the second word, it's unclear if \"watering\" is a compound noun.\n\n[1] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding\n[2] Bidirectional LSTM-CRF Models for Sequence Tagging\n[3] Deep contextualized word representations\n[4] LUKE: Deep Contextualized Entity Representations\n[5] Acquiring Bidirectionality via Large and Small Language Models\n[6] Looking Right is Sometimes Right: Investigating Decoder-only LLMs for Sequence Labeling\n\n(ii) **Empirical Evidence**\nWe train PLUM with/without the bidirectional layer on Partonomy-PartImageNet:\n\n||b-acc|i-acc|o-acc|\n|-|-|-|-|\n|PLUM (bi)| 98.59 | 87.32 | 99.98 |\n|PLUM (no bi)| 100.00 | 15.86 | 99.78 |\n\nThe bidirectional layer is vital for accurate `I` predictions. The model likely uses answer templates to get good `B` accuracy.\n\nOn Referring Expression Segmentation (RefCOCO | Unc | Val):\n\n||b-acc|i-acc|o-acc|\n|-|-|-|-|\n| PLUM (bi) | 99.98 | 99.87 | 100.00 |\n| PLUM (no bi) | 6.68  | 4.92  | 99.98  |\n\n---\n\n> **W3: Unclear Roles of Mask Feedback Loop Components**\n\nThe feedback loop lets prior mask predictions inform current predictions. SAM's mask decoder can take a prior mask (feature map) as a prompt; a natural approach is then to modify SAM's mask encoder to integrate multiple masks.\n\nAn initial design might use SAM's mask encoder to embed previously predicted masks (n_masks, h, w, d), fusing them into a single feature map representing previously predicted masks (patch-wise attention to pool into learnable CLS tokens).\n\nHowever, naively fusing prior masks neglects their semantics. As intuition, one can imagine that if parts are typically disjoint, then prior mask areas should not be predicted in the future. On the other hand, a prior segmentation of an airplane's body can be used to constrain a prediction for windows. \n\nFiLM layers use the textual span embeddings corresponding to masks to modulate their feature maps. [7]'s original usage supports our design. Cross attention is limited by there being one text embedding per mask (it would receive all of the attention weight), but an alternative could be to perform an attention pool by stacking the text embedding with all mask features. We leave this design to future work. \n\n\n[7]: FiLM: Visual Reasoning with a General Conditioning Layer\n\n---\n\n> **W4: Focal-Tversky Loss Lacks Justification (Impact of DICE vs. Focal-Tversky)** \n\nFocal-Tversky loss (FTL) generalizes DICE: they are equivalent when FTL's coefficients are 0.5. The marginal improvements of FTL over DICE shown below (\\alpha=0.7 and \\beta=0.3 in Section A.1) are why we select it. PLUM's large performance gains, especially zero-shot, do not stem from FTL. (Evaluation on Identification task of Partonomy-PartImageNet):\n\n| | micro-giou | macro-giou | b-acc | i-acc | o-acc |\n|-|-|-|-|-|-|\n| PLUM (DICE) (ft)| 66.47 | 79.86 | 92.99 | 86.48 | 99.99 |\n| PLUM (FTL) (ft) | 67.90 | 80.30 | 98.59 | 87.32 | 99.98 |\n\n---\n\n> **W5: Incomplete Ablation of Core Components**\n\nFigure 5 ablates starting from LISA, then adds our span extractor and mask feedback loop incrementally, showing performance on Partonomy-PartImageNet and TextVQA across seven KL-weights.\n\nPlotting all combinations of ±span extractor/±feedback loop is infeasible, but Figure 5 covers 3/4: LISA (–SE, –F), PLUM no feedback loop, and PLUM (both components). At reviewer request, we added LISA + feedback loop. Both components are beneficial.\n\n|Model | micro-gIoU | macro-gIoU|\n|-- | -- | -- |\n|LISA *(-SE, -F)* | 65.6 | 67.7 |\n|LISA + Feedback Loop *(-SE, +F)* | 66.7 | 68.6 |\n|PLUM - Feedback Loop *(+SE, -F)* | 61.4 | 73.9 |\n|PLUM *(+SE, +F)* | **67.9** | **80.3** |\n\nWe also ablate the span extractor by removing the bidirectional layer. (ReasonSeg, conducted on epoch=5/25 due to shortage of time):\n\n| | gIoU| cIoU|\n|-|-|-|\n| PLUM (w/ bidirectional)| 41.86 | 34.97 |\n| PLUM (w/o bidirectional)| 37.28 | 31.54 |\n\n---\n\n> **W6: Lack of Analysis on Loss Weights**\n\nAppendix A.1 shows PLUM uses hyperparameters from LISA, GLaMM, and PixelLM ($\\lambda_{CE}=1.0, \\,\\lambda_{BCE}=2.0$); only $\\lambda_{seg}$ and $\\lambda_{cls}$ differ. The table below shows the search for our choices:\n\n| $\\lambda_{CE}$ | $\\lambda_{seg}$ | $\\lambda_{BCE}$ | $\\lambda_{cls}$ | $\\lambda_{KL}$ | lr | gIoU | cIoU |\n|-|-|-|-|-|-|-|-|\n|1.0|2.0|2.0|2.0|0.1|3e-4|0.385|0.331|\n|1.0|4.0|2.0|2.0|0.1|3e-4|0.489|0.506|\n|1.0|5.0|2.0|2.0|0.1|3e-4|0.517|0.494|\n|1.0|6.0|2.0|2.0|0.1|3e-4|0.527|0.501|\n|1.0|8.0|2.0|2.0|0.1|3e-4|0.573|0.546|\n\nWe also show early results on ReasonSeg with $\\lambda_{cls}$. Increasing $\\lambda_{cls}$ leads to slightly better performance.\n\n|$\\lambda_{CE}$|$\\lambda_{seg}$|$\\lambda_{BCE}$|$\\lambda_{cls}$|$\\lambda_{KL}$|lr|gIoU|cIoU|\n|-|-|-|-|-|-|-|-|\n|1.0|2.0|2.0|0.5|0.5|3e-4|0.110|0.133|\n|1.0|2.0|2.0|1.0|0.5|3e-4|0.103|0.108|\n|1.0|2.0|2.0|2.0|0.5|3e-4|0.127|0.138|\n|1.0|4.0|2.0|0.5|1.0|3e-4|0.239|0.184|\n|1.0|4.0|2.0|2.0|1.0|3e-4|0.263|0.181|\n\n---\n\n> **W7: No Evaluation of Span Extractor’s BIO Classification** \n\nPlease refer to W2 for these results.\n\n---\n\n> **W8: Misleading Conclusion in L268–273**\n\nWe acknowledge this is an unintended misrepresentation of the results. Table 2 shows that predicting objects prior to part masks (Whole2Part) has clear benefits for the pretrained models—all four models have higher micro and macro gIoUs than Part2Whole. This advantage evaporates once the models have seen enough part data, as noted about the finetuned models.\n\nConditioning also shows clear benefits in Table 3, where knowing the object assists with predicting its parts (Whole2Part's PA >= Part2Whole's PA) and knowing parts helps predict the object (Part2Whole's OA >= Whole2Part's OA). \n\nWe will clarify both of these points in the text.\n\n---\n\n> **W9: No Discussion of Results in Figure 4**\n\nFigure 4 motivates our work (noted on line 201) by showing existing segmenting LMMs perform poorly on part segmentation benchmarks despite training. Finetuning on Partonomy boosts performance, and PLUM outperforms other baselines in zero-shot part segmentation, demonstrating strong generalizability. We will expand the discussion.\n\n---\n\n> **W10: Lack of Insightful Analysis in Table 3**\n\nOne key finding of Table 3 is that prior knowledge of objects increases part selection accuracy (and vice versa). This is nontrivial as we intentionally create difficult wrong answer choices close to the correct answer:\n\n```\n> \"The dirt bicycle in the image has the following visible parts: frame, handlebars, knobby tires/wheels, pedals, and seats.\",\n> \"The dirt bicycle in the image has the following visible parts: frame, handlebars, lock, and seats.\",\n> \"The dirt bicycle in the image has the following visible parts: frame, grip, handlebars, knobby tires/wheels, pedals, seats, and tires/wheels.\",\n...\n```\n\nWe highlight this difficulty on line 266 . Answer similarity is reflected by the high precision and recall of the \"Random\" baseline in Table 3. \n\nModel variance is explained in part by the models' pretraining datasets. PixelLM and GLaMM incorporate their own data into their pretraining mixes (noted in W11 below). These pretraining differences can make a large difference on the zero-shot Partonomy-Core, many of whose parts are not contained in the Partonomy training splits. \n\nMultiple-choice accuracy drops after finetuning, likely due to catastrophic forgetting and low-variance data; next-token prediction alone seems insufficient for part detection, suggesting a future research direction.\n\n---\n\n> **W11: Unsubstantiated Claim About Training Data Size**\n\nGLaMM and PixelLM are trained on large segmentation datasets: GLaMM on GranD (7.5M concepts, 810M regions) and PixelLM on MUSE (239k instances, 910k regions). We will add these details.\n\n---\n\n> **W12: Missing Metric Descriptions in Table 3** \n\nWe describe the Precision ($P$), Recall ($R$) and Part and Object Accuracy ($PA$ and $OA$) metrics in the caption of Table 3 and on lines 186-191. We use $P$ and $R$ to capture the similarity between the predicted part texts and the ground-truth part texts. $PA$ and $OA$ denote all-or-nothing prediction accuracy.\n\n---\n\n> **Minor 1: function g and value N+ are undefined** \n\nWe appreciate the reviewer’s note for clearer definitions. The function $g$ is a single linear projection layer whose output $q_i$ acts as a query in the SAM decoder. $N+$ refers to the number of positive (B/I) tokens. We will clarify these in the manuscript.\n\n---\n\n> **Minor 2: Figures should be rendered as vector graphics**\n \nWe will rework Figure 4 to use vector graphics or be of higher resolution for clarity.\n\n> **Limitations**\n\nA shortened version of our limitations:\n\nPartonomy-Core has the most part labels among part-grounding benchmarks but omits rare/domain-specific concepts and lacks object variety versus PartImageNet++. Extending to more concepts (e.g., via PartImageNet++) would improve part-level LMM understanding.\nThough PLUM addresses key architectural challenges, part segmentation and identification remain difficult. Autoregressive feedback may accumulate errors and requires sequential decoding.\nAs an LLM, PLUM can be misused (e.g., for generating fake or abusive content)."}, {"review_id": "VkWGd0oITa", "reviewer": "Reviewer_KqXb", "summary": "The paper investigates fine-grained part understanding with large multimodal models (LMMs). In doing so, the authors first introduce a pixel-level part grounding benchmark, PARTONOMY, which contains a training split and a test split, aiming to examine the capabilities of LMMs for part grounding, part comparison, and part-whole relationship reasoning. Moreover, they identify two issues with existing part grounding models: a distribution shift due to the reliance on a special segmentation token and a lack of modeling history part groundings. They further address the issues using an improved pre-training strategy and deliver a more performant LMM: PLUM. The experimental results demonstrate the superior performance of PLUM.", "questions": "Text evaluation in Table 3 shows marginal performance gaps between different models. Does that mean the answer choices construction is suboptimal? The authors argue that \"each answer choice shares extensive lexical overlap.\" Is it possible to quantify this, and are there any plans to resolve this issue?", "limitations": "1. \\# images and segmentation masks in Table 1 seem missing.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## strengths\n\n1. The paper addresses an important problem: pixel-level part understanding, which has been largely overlooked in LMM research.\n2. The authors introduce a benchmark and a set of evaluation metrics for part understanding.\n3. A new training paradigm is proposed to address the limitations of existing segmentation LMMs in part grounding.\n4. A new LMM, PLUM, which is trained using the proposed learning paradigm, is delivered and further demonstrates strong part-understanding performance.\n\n## weeknesses\n\n1. The chosen baseline open-vocabulary segmentation models seem outdated. Here are some more recent models: Grounded SAM/DINO: https://github.com/IDEA-Research/Grounded-SAM-2; https://arxiv.org/abs/2412.16334v1\n2. Lacking comparison with frontier models like GPT-4o. A potentially stronger baseline could be a combination of pre-trained VLMs for visual reasoning and Grounded SAM/DINO for part grounding. Including this baseline can better justify the superiority of the proposed unified model.\n3. Lacking an in-depth analysis of the span extractor. The incorporation of a specific span extraction module might complicate the model design. VLM should be able to identify segmentation-related text spans or can be trained to predict them directly.", "was_revised": true, "final_justification": "accept. a solid work and rebuttals cleared my concerns.", "rebuttal": "We sincerely appreciate the time and effort you have dedicated to reviewing our work. In the following, we have provided our response to your questions and comments for clarification.\n\n> **W1: Outdated Open-Vocabulary Segmentation Model:**\n\nWhile our work focuses on segmenting LMMs and the performance of open-vocabulary segmentation models does not change our conclusions, we agree that the comparison provides helpful context. We actually played with Grounded-SAM in preliminary experiments but chose to investigate other models due to Grounded-SAM's poor performance. The DINO.txt model was not released at the time of our paper's submission, but we agree it appears promising. We value the suggestion to include newer baselines and will provide additional results on newer open-vocabulary segmentation models in the camera-ready.\n\n---\n\n> **W2: Lacking comparison with frontier models like GPT-4o:**\n\nIn Table 2, the part grounding using open-vocabulary segmentation model is done by providing these models with the ground-truth (gt) part text labels. Meaning, their results in Table 2 can be seen as an upper bound for GPT-4o pipelined with X-Decoder, SEEM, or GroundedSAM/DINO (and other additional open-vocabulary segmentation models). Nonetheless, we do agree that this experiment could better highlight the need for a unified, end-to-end model such as PLUM, so we will include additional experiments on GPT-4o augmented with open-vocabulary segmentation models in our upcoming camera-ready manuscript.\n\n---\n\n> **W3: Lacking an in-depth analysis of the span extractor** \n\nWe agree that performing tagging on top of the LMM requires additional data preprocessing to ground the answer in the original text. However, we would like to respectfully point out that reducing the LMM's distribution shift and the subsequent performance gains pretrained PLUM obtains over its peers justifies the addition of the span extraction module. To further analyze our span extractor, we provide additional experiments in the following tables:\n\nTo demonstrate the importance of bidirectional attention, we train PLUM with and without the bidirectional self-attention layer for BIO tagging. We finetune on the training split of Partonomy-PartImageNet and evaluate on its validation set.\n\n|                   | b-acc | i-acc | o-acc |\n|-------------------|-------|-------|-------|\n| PLUM (w/ bidirectional layer) | 98.59 | 87.32 | 99.98 |\n| PLUM (w/o bidirectional layer) | 100.00 | 15.86 | 99.78 |\n\nThe results show that the bidirectional layer plays an essential role in BIO tagging, and thus the span extraction. The result suggests that the bidirecitonal layer plays a critical role in identifying the text spans (as shown by the `i-acc`), while `b-acc` appears to be high in both settings. We hypothesize that finetuning on a fixed pattern of answers in Partonomy, e.g., `ASSISTANT: frame, handlebars, lock, and seats` across the dataset should have given the easy positional bias for the model to exploit, achieving good accuracy for `B` class. Nonetheless, the substantial loss of accuracy for the `I` class shows that the bidirectional layer is an essential part of our proposed PLUM model's span extraction capability.\n\nWe also provide additional results for Referring Expression Segmentation to demonstrate the importance of employing the bidirectional attention layer as the span extractor to identify the textual span for segmentation. We note that referring expression segmentation is more challenging than simple part text tagging as it requires highlighting longer text spans for segmentation (Evaluation was performed on RefCOCO | Unc | Val set; “ft” indicates finetuned on RefCOCO training dataset):\n\n| | b-acc | i-acc | o-acc |\n|-|-|-|-|\n| PLUM (w/ bidirectional layer) (ft) | 99.98 | 99.87 | 100.00 |\n| PLUM (w/o bidirectional layer) (ft) | 6.68  | 4.92  | 99.98  |\n\n\n---\n\n> **Question: Text evaluation in Table 3 shows marginal performance gaps between different models.**\n\nThank you for your question. We would like to kindly note that the high lexical overlap among answer choices is intentional. Our goal was to keep each answer choice as similar as possible to force the model to pay attention to small changes in the parts (e.g., “wings, landing gear and propulsion component” vs. “wings, guidance system and propulsion component”) when determining the most probable answer choice. The low accuracy and high variance in Table 3 suggest that the next-token prediction objective is insufficient for part detection and to establish part-whole relationships between objects and their constituent parts. This indicates a promising area for future research (e.g. contrastive losses on close answer choices).\n\nTo quantify the overlap, we tokenized each answer choice and computed the average IoU between the correct answer choice and the most similar wrong answer choice. If we consider all tokens in each answer, we obtain an overlap of 89.43% (on average, the most similar answer choice has 89% of the same tokens as the correct answer). If we only consider the parts, we obtain an overlap of 41.01% (on average, the most similar answer choice has 41% of the same parts as the correct answer). An example of a challenging set of answer choices is below. Here, the first answer choice is the correct answer.\n\n```\n> \"The dirt bicycle in the image has the following visible parts: frame, handlebars, knobby tires/wheels, pedals, and seats.\",\n> \"The dirt bicycle in the image has the following visible parts: frame, handlebars, lock, and seats.\",\n> \"The dirt bicycle in the image has the following visible parts: frame, grip, handlebars, knobby tires/wheels, pedals, seats, and tires/wheels.\",\n> \"The dirt bicycle in the image has the following visible parts: antennas, frame, handlebars, knobby tires/wheels, pedals, and seats.\",\n> \"The dirt bicycle in the image has the following visible parts: frame, handlebars, knobby tires/wheels, pedals, pivot, pop filter, and seats.\"\n```\n---\n\n> **Limitations: # images and segmentation masks in Table 1 seem missing.**\n\nThank you for pointing out the omission. The numbers are as follows: the number of images = 74,500, and the number of segmentation masks = 407,101. We have updated Table 1 with these values."}, {"review_id": "YkQJCULHgk", "reviewer": "Reviewer_bML3", "summary": "This paper introduces a new benchmark called PARTONOMY, which requires Large Multimodal Models (LMMs) not only to identify the generic parts of an object but also to learn the part-whole relationships between entities. Experiments on baseline models such as LISA show that current methods fail on this benchmark, thereby highlighting the gap in their part-grounding abilities. Furthermore, the authors address two weaknesses of existing LMMs in segmentation tasks: the need for an additional special token and the auto-regressive paradigm. They propose a new LMM called PLUM that uses span tagging rather than a special token and conditions its predictions on prior ones in a feedback loop. Several experiments on different tasks show PLUM's outstanding performance.", "questions": "PLUM generates part masks serially, one by one. It is questionable whether this is the most efficient approach. For a prompt like, \"Please segment the airplane's wings, engines, and tail,\" could the model first fully comprehend all target parts from the text and then generate all masks in parallel? Parallel decoding could potentially be faster and might avoid the error accumulation issue mentioned above. The authors should provide a stronger justification for adopting a serial generation approach.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Stengths\n\n1. The proposed PARTONOMY benchmark provides a valuable research topic and resource for the community. It effectively demonstrates a key weakness of existing LMMs: their failure to identify part-whole relationships.\n\n2. The observation of the two weaknesses is insightful and correct. The proposed methods address these issues to a certain extent.\n\n3. Using span tagging instead of a `[SEG]` token avoids the need to train an additional special token, which can be beneficial for fine-tuning with limited data.\n\n4. Introducing past information as a condition is a good idea to enhance the segmentation ability of LMMs.\n\n5. Experiments show that PLUM achieves better performance on various tasks, even when using less data compared to its competitors.\n\n## Weaknesses\n\n1. A dataset of 1k images may seem insufficient for a benchmark. I am wondering if the authors can prove that 1k images are enough to robustly evaluate LMMs. Specifically, the authors could evaluate models using varying numbers of images from PARTONOMY and plot a performance curve to determine the point at which the results converge, thereby demonstrating the sufficiency of the benchmark size.\n\n2. The second problem addressed by the authors is very similar to that in paper [1], which also introduces entity relationships into LMMs. It would be better to discuss the relationship with this paper and include it as a baseline for comparison.\n\n3. PLUM designs a Mask Feedback Loop to use historical masks as conditions to enhance future predictions; however, this inevitably introduces error accumulation. Specifically, if the model outputs a wrong or inaccurate mask, subsequent predictions will be dynamically affected. The paper should discuss this phenomenon.\n\n4. The authors frame this as an entirely new task. However, it could essentially be considered a more challenging variant of \"Referring Expression Segmentation\" that involves complex reasoning, or a form of \"Multi-object Segmentation.\" The core objective remains the segmentation of specific image regions based on textual instructions. While the question formats (e.g., Part Comparison, Part-Whole Reasoning) are well-designed, defining this as a completely new and independent task category may be an overstatement.\n\n[1] SegLLM: Multi-round Reasoning Segmentation with Large Language Models", "was_revised": true, "final_justification": "The author has addressed my concerns about the insufficient benchmark and my confusion regarding the design of the Mask Feedback Loop. I think the paper will be ready for acceptance once the author incorporates the discussed revisions into the final version. Therefore, I have decided to raise my score.", "rebuttal": "We sincerely appreciate the time and effort you have dedicated to reviewing our work. In the following tables, we have provided our response to your questions and comments.\n\n> **W1: A dataset of 1k images may seem insufficient for a benchmark.**  \n\nWe appreciate the concern about the small evaluation set. We would like to note that in addition to Partonomy-Core, we also provide evaluation results on holdout sets of the other part datasets (Figure 4). Further, 1,000 examples correspond to ≈ 5K segmentation masks. Additionally, past benchmarks have far fewer examples (e.g., LISA's ReasonSeg benchmark which defines the task has only 200 evaluation samples). Our goal was to provide a wide range of highly technical parts and concepts, instead of providing more examples of fewer parts (an easier annotation process) as in previous benchmarks.\n\nFollowing the suggestion to compute performance on varying amounts of evaluation data, we evaluate the finetuned LISA model on varying percentages of our 1K Partonomy-Core examples. We see that results are relatively stable across evaluation sizes.\n\n| Model (% Eval Dataset) | Identification | Intersection | Difference |\n| - | - | - | - |\n| LISA ft (20%) | 34.7 / 35.9 | 38.3 / 40.0 | 30.4 / 32.8 |\n| LISA ft (40%) | 32.7 / 34.2 | 37.0 / 37.8 | 29.0 / 29.4 |\n| LISA ft (60%) | 33.9 / 34.1 | 38.0 / 39.5 | 31.3 / 32.7 |\n| LISA ft (80%) | 33.2 / 35.2 | 37.0 / 38.2 | 30.7 / 31.8 |\n| LISA ft (100%) | 33.6 / 35.4 | 37.0 / 38.4| 30.4 / 31.6 |\n\n---\n\n> **W2: Similar Motivation to SegLLM**\n\nThank you for referring us to the work. SegLLM indeed has a similar architectural motivation in noting that existing segmenting LMMs are unable to refer to past mask predictions. However, the primary motivation of SegLLM is to perform multi-round conversations where the user refers to previous objects—SegLLM is even trained with an explicit loss function for the referent mask. Our goal, in contrast, is to broadly enable mask decoding within the context of prior masks, applying this to part segmentation. This is reflected in our architectural change without any additional objective function.\n\nSome core differences are as follows:\n- SegLLM passes previously encoded masks through the entire LLM. In contrast, we encode prior masks (conditioned on their corresponding text) and pass them right back into the mask decoder without having to propagate through the LLM backbone. While their approach is more expressive (having passed the masks through the LLM), it is much more computationally expensive: SegLLM must perform as many LLM forward passes as there are [SEG] tokens, whereas ours only requires a single LLM forward pass.\n- Each of SegLLM's mask embeddings require a forward pass of the masked image through a CLIP vision encoder and a projector. Ours, by contrast, takes the low-resolution predicted masks and encodes them with SAM's lightweight prompt encoder.\n- SegLLM is trained with teacher forcing on the segmented masks. Ours is trained directly on its generated predictions which may make it more robust to poor mask predictions.\n- SegLLM is trained with a loss function that explicitly forces the model to decode referenced masks. Our decoding process requires no \n\nWe will include a discussion of SegLLM in the related works and would be happy to train it as an additional baseline due to its similarity.\n\n---\n\n> **W3: PLUM designs a Mask Feedback Loop Introducing Error Accumulation:**\n\nIt is true that the common characteristic of conditioning on the past predictions to generate the current output (e.g., autoregressive text generation) can introduce error accumulation. However, we would like to underscore the great success with models which decode autoregressively (e.g. LLMs) and while a possibility, it is not a guarantee that earlier errors will affect future ones (unless the earlier predictions are significantly off-the-mark, which we have not observed yet).\n\nWhile theoretically plausible, we manually inspected samples with and without the feedback loop and did not observe significant error accumulation. In fact, both models seemed to make very similar mistakes, sometimes incorrectly predicting the first mask but going on to predict the remaining ones correctly. Differences arose when the feedback loop predicted a larger object area, as future masks were less likely to be contained in that previously predicted object area.\n\nWe will discuss these analyses and drawbacks in our limitations and supplementary material.\n\n---\n\n> **W4: Defining Explanatory Part Segmentation as an entirely new task is an overstatement.**\n\nWe do agree that our proposed Explanatory Part Segmentation task can be viewed as a variant of previous segmentation tasks in some way. We, nevertheless, would like to kindly note that our proposed task is beyond simple grounding of textual phrases, but requires implicit visual reasoning of part-whole relations across different visual objects. Furthermore, our task is notably distinct in its level of challenge (e.g. even though the pretrained LMMs are trained on part datasets, their performance remains low), and our question formulation requiring capabilities beyond pixel-level grounding (e.g. requiring reasoning over intersection, difference, and part-to-whole) make it unique from the existing referring segmentation landscape. We will note our task's relationship to referring segmentation in the related works.\n\n---\n\n> **Q: Generating part masks serially may not be the most efficient**\n\nIndeed, generating the part masks serially is slower than parallel decoding. However, our experiments show that the feedback loop improves performance, and we would argue that the benefits outweight the costs. Further, timing experiments show that mask prediction is dominated by the LLM forward pass: averaged over 1000 examples, the LLM forward pass took 2.47 seconds, decoding without a feedback loop took .0097 seconds, and decoding with a feedback loop took .0162 seconds, a .2% increase compared to the language model's forward pass time.\n \nAn architecture that performs joint attention over predictions and refines the masks based on the attentional outputs in parallel would be a promising direction for future research."}, {"review_id": "kbsLv0lonN", "reviewer": "Reviewer_Ke3p", "summary": "This paper addresses a key limitation of large multimodal models (LMMs): their inability to perform fine-grained part-level visual understanding. They propose a dataset/benchmark PARTONOMY  for \"Explanatory Part Segmentation,\" evaluating LMMs on identifying object parts, comparing parts across objects, and reasoning about part-whole relationships. It spans 787 objects and 2,768 part labels, with diverse coverage beyond prior datasets. A part-level segmenting LMM PLUM is proposed to improve part-level understanding by avoiding special [SEG] tokens (via span tagging) and using a feedback loop to refine segmentations with prior masks. PLUM outperforms state-of-the-art LMMs on part segmentation tasks while retaining strong performance on general vision-language tasks.", "questions": "1. PLUM’s feedback loop uses FiLM layers to condition on prior masks. How are multiple prior masks (e.g., for overlapping parts) weighted or disambiguated? Does the loop ever propagate errors (e.g., a poorly segmented first part worsening subsequent masks)?\n2. The authors use Llama 2-70B to generate part labels for PARTONOMY-Core, then manually prune \"non-visible\" parts. How were disagreements between annotators resolved? Was inter-annotator agreement measured?", "limitations": "Yes; the authors partially addressed limitations in the paper. For additional limitations, please refer to the \"Weaknesses\" section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The benchmark and model are rigorously designed. PARTONOMY-Core’s manual annotation ensures high-quality part labels, and PLUM’s architecture (span tagging, feedback loop) is theoretically motivated to mitigate distribution shift and improve segmentation consistency.\n2. Part-level understanding is a foundational yet underexplored challenge for LMMs. PARTONOMY fills a critical need for standardized evaluation, while PLUM provides a novel architectural paradigm to improve grounding without sacrificing general reasoning.\n3. The span tagging mechanism (replacing [SEG] tokens) and mask feedback loop are innovative solution.\n4. The paper is well-written and easy to follow.\n\n\nWeaknesses: \n1. The FiLM-based feedback loop’s implementation details are vague. For example, how prior masks are encoded into feature maps and pooled via \"patch-wise attention\" is not fully explained.\n2. PLUM’s feedback loop and span tagging may increase computational cost, limiting real-world deployment.", "was_revised": true, "final_justification": "The authors have addressed my concerns in detail and I would like keep my original rating as 5.", "rebuttal": "We sincerely appreciate the time and effort you have dedicated to reviewing our work. In the following, we have provided our response to your questions and comments for clarification.\n\n> **W1: The FiLM-based feedback loop’s implementation details are vague.**\n\nWe elaborate the two specific components in our proposed Mask Feedback Loop mechanism down below:\n(i) **FiLM**, in the original paper, was introduced to condition the visual features in the intermediate layers on latent text representations, which is specifically what we used the FiLM layer for in PLUM to condition the mask region representation on the text span embeddings. This enables us to retain the text-conditioned mask representations throughout the feedback loop.\n(ii) **Patch-wise Attention Pooling** was used in PLUM to temporally incorporate information across all prior masks; meaning, we use the attention pooler to encode the previously generated masks before we pass them into the mask prompt encoder.\n\nSpecifically: we modify SAM's prompt encoder by injecting FiLM layers between its convolutional layers. We encode each mask using the modified prompt encoder and condition its FiLM layers using the text embeddings corresponding to each mask. This yields a set of feature maps, one for each mask. As SAM's mask decoder takes a single feature map as its mask prompt, we then perform a patch-wise attention pool to aggregate information across all feature maps into a set of learnable CLS embeddings. These CLS embeddings constitute the feature map that is passed into SAM's mask decoder as the mask prompt.\n\n---\n\n> **W2: PLUM’s feedback loop and span tagging may increase computational cost, limiting real-world deployment.**\n\nSpan tagging happens in parallel (a single forward pass over the embeddings). Our feedback loop is a straightforward approach that improves performance—however, it is true that parallelizing our current feedback loop design isn't possible, suggesting an architecture which decodes in parallel, performs joint attention over predictions, and refines the masks based on the attentional outputs in parallel would be a promising direction for future research. However, timing experiments show that mask prediction is dominated by the LLM forward pass: averaged over 1000 examples, the LLM forward pass took 2.47 seconds, decoding without a feedback loop took .0097 seconds, and decoding with a feedback loop took .0162 seconds, a .2% increase compared to the language model's forward pass time.\n\n> **Q1: How are prior masks weighted or disambiguated? Error Propagation?**\n\nPrior masks are not manually weighted in the patch-wise attention pool—instead, the learned CLS tokens (each corresponding to a patch) each perform attention across the patches for the embedded masks. We do not disambiguate between mask embeddings (e.g. we do not add temporal embeddings) as we believe that temporal decoding order should not affect performance. \n\nAutoregressive models like our feedback loop can indeed introduce error accumulation. However, we manually inspected samples with and without the feedback loop and did not observe significant error accumulation. In fact, both models seemed to make very similar mistakes, sometimes incorrectly predicting the first mask but going on to predict the remaining ones correctly. Differences arose when the feedback loop predicted a larger object area, as future masks were less likely to be contained in that previously predicted object area.\n\n\n> **Q2: How was Annotator Agreement Resolved?**\n\nAs noted, our ontology construction process started with Llama-2-70B proposing a set of parts (especially useful for highly technical objects, like military vehicles), then an initial pruning occurs based on individual research. However, annotators do not know with certainty whether a part is commonly present in images of the object until they start annotating them. Therefore, we do not measure inter-annotator agreement for part pruning, but instead did a three-round filtering process including the initial pruning after research, then large-scale part annotation with further pruning, and finally a third filtering/refinement pass where annotations are refined and parts not found in the images are dropped."}]

Author rebuttal:

Meta-review: Reviewers think that this paper addresses an important problem, as well as introducing a benchmark which would provides a valuable research topic and resource for the community. After rebuttal and discussion, most of the concerns raised by reviewers are resolved the authors and all the ratings are positive.  I thereby recommend to accept this submission.
Decision: Accept (spotlight)