MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentation
Authors: Donggon Jang
Yucheol Cho
Suin Lee
Taehyeon Kim
Daeshik Kim
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Dataset
Multi-target and Multi-granularity Reasoning Segmentation
Benchmark
Primary area: datasets and benchmarks
TL;DR: Introducing the Multi-target and Multi-granularity Reasoning (MMR) dataset and a new framework for detailed object and part-level reasoning segmentation in complex, multi-target scenarios.

Abstract: The fusion of Large Language Models (LLMs) with vision models is pioneering new possibilities in user-interactive vision-language tasks. A notable application is reasoning segmentation, where models generate pixel-level segmentation masks by comprehending implicit meanings in human instructions. However, seamless human-AI interaction demands more than just object-level recognition; it requires understanding both objects and the functions of their detailed parts, particularly in multi-target scenarios. For example, when instructing a robot to \textit{“turn on the TV"}, there could be various ways to accomplish this command. Recognizing multiple objects capable of turning on the TV, such as the TV itself or a remote control (multi-target), provides more flexible options and aids in finding the optimized scenario. Furthermore, understanding specific parts of these objects, like the TV's button or the remote's button (part-level), is important for completing the action. Unfortunately, current reasoning segmentation datasets predominantly focus on a single target object-level reasoning, which limits the detailed recognition of an object's parts in multi-target contexts. To address this gap, we construct a large-scale dataset called Multi-target and Multi-granularity Reasoning (MMR). MMR comprises 194K complex and implicit instructions that consider multi-target, object-level, and part-level aspects, based on pre-existing image-mask sets. This dataset supports diverse and context-aware interactions by hierarchically providing object and part information. Moreover, we propose a straightforward yet effective framework for multi-target, object-level, and part-level reasoning segmentation. Experimental results on MMR show that the proposed method can reason effectively in multi-target and multi-granularity scenarios, while the existing reasoning segmentation model still has room for improvement. The dataset is available at \url{https://github.com/jdg900/MMR}.

Published as a conference paper at ICLR 2025
MMR: A LARGE-SCALE BENCHMARK DATASET FOR MULTI-TARGET AND MULTI-GRAUULARITY REASON-
ING SEGMENTATION
Donggon Jang" Yucheol Cho" Suin Lee Tachyeon Kim Dac-Shik Kim
Department of Electrical Engineering, KAIST {idg900, yo cho, suinlee, rlaxogus0814 4, daeshiikjkaiist.ac.kk
ABSTRACT
The fusion of Large Language Models (LLMs) with vision models is pioneering new possibilities in user-interactive vision- rision-language tasks. A notable application is reasoning segmentation, where models generate pixel-level segmentation masks by comprehending implicit meanings in human instructions. However, seamless human-Al AI interaction demands more than just object-level recognition; it requires understanding both objects and the functions of their detailed parts, particularly multi- multi-target scenarios. For example, when instructing a robot to "turn the TV", there could be various ways to accomplish this command. Recognizing multiple objects capable of turning on the TV, such the TV itself (multi-target), provides more flexible options and aids in finding the optimized scenario. Furthermore, understanding specific parts of these objects, like the TV's button or the remote's 's button (part-level), is important for completing the action. Unfortunately, current reasoning segmentation datasets predominantly focus on a single target object-level reasoning, which limits the detailed recognition of an ob- ject's parts in multi-target contexts. To address this gap, we construct large-scale dataset called Multi-target and Multi-granularte Reasoning (MMR). MMR com- prises 194K complex and implicit instructions that consider multi-target, object- level, and part-level aspects, based on pre-existing image-mask sets. This dataset supports diverse and context-aware interactions by hierarchically providing ob- ject and part information. Moreover, propose straightforward yet effective framework for multi-target, object-level, and part-level reasoning segmentation. Experimental results MMR show that the proposed method can reason effec- tively in multi-target and multi-granularity scenarios, while the existing reasoning segmentation model still has room for improvement. The dataset is available https: :
remote control
/tte::///tttuut httptbttt.uu.ggogggg com 3000://dg9000MM
INTRODUCTION
Human-machine interaction is key focus in AI for real-world applications, driving interest multi- modal perception models that integrate vision and language modalities. The model perceives the context within the image related to explicit text query inputs and predicts pixel-level masks or bound- ing boxes accordingly. For example, Open Vocabulary Segmentation (OVS) (Liang al. 2023 Cho et al. 2023 Xu et al. (023). leveraging models like CLIP (Radford et al. (2021), generates segmen- lation masks from open-set text categories. Similarly, Referring Expression Segmentation (RES) Waag et al. 2023) Hu et al. 2023: Liu et al.) 2023a Yang et al., (022) predicts the segmentation mask corresponding to the objects referenced by the text input within the image. However, these models encounter challenges with implicit and complex text queries, limiting their effectiveness in real-world scenarios.
The emergence of Large Language Models (LLMs) (heng et al. 2024 Roumeliotis & Tselikas 2023
Achiam et al. 2023 Zhang et al. 2023a offers promising solutions to this challenge. Recent
"Equal Contribution
1 "Correponndig Author

---

Published as a conference paper at ICLR 2025
studies (Bai al., 2023 Li al. 2023 Liu et al. 2024 Zhu al. 2023 Zhang et al. 2023b
Chen et al. 2023 You et al. 2023) have witnessed that multimodal LLMs with superior reasoning capabilities can effectively perform vision tasks when given implicit text inputs. However, current multimodal L.Ms primarily provide information corresponding to images or regions in text form,
lacking pixel-level mask generation.
To address these limitations, LISA (Lai et al. 2023) introduces reasoning segmentation. Unlike previous tasks that rely on explicit text (e.g., "steak"), reasoning segmentation handles implicit queries that require intricate reasoning or world knowledge (e.g., "the food with most protein"), by combining LLMs with the Segment Anything Model (SAM) (Kirillov et al., 2023) that has robust mask generation capabilities. LISA also introduces ReasonSeg. a benchmark dataset for reason- ing segmentation. ReasonSeg consists of 1 1,218 image-instruction pairs, each containing implicit text question-answer pairs that involve complex reasoning for each image. Nevertheless, Reason- Seg has two limitations: 1) does not adequately address scenarios involving multiple targets, and 2) it primarily focuses on object-level reasoning, treating part-level targets ambiguously. Al- though the recently proposed MUSE dataset by PixelLM (Ren et al.] 2023) addresses multi-target object-level reasoning, it does not consider part-level reasoning. These observations underscore that current datasets for reasoning segmentation overlook the complexities multiple targets and part- level scenarios, concentrating instead solely on object-level reasoning. This limitation restricts more advanced functionalities in reasoning segmentation.
In this paper, we introduce Multi-target and Multi-granularity Reasoning segmentation (MMR) dataset to overcome these limitations, which covers both multiple targets and fine-grained part- level reasoning. We collect image and mask annotations from the publicly available PACO-LVIS dataset (Ramanathan et al.] 2023). These annotations include class names and bounding box in- formation of objects and parts. Then, inspired by LLaVA (Liu al. 2024), we generate intricate question-answw answer pairs using the GPT-4V API (Achiam et al. (023). Through this, the MMR dataset contains a vast collection of 194K complex and implicit instructions for comprehensive reasoning segmentation. A distinguishing characteristic of the proposed MMR dataset is its ability to handle multiple objects and diverse parts in the question-ansee -nnswer pairs. This diverse granularity enables models to reason and comprehend complex questions about both multiple target objects and their parts within a single query, providing more meaningful and high-quality masks.
Moreover, propose simple yet effective model, tion Assistant (M'SA), for S model incorporates an early local feature fusion and multiple [SEG] tokens, which enables the model to enhance fine-grained visual understanding and consider multi-target segmentation. Ex- perimental results benchmarks, such MMR, single-target referring expression segmentation datasets, and multi-granularity referring expression segmentation dataset, demonstrate that M outperforms existing state-of-the-art methods. We believe that our dataset and model serve valuable resource for potential applications in real-world reasoning segmentation tasks, offering en- hanced versatility and robustness.
and Multi- granularity Segmenta- object-level, and part-level reasoning segmentation. The
Our contributions are summarized as follows:
We construct the MMR dataset, which includes 194K complex and implicit question pairs for multi-target, object-level, and part-level reasoning segmentation. This dataset facilitates advanced reasoning segmentation tasks in open-world scenarios.
We propose M for multi-target, object-level, and part-level reasoning segmentation. It incorporates an early local feature fusion and multiple [SEG] tokens improve fine- grained visual understanding and segment multiple targets.
Experimental results on MMR and other benchmarks show that M outperforms state- of-the-art methods, validating the effectiveness of its components.
2 RELATED WORK
Multimodal Large Language Models Recent advancements (Peng et al. 2023 Taori et al. 2023
Touvron et al. 2023 Zhang al. (022) in multimodal Large Language Models (LLMs) have greatly
improved the integration between language models and vision tasks by comprehensively understand- ing and recognizing multiple modalities. Recently proposed models such BL.P-2 (Li al. 2023
2

---

Published as a conference paper at ICLR 2025
Flamingo (Alarrac al. 2022), MiniGPT-4 ZZuu et al. 2023). Ilama-adapte (Gao al. 2023
Zhang et al. 2023a), LLaVA (Liu et al. 2024). InstructBLIP (Dai et al. 2024). IntemnPTT (Liu et al. 2023b and Qwen VL (Bai et al. 2023) have shown supenooiiyy multimodal tasks such as and captioning. leveraging the multimodal understanding capability of L.MMs. While these methods have demonstrated improved performance in vision-language tasks through instructional tuning, they only provide the text output about the visual target and focus on holistic understanding of global information in the image. Therefore, their applicability is limited in tasks requiring finer-graines understanding at the pixel level.
visual question-answerre
Reasoning Segmentation The task of reasoning segmentation, introduced by LISA (Lai et al, 2023), is understanding implicit text instruction and providing corresponding mask for the answer. This task is more challenging and important than the referring expression segmentation task which deals with explicit and simple text queries. For instance, when user wants segment pepper in an image, handling an implicit query like 'the food with a spicy taste' instead of a direct reference such as 'the pepper' is significant for improving human-Al interaction. To tackle this, LISA introduces ReasonSeg, a benchmark containing implicit text queries that require complex reasoning for each image. Recently, PixeILM (Ren et al.] 2023), has addressed the limitation of ReasonSeg which considers only single target in a query text. PixeILM constructs MUSE, new dataset with multiple target objects in the text instructions. However, both studies are still limited to object-level reasoning segmentation. Methods such as GSVA (Xia et al. 2024) and GLaMM (Rasheed et al. 2024 have also been proposed, but they focus on frameworks for object-level reasoning segmentation rather than introducing new datasets. In this paper, we extend these existing tasks and propose a new benchmark dataset that considers both part-level and object-level reasoning.
Part-level Segmentation Recent research (Li et al. 2022 Kirillov et al. 2019 Michicli et al. 2020 Zhou et al.] 2021] Pan et al. 2023) has delved into a fine-grained understanding of objects at the pert-level. For the part-level visual understanding, datasets with detailed annotations for each part are required. To this end, some initial studies (Gong et al.] 2017 Li et al. 2017]] Yang et al.)
2019 Wah al. 2011 Jia et al., 2020: Zheng et al. 2018) have introduced datasets with part-level
masks on specific domains, such as human body parts (Gong et al. 2017 Li et al. 2017 Yang
al.
2019), bird parts (Wah et al. 201T). and fashion cloth parts (Jia et al. 2020 Zheng et al.
2018
Moreover, recogmizing the need for annotations on general objects, some approaches (Chen
et al.
2014
2019 He et al. 2022 Zhou al., 2019 Meletis et al., 2020; Ramanathan 2023 Wei et al.
Mo et al.)
2024 have extended the existing object-level datasets by including more fine-grained annotations. Furthermore, there has been an attempt (Wang et al.) (023) to extend the
et al.
previous Referring Expression Segmentation (RES) task to provide part-level segmentation masks matching explicit text queries. In line with this effort, our work introduces a new dataset that includes multiple target parts and diverse implicit text queries for multi cunlti-rranulrity reasoning segmentation.
MMR DATASET
Current publicly available datasets for reasoning segmentation primarily emphasize object-level rea- soning. Consequently, Multimodal Large Language Models (MLLMs) often struggle with questions that involve multiple targets require reasoning at both the object- and part-levels. To address these limitations, we introduce the Multi-target and Multi-granularity Reasoning (MMR) dataset. MMR includes multi-target, object-level, and part-level reasoning scenarios. This dataset comprises im- ages and masks from the publicly available PACO dataset (Ramanathan et al. 2023), supplemented with implicit and complex question-answer pairs generated by the GPT-API (Achiam et al., 2023).
Unlike existing datasets, MMR includes large-scale question- question-answer answer pairs that consider multiple target cases and require reasoning at both the object- and part-levels, enhancing its versatility and applicability. In the following sections, we detail the dataset generation process (.1] describe the data filtering process (Sec. (.2). provide a statistical analysis of MMR (Sec. 3.3). and highlight its distinctiveness compared to existing datasets (Sec. 3.4).
3.1 DATA GENERATION
To generate a multi callit-trrge, object-level, and part-level reasoning segmentation dataset, we leverage
the PACO-LVIS dataset (Ramanathan et al. 2023). PACO-LVIS includes 456 object-specific part
3

---

Published as a conference paper at ICLR 2025
System Message
@ Tast Prompt
You ricell auitant capsziic anehding singl: mage
Create glozal caction for the image
Crute qpect cevansert yair recsoringg tinngg nucmrttio
sumbene ntpechname, inc inte, J. kxite 2)
nostone (beanding
Rabatorndyp bemwnen objects parts
Part be orging slact, for 3 hundlet
the funzten parerd information about part..
Gussiona thatsspurr imgtt sutiict commion sansu ressunig
@ Part Information jarnctaredd USS
i Requirements
Bull metii rond natural question answer pains Aucid mentin cotndiates names oéject parts
Object-lnol
Handle Rodey
Global
[ Bomik
Image
GPT-4V
Figure 1: The prompt used in our data creation process with GPT-4V.
classes across 75 object categories, offering 502K part-level masks and bounding boxes annotated across 273K object-level masks and bounding boxes. By utilizing these comprehensive images and multi-graan cualti-rranllartty mask annotations, we can reduce annotation costs while ensuring detailed and accurate segmentation data. To create intricate and implicit question-answer pairs for multiple target and multi-granulae
reasoning, we employ a GPT-assisted data generation scheme similar to LLaVA (Liu et al. 2024). Specifically, we adopt GPT-4V API which has robust visual understanding capabilities. Fig. I iillustrates the entire data generation process.
To guide the GPT-4V API effectively, we carefully craft prompts that include GPT role, object and part information, task prompts, and requirements. GPT role defines the persona of the GPT-4V API, informing about the context and objectives of the data generation process. Object part information provides comprehensive annotations, such object and part names within the image and their corresponding bounding box coordinates. Task prompt informs the GPT-4V API about pairs. Requirements set the
the task definition and considerations for generating question- rules and patterns that the GPT-4V API should follow when generating question-answer pairs (c.g., "questions should avoid direct mention of coordinates of objects or parts" "Q&A pairs should contain multiple objects or parts"). Please see the Appendix A.5 for the detailed prompt.
The -aassised data generation follows two-step process: 1) Global Caption Generation: GPT-4V API first generates a global caption based on the image to foster a deep understanding of its context. 2) Question-Answer Pair Generation: Leveraging this global caption along with object and part information, GPT-4V autonomously crafts multi-target, multi-granularity question-answer pairs. Carefully designed prompts and two-step generation process enable GPT-4V deeply comprehend image context and generate contextually relevant question-answer pairs.
3.2 DATA FILTERING
Despite meticulously crafted prompts for guiding GPT-4V, occasional deviations from established rules result in the generation of subpar question-answer pairs. These deviations include questions that reveal explicit target coordinates or provide overly direct hints, as well as answers that offer irrelevant information
omit essential details. To enhance the reliability of the question-answer pairs in our dataset, a rigorous filtering process essential. Therefore, we engage four skilled human inspectors to review the dataset according to strict criteria:
Logicality and Reasoning: Questions should avoid explicit target coordinates or strong hints. Non-compliant questions and their corresponding answers are removed. For exam- ple, a question like "Which part of this animal [coordinates] uses its sense of smell?" would excluded.
Coherence and Relevance: Answers lacking essential target information containing irrelevant details are corrected for precision and relevance. This includes cases where an- swers mention objects parts not provided in the annotations.
4

---

Published as a conference paper at ICLR 2025
LINCA
Caption: inserted verdicllly
Question t: Whete iamn on the batte greed to held al one and while the other and mrssrn for cutling through food?
Answert: The knnfe [204. 78, 230 530 dedigned with snife, 1to handle 1200, 2. 23B, 120) held while the km Re_ 3lde (215)
237, moant satling through
Question2: Which oby
bevnng used
sandtwin
Answer2: The kntle_ [301.84 820.65, 5.36 427.0] with its innoo 2% biado (800, 321, 601 Insorted into the sandwich being used keep
upriht
Question3:11 wanted
product information on conteiner
2821
2's
Questiond: Which 00 ects in the soen ane meant to contain trgudds its what part them touches the cuthoc they neet
Answord: The objects nant to co contaan ligudds are bottle I (468.0 of D 803 40, 315.78) and bodie 2 1200 416.18 242.2) part that bollke boom (183, Sos, sTd) and botlle 2
koadhes She (aoa they
220, 261).
Figure 2:
An example from the MMR dataset generated through our data creation process. The left and right pictures show the object- and part-level segmentation masks, respectively.
Questions and answers should be clear, concise, and free of ambi- guity. For example, ill-defined data, such asking about the function of object or part from a segmentation perspective, removed (e.g.. "What Answers should provide precise information that directly addresses the question without causing confusion.
Clarity and Precision:
function of object_ 1?").
Originally, 222K question-anae answer pairs are generated. Of these, 12.6 are filtered out through a review process conducted by the four inspectors, resulting in the final MMR dataset. Since dataset generation is a key contribution to our work, each inspector thoroughly reviews the entire set of 222K question-answer pairs. To minimize human error, we only filter out question-answer pairs flagged by two or more inspectors. This meticulous filtering regimen ensures the integrity and trustworthiness of the MMR dataset. An example of the generated qnestion-answer pairs is illustrated in Fig,
3.3 DATA STATISTICS
The MMR dataset includes 194,398 intricate and implicit question-answer pairs with 57,643 corre- sponding images and masks selected from PACO-LVIS. The entire dataset is split into distinct sets for training (154,127 pairs), validation (8,194 pairs), and test (32,077 pairs). Moreover, the test set is further categorized into three subsets: object-only, part-only, and mixed sets, providing a benchmark for evaluating multi-granularing reasoning segmentation capabilities.
Additionally, our dataset inherits rich coverage of 75 object categories and 445 part categories from PACO-LVIS, enhancing its diversity and utility. We delve into the frequency distribution per object and part category across question-answer pairs. Fig. 3 (b) and (d) provide comprehensive overview of the number of questions per object category and part category, respectively. The results show that our dataset encompasses a wide range of categories, ensuring that the question-answer pairs are biased toward specific categories and exhibit high level of diversity. Furthermore, the word clouds illustrated in Fig. 3 (a) and (c) highlight the prevalent head object and part categories, respectively. These word clouds demonstrate that our question-answer pairs are grounded com- mon and general objects and their associated parts. Fig. (e) presents statistics on the number of targets in each question-answe pair. On average, there are 1.8 targets per answer, with the maximum number of targets in single pair being 16. This demonstrates that our dataset can consider multiple targets in an image and cover diverse target reasoning. To evaluate the comprehensiveness both objects and parts in the proposed dataset, compare their occurrences within the total question- answer pairs. As depicted Fig. (0), there are I 114,704 descriptions for objects and 226,869 for parts, maintaining ratio of approximately 1:2. This ratio is reasonable because objects typically
5

---

Published as a conference paper at ICLR 2025
lal
PI pillow
Low
book chapkin Besduarmiin trash can
Mhee
pipe wax Clock
Back body
les hel net
Bar boTtoR
2ea lamp
hand
(6
Figure 3: Statistics of the proposed MMR dataset. (a) the word cloud for the object categories, (b) the number of objects per each object category in questions (log scale), (c) the word cloud for the part categories, (d) the number of parts per each part category in questions (log scale), (e) the distribution of target count in answers, and (f) the total number of expressions of objects and parts.
Table 1: Comparison among several reasoning segmentation datasets, including ReasonSeg (Lai et al. 2023), MUSE (Ren et al., 2023), and the proposed MMR. Here, part-level is an expression that refers various parts of an object that appear in the image.
Datasets
Object-level
Part-level
Mulii-tageet
Q&A pairs
GPT models
ReasonScg
X
1.2K
GPT-3.5
MUSE
/
7
214K
GPT-4V
MMR
7
194K
GPT-4V
consist of multiple parts. Therefore, it reflects a balanced distribution, contributing to the dataset's comprehensiveness and facilitating multi multi-gramularigg knowledge understanding.
3.4 COMPARISON WITH EXISTING REASONING SEGMENTATION DATASETS
Tab. presents a comparative overview of existing reasoning segmentation datasets and the proposed MMR dataset. As observed, MMR offers several notable advantages over existing datasets.
First, MMR contains 194K question-answer pairs, comparable to MUSE (Ren al. (023). and far exceeds ReasonSeg (Lai et al.) 2023) which only 1,218 question-answe pairs primarily designed for validation and testing purposes. This extensive scale facilitates both training and evaluation for reasoning segmentation.
Second, MMR supports question-answe answer pairs covering multi-target and multi-granularig (object- level and part-level) visual reasoning. Although MUSE includes multi-targe instances, its coverage is limited to object-level reasoning. This lack of part-level detail reduces its effectiveness in fine- grained visual tasks. Part-level reasoning in MMR enables more comprehensive understanding of visual contexts and hierarchical relationships between parts and objects. While ReasonSeg appears to include part-level reasoning, ReasonSeg often has ambiguous boundaries between objects and their parts because doesn't specify which object part belongs to. For instance, in a scene with a "car" and a "tire", ReasonSeg considers the "tire" as part of the "car", even if the tire is not attached. In contrast, MMR clearly distinguishes the boundaries between objects and their parts by specifying hierarchy like which object a part belongs to based on their spatial context. Additionally, unlike ReasonSeg, MMR distinguishes multiple objects of the same class within a single image at the instance level. For example, ReasonSeg might group all buses in scene under single "Bus" label. On the other hand, MMR treats them as distinct entities like "Bus.1," "Bus 2", etc. Also, ReasonSeg treats all screens simply as "sereen," whereas MMR would specify "laptop_1's screen," "laptop 2's screen." and so forth. This allows MMR to handle objects or parts of the same class separately by considering their spatial context within the image.

---

Published as a conference paper at ICLR 2025
Third, MMR leverages the advanced visual understanding capabilitiees of GPT4VV for question- answer generation. GPT-4V receives the image along with information such as class names and bounding boxes of objects and parts, enabling detailed and contextually accurate questirn-answer generation. In comparison, ReasonSeg generates questions using the language-specialized GPT-3.5 and pre-trained image tagging models, which do not fully capture the visual context, leading less relevant question-answer pairs with the image.
In summary, MMR provides substantial improvement over ReasonSeg and MUSE by including large-scale, multi-target, and multi-granulan question-answer pairs. It strengthens real-world ap- plicability, making it valuable asset for advancing research in reasoning- based segmentation tasks.
4 BASELINE FRAMEWOK
We propose a novel baseline framework for multi-target and multi-granularity reasoning seg- mentation, M2SA. M en- hances the LISA framework with two key features: 1) Early Local Feature Fusion and 2) multiple [SEG] tokens. For
Vi
Keng
SAM Masi Decoder D
King
MILMM
fucm
Early Local Feature Fusion, we extract local features from the early layer of the SAM's vision encoder, which contains fine- grained details such as image edges and boundaries. These lo- cal features are fused with the global semantic context features from the last layer of SAM's vi-
Projector
Projector
pliking
scents, and the dog, 3's oar [SE0) la used detect sounds.
CUP Valan Enceder
Large
In case the snimal hears a sound and trleco ideniity its parts would most Skely
fany
dioction,
000
tokens
LoRA
Ses
Apst
Figure 4: The overview of M framework.
sion encoder for more informative visual features in the mask decoder. Multiple [SEG] tokens overcome the LISA framework's 's limitation of a single [SEG] token, which struggles to segment multiple targets simultaneouslyy. To overcome this, we propose utilizing multiple [SEG] tokens. In our MMR dataset, we append [SEG] token to each target object and part in the answer annota- tions (e.g.. "When closing the laptop, laptop computer's screen [SEG] would come into contact with laptop computer's 's base panel [SEG]."). This approach enables the model to predict separate [SEG] tokens for each target, reducing ambiguity among multiple targets.
Model Architecture Fig. presents the overall architecture of the proposed MASA framework,
which integrates two core components: Segment Anything Model (Sriloi et al. 2023) and Multimodal Large Language Model (MLLM), specifically LLa VL et al. 2024). SAM module consists of SAM Vision Encoder (E) and SAM Mask Decoder (D), while the MLLM comprises CLIP Vision Encoder (I), vision-to-text projector (W), and Large Language Model (F). The image Ling E Rhxwx3 is fed into the SAM Vision Encoder (E), which generates global context features 1/g E(Zimg) RR 12/116xw/16xe and early local features ea = (Z Rh/11xx//166
To align the channel dimensions of by with Ug. the early local features ty are passed through two convolution layers, resulting in refined features 0j E 1p/116xw/16xx, 1y and ê; are then summed to obtain visual features Useg E Rh/166x0 1/16xc for segmentation. Simultaneously, the image Tamg is input into the CLIP Vision Encoder (I), producing visual token embeddings fimg = (5 )) E Nimg xd which are mapped to the LLM input space using the vision-to-text projector tb. In parallel, the text queries Zist are tokenized by the F's tokenizer, producing text token embeddings fezt E R.Nax The visual token embeddings fimg and text token embeddings ftxt are concatenated and processed by LLM F, resulting in output response Utat [[concat(fimg ftat)) Dtzt contains the textual response to the text query and special [SEG] tokens that correspond to each target entity to be segmented. These multiple [SEG] token embeddings extracted and projected into SAM's prompt space via the projector o resulting in embeddings freg = of [SEG) E Nee, Xc, Finally, the SAM Mask Decoder (D) takes the visual features Useg and the multiple [SEG] token embeddings
7

---

Published as a conference paper at ICLR 2025
fae input to generate the segmentation mask M D(concat(Veeg) Saeg)), which identifies the target regions in the image corresponding to the text queries.
Optimization Our model is trained end-to-end through two sources of supervision. For the text generation, we compute auto-regressive cross-entropy loss List between the text output Øtzt and the ground-truth text answer Vtrt- For the high-quality segmentation mask generation, the mask loss Lmaak calculated between the output mask M and the ground-truth mask M. The mask loss Timask is a weighted sum of per-pixel binary cross-entropy loss Lace and a DICE loss Laices determined by Abce and Adice. The overall loss is formulated follows:
= Lext + Lmmak:
(1)
Lmask Lbcce + Adiie
where Aboe and Adice are set to 0.5 and 2.0, respectively.
EXPERIMENT
5.1
EXPERIMENTAL SETUP
Implementation Details We use pre-trained LLaVA-7B (Liu et al. 2024 and L.a
13B with -LP--VIT-//14 (Rafford et al.) 2021) and Vicuna-7B (Chiang et al., 2023
13B (Touvron et al. 2023) to form Multimodal Large Language Model (MLLM). We adopt the pre-trained SAM-VIT-H (Kinillvv et al. (023) for the segmentation model. For CLIP-ViT-L/14 input image Fimg is resized to 224 x 224
and processed with a patch size of 14, resulting in N Ning = 256. LLM dimensions d are set to 4096 and 5120 for Vicuna-7B and Llama2- 13B. For SAM-ViT-H, e and c are 256 and 280, respectively. Efficient fine-tuning of the MILM is facil- itated using LoRA (uu et al. 2021 The trainable components in M include the SAM Mask Decoder D, the projector Φ, two convolution layers, the LoRA adapter in MLLM. and the token em- beddings. We use features from the 8th layer in the SAM Vision Encoder E for early layer feature fusion. Our model trained for 10 epochs, with each epoch consisting of 5,000 steps. We employ the AdamW (Loshchilov & Hutter 2017) optimizer with learning rate of 0.0003 and set gradient accumulation to 10 steps per update. Additionally, we use WarmupDecayL. as the learning rate scheduler. The learning rate is linearly decayed after 100 steps. The batch size and LoRA rank are set to and 8, respectively. All experiments are conducted using NVIDIA RTX A6000 GPUs. The results reported in the paper are the average values obtained from experiments conducted with 3 different random seeds.
Datasets For model training, adopt the mixed training dataset composition scheme proposed by LISA LLi et al. 2023), comprising four types: semantic segmentation datasets (ADE20K (Zhou et al. 2019), COCO-Stur (Caesar al. 20188). Mapillary (Neuhold et al.) (017). PACO-LVIS RRa- manathan et al. 2023). and PASCAL Part (Chen et al.] 20144), referring expression segmenta- al. al. 2014), RefCOCO (Kazemzadeh et al., 2014), Re-
tion datasets (RefCOCO ( (Kazemzadeh fCOCOg Mao et al. 2016), and RefCLEF (Kazemzadeh et al. 2014)), a visual question answering dataset (LLa Instructt 150K (Liu et al. 2024)). and the proposed MMR dataset for multi-target and multi-granularity reasoning segmentation. We sample the data from the mixed training dataset in ratio of 2:9:2:6, where represents semantic segmentation datasets, represents referring ex- pression segmentation datasets, represents the visual question answering dataset, and 6 represents the proposed MMR dataset.
Baseline Methods To validate the effectiveness of the M2SA for a multi-target and multi- granularity reasoning segmentation task, we adopt LISA (Lai et al. 2023). GSVA (Xia al. 2024), and GLaMM (Rasheed et al.) 2024) along with their variants. The traitrd models refer to those trained solely on their respective datasets. In contrast, the variant models referred to as model,,, are trained from scratch on mixed training set that includes the MMR dataset. Due to issues with the publicly available code from the PixelLM. we exclude PixeLM from the baseline methods to ensure reliable and consistent comparison results. For Multi- granularity Referring Expression Segmenta- tion (MRES) task, we additionally adopt the class RES models (Yang et al. 2022 Liu et al. 2023a
Wang et al.
2023 2022) and the general models (hh et al. 2022 Zou et al. 2023
2024).
8

---

Published as a conference paper at ICLR 2025
Table 2: Results MMR benchmark. The gloU and cloU metrics reported for the comparison.
Obj & Part, Obj, and Part denote multi goaniiigaanlarrty.. object-only, and part-only evaluation settings. The best results are highlighted in bold.
val
Test
Methods
Part
Part
Cbty
Part
gloU
cloU
eluU
gloU
cloli
glou
cloU
LISA-7 Lai
2023
13.8
15.3
23.5
25.1
6.6
7.9
14.3
17.5
L
19.4
31.5
34.7
41.8
13.1
19.5
27.1
VA-785p
19.8
38.9
30.2
41.1
8.0
18.6
21.2
34.5
GLaMM
(Raslseed et 2024
12.6
19.2
23.7
31.9
3.9
6.4
13.3
18.7
GLaMMM
40.3
MSSA7BB
27.8
48.6
41.0
55.6
13.5
27.9
M..9
46.8
LISA-Llama2-1 [Lai et 2023
15.4
2000
26.1
27.9
7.4
8.4
16.1
19.8
LISA-Lhamzzre
40.2
45.2
23.0
M SAA--lamm2--138
28.4
49.1
42.3
57.6
13.6
27.2
31.6
47.6
Referring expression segmentation results on RefCOCO, RefCOCO+ (Kazemzadeh al.) 2014) and RefCOCOg (Mao et al. 2016) among M and existing methods. For fair comparison with previous methods, the cloU metrics are adopted. The best results are highlighted bold.
Table 3:
Methods
ReiCOco
ReiCOCO
vakUi
ReiCOCOp
U)
MCN
LLp 8 MCN
2020 62.1
61.2
39.7
30.6
55.3
44.7
19.2
29.1
VLT
Dine 2 3
2021
67.5
70.5
65.2
56.3
61.0
50.1
55.0
$7.0
CRIS
70.5
73.2
66.1
62.3
68.1
53.7
59.9
601.4
RcLA
73.8
76.5
70.2
66.0
71.0
57.7
65.0
66.0
X-Deoo ST
Ziire ai
2023
64.6
SEEM
Zer 2012
65.7
OSVA-78
77.4
64.5
67.7
58.6
71.1
72.0
GL.MM
[Raiiss ei 2174
79.5
83.2
76.9
72.6
78.7
64.6
74.2
74.9
MPSA-18
74.0
76.8
63.
67.2
56.1
67.0
68.3
LISA-Llama2-133
77.3
M M'SA-LLamaa--135
74.6
77.6
71.0
64.0
68.1
57.6
69.0
69.3
Evaluation Metrics Following the implementation of the referring expression segmentation works, we adopt gloU and cloU scores to assess the quality of the segmentation mask. gloU denotes the mean IoU for each mask, whereas cloU is computed by the cumulative intersection area over the cumulative union area across the entire dataset. Given that cloU may exhibit bias towards large-area objects, gloU is preferable for evaluating part regions.
5.2 RESULTS BENCHMARK DATASETS
Comparison MMR Tab. 2 compares M and the baseline models in multi-target and multi-granularity reasoning segmentation task (MMR dataset). The pre-trained models perform poorly the proposed MMR dataset, particularly struggling with the part-only set due to its lack of detailed part-level understanding. Conversely, LISA,, GSVAtr and GLaMM trained using the proposed MMR dataset, exhibit superior performance as they acquire both object-level and part- level knowledge. However, its ability to handle multi-target and fine-detail reasoning remains lim- ited. contrast, the proposed M shows highly competitive performance, effectively managing multi-target scenarios and fine-detail tasks, thus showcasing its strength in comprehensive reasoning segmentation. Qualitative results are provided in the Appendix A.13
Comparison on Referring Expression Segmentation Task Tab. presents the single-target object-level RefCOCO series dataset results. While MΩSA achieves commendable performance. it is important to note that single-target referring expression segmentation a relatively simple task, involving explicit queries that focus on identifying a single object. The true strength of M'SA lies in its ability to excel in more complex and challenging tasks, such as multi-target referring expres- sion segmentation and multi-granularity referring segmentation. To evaluate its performance on multi-target referring expression segmentation, we curate text queries for multi-target objects using annotation information from the RefCOCO-send
datasets. Each query is constructed by randomly selecting to object categories from each image and generating text prompts like "Can you seg- ment the class 1, class 2, ..., and clasy n?". We then compare SAS performance against LISA,

---

Published as a conference paper at ICLR 2025
Table 4: Multi-referring expression segmentation results. We adopt the cloU metric for comparison. The best results are highlighted in bold.
Mult- RaTORRO
Man-RacCOCOG
IestA
val
IestA
val(I)
tea((U)
LISA-1B IS LA cLa 20233
37,0
32.7
36.4
26.2
28.6
28.3
45.2
18.7
GSVA-7B
2074
50.7
53.3
47.8
44.8
47.4
40.6
47.7
48.6
LISA- 1oAA-Lamaz-11 135 a 2023
35.2
32.6
12.4
27.7
20.9
26.7
44.0
47.1
MSSA-LLamn--113
72.0
75,6
68.0
62.3
67.1
36.1
65.4
65.8
Table 5: Multi-graun multi-graaularity referring expression segmentation results RefCOCOm (Wang al. 2023). For fair comparison with previous methods, the mloU metries are adopted. Part denotes part-only evaluation, and Obj & Part denotes multi-granularity evaluation. The best results highlighted in bold.
Methods
Part
Ohj &
Part
Part
Cihy & Part
Part
Ohj & Pari
SoiTk Zan al
K0N
13.9
28.2
12.1
22.8
18.1
34.7
LAVT
Yam ee
15.3
29.9
13.2
24.4
18.7
35,5
X-Deedder
2174
16.2
29.5
13.6
23.6
20.3
33,8
UnIRES
Waig Cl
2024
34,3
16.4
27.8
25.2
41,7
LISA 78
Lall & de
21.3
34,3
18.5
28.6
25.7
40.1
GSVA-78 7B
28.2
(Rannze
41.1
M'AA-7B
22.4
35.5
19.9
30.1
27.1
41.4
LISA-Llama2-11 (Lai
MPSA-LLlama-1131
24.5
37.3
21.9
31.9
28.5
42,7
GSVA, and GLaMM. As shown in Tab. M significantly outperforms these methods, show- casing its ability to reason about multiple objects simultaneously and effectively leverage its multi [SEG] tokens for diverse and intricate queries.
Additionally, we evaluate MASA on RefCOCOm, a multi-granularity referring segmentation dataset. As demonstrated in Tab. 5 M2SA surpasses existing methods in this task, though the performance improvement is less pronounced. This is likely because the MMR dataset does not include the person class, which constitutes a significant portion of the categories in RefCOCOm. These results emphasize the versatility and effectiveness of M in addressing complex, real-world scenarios, extending well beyond simple single-target segmentation tasks.
CONCLUSION
This paper addresses the limitations current reasoning segmentation datasets, which often over- look multi-target part-level reasoning. To resolve these issues, we introduce the Multi-target and Multi-granularity Reasoning (MMR) dataset, providing 194K comprehensive question-answer pairs that cover multi-target, object-level, and part-level aspects, enhancing diverse and context-aware interactions. We also propose the M model, designed for multi-target, object-level, and part- level reasoning segmentation. M incorporates early local feature fusion and multiple [SEG] tokens, improving fine-graing geaigreindd visual understanding and multi-target segmentation. Experimental results show that M outperforms existing models on the MMR benchmark. The MMR dataset aims to drive progress in reasoning segmentation by emphasizing the importance of multi-target and part-level aspects in human-Al interactions.
ACKNOWLEDGMENTS
This research has been supported by the LG Electronics Corporation. (Project No. G01230381)
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv: 087744 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 23736, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large languala model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209- 1218, 2018.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306. 2555:25306./1155, 2023.
Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. De- tect what you cam: Detecting and representing objects using holistic models and body parts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1971-1978, 2014.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open- chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://vicuna Imsys. org (accessed 14 April 2023), 2(3);6, 2023.
Seokju Cho, Heeseong Shin, Sunghwan Hong. Seungjun An, Seungjun Lee, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary se- mantic segmentation. arXiv preprint arXiv:2303.11997 2023.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 223-3223, 2016.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general- purpose vision- language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp.
2021.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, al. lama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304. 15010, 2023.
Ke Gong, Xiaodan Liang. Dongyu Zhang, Xiaohui Shen, and Liang Lin. Look into person: Self- supervised structure- sensitive learning and new benchmark for human parsing. In Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 932-940, 2017.
Ju He, Shuo Yang, Shaokang Yang, Adam Kortylewski, Xiaoding Yuan. Jie- Chen, Shuai Liu, Cheng Yang, Qihang Yu, and Alan Yuille. Partimagenet: A large, high-quality dataset parts. In European Conference Computer Vision, pp. 128-14.. Springer, 2022.

---

Published as a conference paper at ICLR 2025
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.099 09685, 2021.
Yutao Hu, Qixiong Wang. Wenqi Shao, Enze Xie, Zhenguo Li, Jungong Han, and Ping Luo. Be- yond one-to-one: Rethinking the referring image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4067-4077, 2023.
Menglin Jia, Mengyun Shi, Mikhail Sirotenko, Yin Cui, Claire Cardie. Bharath Hariharan, Hartwig Adam, and Serge Belongie. Fashionpedia: Ontology, segmentation, and an attribute localization dataset. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 1 16, pp. 316-332. Springer, 2020.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen- tation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, 2019.
pp.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026 2023.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. arXiv preprint arXiv:2308.00692, 2023.
Jianshu Li, Jian Zhao, Yunchao Wei, Congyan Lang, Yidong Li, Terence Sim, Shuicheng Yan, and Jiashi Feng. Multiple-human parsing in the wild. arXiv preprint arXiv:1705.07206, 2017.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Xiangtai Li, Shilin Xu, Yibo Yang, Guangliang Cheng, Yunhai Tong, and Dacheng Tao. Panoptic- partformer: Learning a unified model for panoptic part segmentation. In European Conference Computer Vision, pp. 729-747. Springer, 2022.
Feng Liang Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open- vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7061-7070, 2023.
Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp- 23592-23601, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang. Yi Wang, Shoufa Chen, Qinglong Zhang, Yang Yang, Qingyun Li, Jiashuo Yu, et al. Internchat: Solving vision-centric tasks by interacting with chatbots beyond language. arXiv preprint SxXiv:2305.05662.. 2023b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp- 10034-10043, 2020.
12

---

Published as a conference paper at ICLR 2025
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 11 11-20, 2016.
Panagiotis Meletis, Xiaoxiao Wen, Chenyang Lu, Daan de Geus, and Gijs Dubbelman. Cityscapes- panoptic-parts and pascal-panopia sanoptic-parts parts datasets for scene understanding. arXiv preprint Set1:2000.009444 2020.
Umberto Michieli, Edoardo Borsato, Luca Rossi, and Pietro Zanuttigh. Gmnet: Graph matching network for large scale part semantic segmentation in the wild. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16, pp. 397-414. Springer, 2020.
Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object under- standing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 909 908, 2019.
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference on computer vision, pp. 1990-4999 2017.
Tai- Yu Pan, Qing Liu, Wei-Lun Chao, and Brian Price. Towards open-world segmentation of parts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15392-15401, 2023.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.0322. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7141-7151, 2023.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing. Ming- Hiugn Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13018, 2024.
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojic Jin. Pixellm: Pixel reasoning with large multimodal model. arXiv preprint arXiv:2312.02228, 2023.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In Computer Vision-FCCCV 2016: 14th European Conference, Amsterdam, The Netherlands, October / /1-14, 2016, Proceedings, Part II 14, pp. 102-118. Springer, 2016.
Konstantinos Roumeliotis and Nikolaos D Tselikas. Chatgpt and open-ai models: A preliminary review. Future Internet, 15(6):192, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-followinng llama model, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
13

---

Published as a conference paper at ICLR 2025
Wenxuan Wang, Tongtian Yue, Yisi Zhang, Longteng Guo. Xingjian He, Xinlong Wang, and Jing Liu. Unveiling parts beyond objects: Towards finer-grane aner-granularity referring expression segmentation.
arXiv preprint S.N1::2312.00007 2023.
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1 11866-11695 2022.
Meng Wei, Xiaoyu Yue, Wenwei Zhang, Shu Kong, Xihui Liu, and Jiangmiao Pang. Ov-parts: To- wards open- vree-vocayulary part segmentation. Advances in Neural Information Processing Systems, 36, 2024.
Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang, Gsva: Gen- eralized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3858-3869, 2024.
Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Learning open- vocabulary semantic segmentation models from natural language supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2935-2944, 2023.
Lu Yang, Qing Song. Zhihui Wang, and Ming Jiang. Parsing r-enn for instance-level human analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 364 373, 2019.
Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18155-18165, 2022.
Haoxuan You, Haotian Zhang. Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07700 2023.
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero- init attention. arXiv preprint arXiv:2303. 16199, 2023a.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601 2023b.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Shuai Zheng, Fan Yang, M Hadi Kiapour, and Robinson Piramuthu. Modanet: A large-scale street fashion dataset with polygon annotations. In Proceedings of the 26th ACM international confer- ence on Multimedia, pp. 1670-1678, 2018
Bolei Zhou, Hang Zhao, Xavier Puig. Tote Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.
Tianfei Zhou, Wenguan Wang, Si Liu, Yi Yang, and Luc Van Gool. Differentiable multi cullti-granularityy human representation learning for instance- aware human semantic parsing. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 1622-1631, 2021. 2021.
Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji. Seqtr: A simple yet universal network for visual grounding. In European Conference Computer Vision, pp. 598 615. Springer, 2022.
14

---

Published as a conference paper at ICLR 2025
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing usion-langues understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Xueyan Zou, Zi-Yi Dou, Jianwei Yang. Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
15127, 2023.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jian- feng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 LIMITATION
While PACO-LVIS provides diverse and comprehensive object-part mask annotations for common objects, lacks information on the human class and its parts. Consequently, our question-answer pairs generated based on PACO-LVIS do not consider reasoning about human class and its parts, which drawback. Therefore, there is need for future dataset expansion include wider range of objects and parts that exist in real-world environments. Additionally, although we carefully design the prompts to ensure the diversity and quality of the dataset, the content of the question- answer pairs is inherently dependent on the pre-trained knowledge of ChatGPT.
A.2 ETHICS CONCERN
The MMR dataset is constructed based the publicly available PACO-LVIS dataset (Ramanathan et al. 2023), which helps mitigate privacy concerns. As the objects and parts within the images are already annotated, only add text question-answe answer pairs, ensuring that potential privacy issues main minimal. These question-answer pairs are generated using the ChatGPT/GPT-4V API (Achiam al. 2023). While there risk of bias from the training data of the ChatGPT/GPT-4V API, have implemented thorough data filtering process to remove any ethically problematic content.
A.3 LICENSE
We utilize the released code from LISA (Lai et al. 2023) for the baseline model code construction. Since LISA follows Apache License 2.0, our code is also licensed under Apache License 2.0. Ad-
ditionally, the PACO-LVIS dataset is licensed under Creative Commons Attribution 4.0 (CC BY 4.0) license. Consequently, our MMR dataset is also licensed under Creative Commons Attribu- tion 4.0 (CC BY 4.0). To download the PACO-LVIS dataset (Ramanathan al., (202)), utilize author-released code under the MIT license. We use CharGPT/GPT-4V API (Achiam et al.] 2023)
developed by OpenAl to generate the question-answer pairs in the MMR dataset. Specific licensing information for the ChatGPT/GPT-4Y API model is proprietary to OpenAl.
A.4 THE SPECIFIC DETAILS OF CHATGPT API
The specific command to use the ChatGPT API (Achiam et al. 2023) for generating question-answer pairs MMR as follows:
response = =openailCompletio.creettt
aessages= te tempe raperatur:00... max tokens =850,
Figure 5: To generate question-answer pairs in MMR dataset, we use waa--vii-----vvieeee model. For the hyper- parametar,,,,,s,, set the temperature 0.7 and max_tokens to 850.
A.5 PROMPTS AND EXAMPLES
General MMR Dataset The MMR dataset fundamentally includes multi-target (both objects and parts) answers to each question. In this section, we discuss the full prompt not covered in the main manuscript. Fig. 6 illustrates the prompt used to generate the train, validation, and test datasets. Both text and image prompts ation question-answer pairs that encompass various information about objects and parts. As shown in Fig. 2 the output includes global caption and question-answer pairs for the image. The
input into GPT-4V (Achiam et al.) 2023), resulting in the cre-
16

---

Published as a conference paper at ICLR 2025
"You are All visual assistant capable analyzing single image. receive the specific lacatinns part allong with deta led coordinatas, These coordinates are in the form bounding boxes, represented as (Xl, yl, x2. y2). These values correspond to top left x. may mutiple abjects
within image.
left bottom right and bottom rightty The height and width the
you receive 427 and 640, respectivlyy Additonally. ambiguity, we "object number" such as pperson, and
image. reselve "person_2"" 2" to differentatee between objects of the same category. If region is part of object the category name is described as "object' part. like "person's body" and "bue's wheel" : The category names and bounding box coordinates of objects and parts follow
same calegory
bottle 1 (459.077 603 49, 78
bottle_ label [450, 500, 282]
bottle 155 neck [470, 0, 593, 62]
In Inspati respatiall aalla allaoor www.orr..iigggn... www.shhnppppp-p..cc swwl-pplpnn CIEE
bottle 15 shoulder (461, 56, 603, 103]
bottle 19 body (460, 94, 604,
bottle 15 bast [463, 287, 596, 316];
1.15, 416.19. 242.2] bottle 2 [22296.85
bottle 2'5 base [307.
botlle 25 label [300, 4133, 176];
bottle 2 body [307. 172, 403, 231]:
(204.99. 2.4,
226.53);
knife 1's blade [213, 121. 237, 226]
knife handle [205. 239, 126]:
2 [30334.65
427.01
knife. 2's blade [305, 321, 601, 422]
knife_ handle ( 399, 616, 426]:
You need create global caption for the mage without given information, A global caption should summarize the content the image within maximum sentences. The need to do next is create question guestion-answers pairs using the information of objects and parts given above. However, when the corresponding object and part name appear the answers, "na me [coordinatas]" used the given information above without changing its form. The goal of questoon-nsswers-pail
global caption strictly follows: "Global caption:
CAPTION _OR_____TT IMAGE. you
the provided information about and object' parts, create plausiole challenging question about the image, and provide the answer in detail for the image reasoning segmentation. The content of the question must address one of the following two:
generating
1l relationship between
image the relationship between a part an object.
21 the function or the general information about the parts.
The question should imolcit require commonsense reasoning, rather than explicily mentioning words, it's important to make the question challanging by not directly including visual content details. The answer should include multiple object's You must 3 rounds of natural queston-answer order
the object part. other
and If there sufficient information create
rounds of question- answers Queslion 1: Answer 1:.
answer pairs. addition. please follow strccly In the answer, the coordinates referring to the target part or object must be attached the objert name or part name the format: object [xl, yl. and object_ 1's sart
atlached the questions
y2] Do use other such "a part object Here are some additional requirements about
generated question and ansswers:
1. Do not mention that the information source is provided in text description. A Always answer as you are directll loaking at the image.
2. Do not ask the question are confident answer. Only include question that definite answer.
part and object directly 4. Make the questions and answers concise and easy to undersland, avoiding averly complex and ambiguous sentences.
mention coordinates
question.
5. The question should describe complete activit. function general information. generated queston should include
object's parts explicityy describe
part object.
Implying other polential parts is striclly prohibited.
7. ven the image includes the real people and the brand name, or is not associated with the mentioned information, make sure still create pairs.
question-answer
8. Aveid using incorreclly forma tted object names or part names, such as located at [coordinates] or part (object 1's part [coordinates]]. In other words, use as it appears in the abject part info mation given above. #** F example: shoe_1* It' outsole (42, 332, 62, 336), not an autsole
1's outsole
9. All generated answers must include the given object or part information, without changing the format."
Figure 6: The text and image prompt used in our data creation for MMR dataset with GPT-4V.
segmentation mask information for the objects or parts mentioned in the answers is sourced from PACO-LVIS (Ramanathan et al.) 2023 to create new annotations.
Part-only MMR Test Dataset The MMR dataset includes substantial amount of information on parts to enhance part-level recognition, which has been overlooked existing reasoning segmen- tation datasets. Consequently, we create a part-level test dataset to evaluate part-level recognition separately. Using the text and image prompts shown in Fig. 7 we generate a part-only test dataset from 2000 images with extensive part-level information from PACO-LVIS annotations. As shown in Fig. g the output includes global caption and question-ande answer pairs for the image. The segmen- tation mask information for the parts mentioned in the answers is sourced from the PACO-LVIS test dataset to create new annotations.
Object-only MMR Test Dataset To evaluate recognition separately for object-level, we create an MMR test dataset that includes only information on objects. We generate object-only test dataset using the text and image prompts shown in Fig. 9 selecting 2000 images with minimal pert-level information. As shown in Fig. 10 the output includes global caption and question-answe pairs for
17

---

Published as a conference paper at ICLR 2025
"You are an visual assistant capable of analyzing a single image. You receive the specific object's part locations withi the image, along with detailed coordiattes These coordiaates are in the form bounding boxes, represented as (xl, x2. y2). These values correspond to the top left wich
image you receive 428 and respectively Addiionally, there may be muttiple otjects of the same catagory in the image. To resoive this ambigulty. we use "object number such as "person_ 1 and "person_ _ difeerntiite between objects of the same celegery. I region pert coject, the category name described as "object's part". like "perzor''s wheel", category names bounding coordinates parts
top bolium right x, and batiom right y. The height
body"
6og. eye [235, 67.
d5g_ ear [324, 36, 426, 145]
60g nose (184, 96, 212, 127]:
dog_1's Ieeth [245, 146, 285, 171]
dog. 135
[169, 427, 202]
60g 135 (B37, 204, 510. 407]
dog_1's (212. 542, 356]:
dog_11 1% body (243, 503, 228):
bowl_1's rim [143, 298, 369, 378;
bowl 1's inner_ body [1.50, 302, 361, 371]
bowl bottom [194,
376);
bowl_1's body [153, 351, 354, 422]
You need create a global caption image without given information. A global caption should summarize content the image within maximum of two sentences. The format for the global caption str follows: "Global capton:::G GLOBAL, CAPTION FOR, THE IMAGE. What you need to do next create question-answer name appear in answers, "name [coordinates]" used the given information above without changing Its The geal of generating the queston-answers-pair is to use the provided information about object's parts, create plausible and chellenging question ebout the image, and provide the answer datail for the image reasoning segmentation. The contant the question must address one of the following two:
using information object's parts given above. However, when the conrespording object's part
relationshlo between different parts within image. 21 the function or the general information about the parts.
The question should be implict require commonsense reasoning, rather than explicily mentionnn words, important make questin challanging by not directly including visual content detais. answer should include multiple object's parts. You bulld at aeast 3 rounds of natural quéston-answer pairs, and if there sufficeett information create up to 5 rounds of question- pairs. In addition, please follow the formal striclly: The order must be allached to the questions and answers like Ques answer, the coordinates referiing to target part must attached other format such as "a part of object Here are some additinal requirements about generated question and answers:
object part
and Answer part name the format: object_ 1's part (x1, yt, x2, y2). Do not use
1. Do not mention that the information source is provided text descrition, Always answer as you are directly looking the image.
2. ask question you are confodndent answer. Only include question
definite answer.
3. Do not mention the courdinates of part and an object directly the question.
4. Make the questions and answers concise and easy to understand. avoiding overly complex and ambiguous sentences.
5. question should describe complete activity. function, or general information.
6. The answer to the generated question should include at least two object's parts and expliitly describe the names of the part. Implying other potentall parts striclly prohibited.
7. Even the image includes real people and the brand queston-answer pairs.
associsted with the mentioned information, make sure still create the
6. Avoid using incorrectly formatted part namee, such as located at (coordinatee) or part [object 1''s pert [coordinstee]. In other words, use at it the information given above. H#d example: shoe I's outsole [42. 332, 82, 336), not outsole [shoe_ 1''s cuisole [42, 332, 62,
A. generated answers must include the given part information, without changing the format.
10. When creating questrons, ask only questions about the object's parts given above without directly mentioning the part name the question.
Please keep
other parts 11. If the number of objoct's parts given for an image is large enough, create a question sa that each round's answer includes different abject's parts.
not dominate
answer.
12. Do not create queshans that are answered by parts other than the part information given above.
part doesn't directly
the question, montion the
Figure 7: The text and image prompt used in our data creation for the part-only MMR test dataset with GPT-4V.
18

---

Published as a conference paper at ICLR 2025
the image. The segmentation mask information for the objects mentioned in the answers is sourced from the PACO-LVIS test dataset to create new annotations.
boAt Eilocy
Caption: dog is sitting bowl, possibly etter water meel.
Question1: What features of the animal suggest that may have just had drink? dog__1 bowl 1''s rim [143, 298, 369, 378) next to the dog suggests that coule have contained water or food that dog might have consumed.
dog
presence
Question2: Considoring the animals current posson, which 356)
supporting
dog_ foot (337, 204, 407] Iikey bearing the weight. considering dog's sitting position adjacent to the bowl_1t 1'' body [153, 361, 364, 422]
Answer2: d's leg [212,
physical Answer3: The dog__e ee (324, 426, 145] would be most engeged detecting sounds as ears are responsible for auditory perceptioo in dlogs.
Question3: the
aperoaching
engeged?
Figure 8: An example from the part-only MMR test dataset generated through the prompt in Fig. 7 This example includes information of some object's parts. The left and right pictures show the original image and part-level segmentation masks, respectively.
A.6 DATA FORMAT
The MMR dataset is given in JSON format. The JSON file for each instance is organized as shown in Fig.
Table 6: The effect of multiple [SEG] Tokens and Early Local Feature Fusion in M²SA-7B on MMR benchmark. Obj & Part, Obj, and Part denote multi caltiigramularity, object-only, and part-only evaluation settings.
val
test
multiple [SRG] Tokens
Early Local Feature Fusion
Ohj
Part
Ohj
Part
Ohj
Part
gloU
cloU
gloU
cloU
gloU
cloU
gloU
cloU
19.4
31.6
34.7
41.8
8.0
13.1
19.5
27.1
26.0
47.7
39.5
55.4
11.7
25.2
28.4
45.2
27.9
48.5
41.0
55.6
13,5
27.0
31.0
46.8
A.7 EFFECTIVENESS OF THE MULTIPLE [SEG] TOKENS AND EARLY LOCAL FEATURE FUSION
We conduct an ablation study to verify the effectiveness of the multiple [SEG] tokens and Early Lo- cal Feature Fusion proposed in MaSA. Tab. 6 demonstrates that merely adding multiple [SEG] to- kens results significant performance improvements in MMR evaluation benchmarks. This finding suggests that using single [SEG] tokens in the LISA is inadequate to fully capture the segmentation capability. Moreover, performance improvements are evident when Early Local Feature Fusion is incorporated. Notably, there is substantial performance enhancement in the part-only evaluation setting the MMR test set. This improvement likely arises because Early Layer features contain local detail information (e.g., edges boundaries). which aids in part and fine-level segmentation.
19

---

Published as a conference paper at ICLR 2025
"You are an visual assistant capable of analyzing single image. You receive the specific objectlocation within the image, along with detalled coordinates. coordinates
bounding boxes, represented lxi, yl, y2). These values correspond the left y. bottom right x, and bottom right y. The height and width of the image you receive are 333 and 500, respectively. Additinally, there may be muttiple objects of the same category the image. resoive this ambigulty, we use "object number" such as "person_ 1 and "person_" 2" differentiate
top
objects the category. The category names bounding coordinates objects
mirror 1 [304.9, 35.42, 08,00,,11 146.49) pillow (169.86, 180.73. 221,21, 229.7]: pillo, (370.81, 175.9, 231
global caption for image maximum of two sentences. The farmet the global taption stricly follow:: 'Global caption: GLOBAL, CAPTION FOR, THE, IMAGE. What you need to do next is create question-anee mrestto-annswes-ppee nswers-pairs using the information of objects given above. However, when the corresponding object name goal of generating question-
nesd
given
global caption
the
image
appear enswers. enswers-pad pair use the provided information about objects, creale a pleusiale and chalenging question about the image, and providle the answer in detail for the image reasoning segmentation hee content question must address one of the following two: relationship 21 the function or the general informetion about abjects.
[coordinates]
given
changing
objects within the image.
The question should implicitn and require commonsense reasoning, rather than explicitly mentioning important to make the question challenging by not directly including visual content details. Some of the answers the rounds should include multiple different types of objects. You must build at least 3 rounds of natural question answer pairs, and if there suffic ient information create up rounds question-answer Question and Answer In answer. the coordinates refering to the object mustbe attached to the object name the foraa: object, Here are some additional requirements about generated question and answers:
of the object. other
addition,
follow the
strictly: The
attached the questions
like
[xl. yl, x2.
1. Do not mention that the information source provided text description. Always answer as you are directly looking the image. not confident to answer. Only include question that have definite answer. object directl in question.
2. Do not ask the question
not mention coordinates
4. Make the queslions and answers concise and easy ls understand, avaiding overly complex and ambiguous sentences.
5. T questinn should describe complete activity, function, general information. generated question objects striclly prohibited.
include at objects explicity describe
object. Implying potential
7. E ven if the image includes the real people and the brand name, or not associated with the mentioned information, make sure still create the question-answer pairs.
8. Avoid using incorrectly formatted object names, such as located [coordinates] or an object_ 1 [object [coordinates]] In other words, use it it aapperss in the abject information given above.
include the object information, without changing When creating questons, ask only questions about the objects given above without directly mentioning the object name the question. Please keep in mind that other objects should not dominate the answer. it the number of ohlante elven
generated
muection each minets
incivrias rifferenttchirrt:
Figure 9: The text and image prompt used in our data creation for the object-only MMR test dataset with GPT-4V.
20

---

Published as a conference paper at ICLR 2025
mirror
pillow
Caption: cozy living room interior with large mirror on wall and decoretive pillows furnturre.
Where might someone rest their head while sitting the furniture gein extra comfort? person might comfort while sitting on the furniture.
Question1:
pillow_ [169.88
piloww (370.81, 231.85] to gain
the could Answer2: Someone could check their appearance the mirro_11 [304.8, 8, 35 42.4.22. 146.49). which located on wall.
Question2:
appearance bafora leaving house?
Question3: a person to the decoreflive cushions, which items would they be handling?
person reerrange decorative cushions, [370.81, 175.9, 436.9.43...1, 231.85).
pillow 11189.86 21, 229.7| pillww
Figure 10: An example from the object-only MMR test dataset generated through the prompt in Fig. 9
This example includes information of objects. The left and right pictures show the original image and object-level segmentation masks, respectively.
data:
'file name': file name of the image. 'height': int, height of the image,
'width:: int, width the image. image_id"s id': int, of the image, 'not exhaustivee category jds': Listfint), list of category ids
that don't have all of their instances marked exhaustively,
'neg_cct hegg_ccttgory..oo idd': Listfint], of category ids were verified as not present in image, "coco, url't str, image URL, questions': List(str], the complex and implicit questions about the objects and parts in an image. annotations':
'bbox': Listifloat), bounding box of the object segmentation':
size': List[int], the size of the image,
counts:: RLE format, segmentation binary mask information,
iimaee id': id of the image, category name': str, category_name _ame of the object part. category id': int category id.
"orree_.cate "sorte_ccttegory.ie category_idd id': int, sorted ascending order.
aasswesss List[dicts], the annotations correspanding to the questions, 'text. answers": List[str], the text answers the questions, "raw answers':: List[str], the answers from API the questions,
Figure 11: MMR dataset format
21

---

Published as a conference paper at ICLR 2025
Table 7: Comparison of computational complexity on LISA, GSVA, and GLaMM, and M2SA.
Methods
GPU Memory Usage (GB)
TFLOPs
LISA-7B
Lai et al.
2023
30.58
32.59
GSVA- 7
Xia et al.
2024
30.39
203.77
GLaMM Rasheed et al. 2024)
17.14
349.28
M'SA-7B
30.60
32.62
LISA-Llama2-133 (Lai et al.
2023)
55.20
56.64
M-SA-Llama2-133
55.23
56.67
A.8
TRAINING TIME
The training takes approximately 40 hours for the M2SA-7B and about 52 hours for the M Llama2-13B, respectively.
A.9 COMPUTATIONAL COMPLEXITY
We aim to compare the computational complexity of the proposed M with LISA, GSVA, and GLaMM. For this comparison, we measure GPU memory usage and TFLOPs. As shown Tab. 7 while the addition of Early Local Feature Fusion and multiple [SEG] tokens leads to slight increase GPU memory usage and TFLOPs, M demonstrates a significant improvement in handling multiple targets and fine-grained grained part-level segmentation compared to LISA. However, despite these performance improvements, there is still room for enhancement from the perspective of computational efficiency. Since S is built upon both MILM and SAM, it requires substantial memory resources. Future research could focus on optimizing the efficiency of the mask decoder, which predicts the final mask by integrating vision and language information.
Table 8: Multi-object referring segmentation results on GTAV and Cityscapes validation sets. We adopt mloU metrie for comparison. We evaluate the zero-shot performance of LISA, GSVA, GLaMM, and MSS. The best results are highlighted bold.
Methods
GTAV-val
Cityscapes-val
LISA-7B Lai et al. 2023
3.7
6.1
GSVA-7B Xia et al. 2024
15.7
14.6
GLaMM
Rasheed et al. 2024)
12.6
12.6
M-SA-7B
35.1
41.3
LISA-Llama2-13B (Lai et al. M'SA--Lamaa2-138
2023)
2.4
3.4
38.2
44.0
A.10 GENERALIZATION ON UNSEEN DATA
To assess S'' generalization unseen data, conduct additional experiments. Although OV- PARTS (Wei et al. 2024) was recently proposed for open-vocabulary part-level segmentation using Pascal-Part (Chen et al. 2014) and ADE20K (Zhou et al. 2019), both datasets were used during training. Therefore, we evaluate the model's generalization performance using semantic segmenta- tion datasets from driving scenes, specifically Cityscapes (Cordts al. (016) and GTAV (Richter) et al. 2016), which were not used during training and pose more challenging test environment. Since these datasets lack pert-level mask annotations, we focus on evaluating multi-target object cases. Furthermore, we curate custom text prompts using predefined category names as they do not provide corresponding text queries. For each query, we randomly select 4 to object categories from an image and create prompts such as "Can you segment the class 1, class 2, ..., and class n?". The model generates masks for the specified objects, and we compute the mloU score to compare its performance with LISA. As shown in Tab. 8 M2SA performs robustly entirely different domains. Notably, while the existing methods struggle with multi-target cases, M handles them effectively. This demonstrates that the use of multiple [SEG] tokens, combined
datasets from
22

---

Published as a conference paper at ICLR 2025
Table 9:
Comparison between LISA-7B (Lai et al.
trained MMR dataset and LISA-7B We measure the performance on ReasonSeg validation set
2023
trained on ReasonSeg (Lai et al. 2023).
Methods
gloU
cloU
LISA-7B w/ ReasonSeg
44.4
46.0
LISA-7B w/ MMR
49.9
55.6
with early local feature fusion, enables M2SA to generalize well to unseen domains by improving its ability manage multi-target cases and fine-grained segmentation tasks.
A.11 MMR AND REASONSEG
To validate the comprehensiveness and effectiveness of the MMR dataset, conduct compara- tive evaluation with ReasonSeg using the LISA-7B model. Specifically, we train the model two configurations:
using ReasonSeg and the other using MMR instead of ReasonSeg. As shown in Tab. / the model trained on MMR shows superior performance on the ReasonSeg validation set than the model trained ReasonSeg. This improvement highlights the comprehensiveness of the MMR dataset. By incorporating multi-target and part-level annotations alongside object-level data, MMR provides more robust knowledge for addressing complex reasoning segmentation tasks.
Table 10: Performance of M2SA on frequently appearing and infrequently appearing object cate- gories. From the total of 75 categories, question-answe answer pairs containing the top 10 most frequent (upper) and bottom 10 least frequent (lower) categories are extracted construct the upper and lower subsets, respectively.
MMR test
Methods
Obj-only (total)
Obj-only (upper)
Obj-only (lower)
gloU
cloU
gloU
cloU
gloU
cloU
M²SA-7B
41.0
55.6
41.0
54.8
39.4
39.7
Table 11: Performance of M2SA on frequently appearing and infrequently appearing part categories. From the total of 445 categories, question-answe answer pairs containing the top 10 most frequent (upper) and bottom 10 least frequent (lower) categories are extracted to construct the upper and lower sub- sets, respectively.
MMR test
Methods
Part-only (total)
Part-only (upper)
Part-only (lower)
gloU
cloU
gloU
cloU
gloU
cloU
M-SA-7B
13.5
27.0
12.8
24.8
13.3
28.1
A.12 ANALYSIS OF THE LONG-TAIL PHENOMENON IN M2SA
To investigate whether M trained on the MMR dataset exhibits a long-tail phenomenon, we evaluate its performance on frequently and infrequently occurring object and part categories. To this end, we construct subsets of the MMR test set by isolating question-ansee answer pairs based on category frequency. Specifically, we extract the top 10 most frequent (upper) and bottom 10 least frequent (lower) categories for both object-only and part-only test sets. This results in four subsets: object-only (upper: 10/75), object-only (lower: 10/75), part-only (upper: 10/445), and part-only (lower: 10/445). The MMR dataset includes a total of 75 object categories and 445 part categories, respectively. The performance comparison is shown in Tab. and Tab. 11
For the object-only dataset, M2SA's performance on frequently occurring (upper) object categories closely aligns with its overall performance across all object categories (gloU: 41.0, cloU: 54.8 gloU: 41.0, cloU: 55.6). However, for infrequent object categories (lower), the performance de- clines, with cloU dropping from 55.6 to 39.7 and gloU from 41.0 to 39.4. In contrast, for the
23

---

Published as a conference paper at ICLR 2025
part-only dataset, M demonstrates consistent performance across both frequent and infrequent categories. The gloU scores are 12.8 (upper), 13.3 (lower), and 13.5 (overall), while the cloU scores 24.8 (upper), 28.1 (lower), and 27.0 (overall). These findings suggest that M2SA is less sensitive to the long-tail distribution in part categories than in object categories.
This analysis highlights the strengths and limitations of M when addressing long-tail distribu- tions. While M demonstrates robust performance across frequent and infrequent part categories, its reduced performance infrequent object categories indicates potential areas for improvement. Future work could explore strategies to mitigate the impact of long-tail distributions in object cate- gories while preserving its strengths in part-level reasoning tasks.
A.13 QUALITATIVE RESULTS
Qualitative results of M on the MMR benchmark are visualized in Fig.
12
Fig.
13 and Fig.
14
Image
Output
Question: someone wants to send an e-mail, which equipments on the desk would they be utilizing?
Answer:
They would be utilizing the laptop_ for typing and viewing the screen, and the mouse for navigating and interacting with the computer.
Figure 12: Qualitative result of M2SA on MMR test set.
24

---

Published as a conference paper at ICLR 2025
Image
Output
HR
AR
Princeton Junction
Princeton Junction
Windeer
Windoer
Newwrrk
Nowarr
eeach
reach
13420
CT 13
.AI.com
LAir.com
Question: Where could someone sit while waiting for transportation, and which part provides support for their back?
Answer: The bench's seat provides a place to sit, and the bench's back offers support for the back
Figure 13: Qualitative result of M on MMR test
Image
Output
Question: If need to check how much time is left before my meal is ready, which part of this appliance should look at?
Answer: You should look at the microwave_oven's time lime_dipplay to check the remaining time.
Figure 14: Qualitative result of M on MMR test
25

---

Published as a conference paper at ICLR 2025
A.14 ADDITIONAL EXAMPLES OF MMR
To facilitate a quick and intuitive understanding of the MMR dataset's characteristics, we present additional examples in Figure 15
Cibhaiiption is tumpuis opeend and ore no tade to non www. menere uggn aggrting hypiccl aorlegoo
Annantt They woll monvrlyy invarr with fpiine urea der beginind 20% 30g tor bygine com porspaatt (40, 291, torolggee the dinurewe.
Oreatond::
manaleturerss brer Annwncc mocestor une'c corting lou bigtng compuder legaa 299) moase jroecutte darneert
pictaned
somen touching lastos, Annwndd use stnare coomuite growe) booe (ivi, akong with unause jesepute ersppaneed SRE
Greatonn:: To move
campseer cuspentt sould
jonepuater
Gu entonis aher insiiing work deccing pack wiich parts of luptes sould coma cntorc
Clobal Captions
thi
you
with
and Is handle 150, 87, 200) srovide contorabie for ing.
Queetiasd worred
mainvdiia cuporaaily
attertinn
Answert: To conta in main dish und pupar separaaally cou use glan 125,26, 24 Got (oU0).oo which has EIRAL''S snar_welll (35, 75 033 hal food supprats plane
prevents burm
Libbus,
plats 20 amau
2000 plata_ ei 2012 ancher portin.
Oovestand:- the blourn ahould one exctioue andi sptla whill surning
plae
Inn 28.66, 10s 200020 nwrid spil shine purs wnt ineeaa the fond with how of piase
LEP
Glpbal dog werring reciin pllie:
Deesteed: ahhore would most dedy sou how would cupreco dertrmao curalay!
Arowerks urenal wos ki back resche scatts cang tng nas EDP 321, 2411 and expens expeni dostenn cote by piiing the of IS
Daesteeth Sarvce elsoriee the urea that uppers the whe its dower foot 253,
the allssi 15.2 260 022.7 427.0.
Deestendh when thes The busures mneived ris head 1155 90, 2803
envromment, when losiing
festurn
(00 and mopeemee
from its partion, Answank To up th niss moria po postton, the dae would inbtally angage Eeg. (176, 386, (87) 199 Wh EV (nd)
Gaenlond: the
Figure 15: Additional Examples of MMR dataset.
26


Reviews:
[{"review_id": "qnttYfPhfU", "reviewer": "Reviewer_WfTF", "summary": "This paper provides a large multi-target and multi-granularity reasoning segmentation benchmark. Based on this benchmark, this paper designs a baseline model trained on it while evaluating public datasets to present the effectiveness of both the benchmarks and the baseline. Experiments demonstrate that the proposed baseline outperforms LISA and other representative approaches.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The distinguishing characteristic of the proposed benchmark is clear, which includes multi-granularity and more images.\n2. Multi-target and multi-granularity reasoning segmentation is a valuable research topic.\n3. The overall writing is fluent.", "weaknesses": "1. This paper provides few comparisons on the proposed benchmark. It is not clear whether the proposed baseline model outperforms other MLLMs on multi-target and multi-granularity reasoning segmentation.\n2. The major contribution lies in the benchmark, while this benchmark is auto-annotated based on the existing dataset PACO-LVIS, which hurts the contribution.\n3. According to Table 1, MMR offers both object-level and multi-target annotations, making it more comprehensive than ReasonSeg and MUSE. This paper could include zero-shot evaluations on these two benchmarks to further demonstrate effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n7uxF70aoV", "reviewer": "Reviewer_JQ5y", "summary": "This paper introduces a dataset named MMR, designed for multi-target and multi-granularity reasoning segmentation tasks. The goal is to address challenges in reasoning across multiple targets and different levels of granularity. The dataset comprises complex and implicit question pairs, covering both object-level and part-level reasoning. Additionally, the paper proposes a baseline model, M2SA, to achieve multi-target, object-level, and part-level reasoning segmentation.", "questions": "1. Will the proposed dataset be made publicly available?\n2. The paper mentions the use of 4 A6000 GPUs. How long does it take to train the proposed model on the MMR dataset?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Clear Writing**: The paper is well-organized and easy to understand.\n\n2. **Significant Contribution of the Dataset**: The MMR dataset contains 196K samples. Although it was generated using large models, a rigorous filtering process was employed to ensure data quality.", "weaknesses": "1. **Lack of Targeted Design in the Baseline Model**: The baseline model (Early Local Feature Fusion and Multi [SEG] Tokens) does not incorporate specific structures to effectively address the multi-target and part-level reasoning required by the MMR dataset. As a result, it lacks novelty, leading to underwhelming performance in Table 3.\n\n2. **Limited Performance in Table 3**: The comparison methods in Table 3 are not sufficiently recent. The authors did not include comparisons with more relevant multi-target approaches, such as GSVA [1] or GLaMM [2]. This limits the impact of the proposed approach, as the results are not very competitive.\n\n3. **Insufficient Comparisons in Table 2**: The methods compared in Table 2 are too limited. I strongly recommend including more methods that could be adapted for the MMR task to facilitate meaningful comparisons for future research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8HnYmggodD", "reviewer": "Reviewer_D24r", "summary": "This paper introduces a novel dataset, MMR, the first part-level dataset for the reasoning segmentation tasks. In addition, a new network framework is proposed to leverage low-level fine-grained information and to address the limitation of the existing LISA model, which can only segment a single object. The authors conduct experiments to evaluate the performance of existing methods on the proposed MMR dataset and demonstrate the advantages of the proposed network framework.", "questions": "1. Why did you remove the generated questions that contain explicit target coordinates or strong hints? I think training with such data would enhance the model’s ability to handle target-specific inputs. For example, if an image contains two different animals, a fish and a cat, users could indicate the coordinates of the animal they are interested in and ask, “Which part of this animal [coordinates] uses its sense of smell?” The model could then segment either the nose or the fish’s gills depending on the coordinates provided. This could be quite interesting.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well written and easy to follow. \n2. The proposed MMR dataset is highly valuable to the research community, as part-level reasoning segmentation is crucial in real-world applications, such as robotic control. However, there is currently a lack of available datasets for research in this area.\n3. A detailed analysis is provided to thoroughly present the characteristics of the MMR dataset.", "weaknesses": "1. The contributions of the proposed M2SA network framework are incremental. The early local feature fusion appears to be only a minor structural modification. Additionally, the strategy of employing multiple [SEG] tokens has already been introduced in earlier methods, such as [a]. The authors should clarify the differences between their approach and [a].\n2. This paper could benefit from more thorough experiments based on the characteristics of the dataset. For instance, does the M2SA trained on the MMR dataset show a noticeable long-tail phenomenon, i.e., better performance on the more frequently occurring object and part categories as presented in Fig.3? Additionally, what is the model’s open-vocabulary performance on categories that do not appear in the training set?\n3. More examples of the image-question-answer triplet in the MMR dataset could be presented in the paper to enable readers to understand the characteristics of the dataset more quickly and intuitively.\n\n[a] GSVA: Generalized Segmentation via Multimodal Large Language Models, CVPR2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)