SegLLM: Multi-round Reasoning Segmentation with Large Language Models
Authors: XuDong Wang
Shaolun Zhang
Shufan Li
Kehan Li
Konstantinos Kallidromitis
Yusuke Kato
Kazuki Kozuka
Trevor Darrell
Venue: ICLR 2025 Poster (2025)
Keywords: LLMs
Reasoning Segmentation
Muiti-round Conversations
Primary area: foundation or frontier models, including LLMs

Abstract: We present SegLLM, a novel multi-round interactive reasoning segmentation model that enhances LLM-based segmentation by exploiting conversational memory of both visual and textual outputs. By leveraging a mask-aware multimodal LLM, SegLLM re-integrates previous segmentation results into its input stream, enabling it to reason about complex user intentions and segment objects in relation to previously identified entities, including positional, interactional, and hierarchical relationships, across multiple interactions. This capability allows SegLLM to respond to visual and text queries in a chat-like manner. Evaluated on the newly curated MRSeg benchmark, SegLLM outperforms existing methods in multi- round interactive reasoning segmentation by over 20%. Additionally, we observed that training on multi-round reasoning segmentation data enhances performance on standard single-round referring segmentation and localization tasks, resulting in a 5.5% increase in cIoU for referring expression segmentation and a 4.5% improvement in Acc@0.5 for referring expression localization.

Published as a conference paper at ICLR 2025
SEGLLM: MULTI-ROUND REASONING SEGMENTATION WITH LARGE LANGUAGE MODELS
XuDong Wang-1
Shaolun Zhang "1
Shufan Li
Konstantinos Kallidromitis
Kehan Li
Yusuke Kato
Kazuki Kozuka²
Trevor Darrell
UC Berkeley
UCLA
Panasonic AI Research
Stanforrd
ABSTRACT
We present SegLLM, a novel multi-round interactive reasoning segmentation model that enhances LLM-based segmentation by exploiting conversational memory of both visual and textual outputs. By leveraging a mask-aware multimodal LLM, SegLLM re-integrates previous segmentation results into its input stream, enabling it to reason about complex user intentions and segment objects in relation previously identified entities, including positional, interactional, and hierarchical relationships, across multiple interactions. This capability allows SegLIMM respond to visual and text queries in a chat-like manner. Evaluated on the newly curated MRSeg benchmark, SegLLM outperforms existing methods in multi- round interactive reasoning segmentation by over 20%. Additionally, we observed that training on multi-rount reasoning segmentation data enhances performance on standard ringle-round referring segmentation and localization tasks, resulting in 5.5% increase in cloU for referring expression segmentation and a 4.5% improvement in Acc@0.5 for referring expression localization.
INTRODUCTION
Image segmentation plays a crucial role in numerous computer vision tasks, while traditional methods have been limited to providing segmentation results for close-set categories (Cheng et al., 2022; He et al., 2017) or simple text queries (Ding et al., 2023; Wang et al., 2024b) using CLIP (Ding al., 2023; Radford et al., 2021) or BERT (Wang et al., 2024b; Devlin et al., 2018) text embeddings as classifiers. Recent advancements in Large Vision-Languaa Zhang al., 2023a; Lai et al., 2024; Wu al., 2024; Liu et al., 2024; Touvron al., 2023; Alayrac et al., 2022; Awadalla et al., 2023; Dai et al., 2024) have reformulated image segmentation as a next token prediction task, enabling segmentation models to engage natural language conversations with users and reason about the presence, location, and relationships of objects in complex visual scenes. For instance, LISA (Lai et al., 2024), a Language Instructed Segmentation Assistant, produces segmentation masks by incorporating [SEG] token into its vocabulary, which, when generated, is decoded into the corresponding segmentation mask.
Models (LVMs) (Pi et al., 2023a;
These L.M segmentation models (Lai et al., 2024; Wu et al., 2024; Pi et al., 2023a; Zhang et al., 2023a) typically achieve their localization capabilities by incorporating decoder that converts the output [SEG] tokens of LLMs into localization results. They are trained on numerous visual queries such as "please find the heart healthy food in the image", where responses include both text outputs and segmentation masks. Essentially, these models are advanced versions of early open-vocabulary segmentation models, with their text encoders upgraded from smaller language models, such as BERT (Devlin et al., 2018), smarter LLMs, such as Llama (Touvron et al., 2023). Consequently, LLM segmentation models are often evaluated on traditional referring expression segmentation (RES) datasets, such RefCOCO, which provide single text query corresponding to each mask. These singlc-round referring expression segmentation (RES) datasets overlook one of the most remarkable properties of LLMs (Achiam al., 2023: Team et al., 2023; Touvron al., 2023; Jiang ct al., 2023): generating multi-round responses in conversational manner. In this paper, we intend to answer the question: can segmentation models reason about previously segmented objects and conversations, responding to multiple visual and text queries in a chat-like manner?
Current LLM segmentation or detection models (Lai al., 2024; Zhang et al., 2023a; Wu al., 2024), despite their impressive single-round performance, fall short as multi-modal conversation
"Equal Contribution

---

Published as a conference paper at ICLR 2025
Referring Queries
Hierarchical Queries
85
01.
012
Round Segment person wealing glasses,
Round Plcase segment
81
60.2
LMgLLM:
SegLLM: Sure,
Referring Segmentation cloU (96)
77
73
69
65
Interactional Querles Round Segment the person to his right SegLLM: M:: Sure. [3]
Relational Queries
Round 4: Segment the bag
that she (3) is holding.
52
SegLLM: Sure, (4]
SegLEM
LISA
# rounds
Figure 1: We present SegLLM, multi-round interactive reasoning segmentation model designed engage chat-like interactions by responding both visual and text queries. reasons about previously segmented objects and conversations to understand complex user intentions. relationships between objects, such as positional, interactional, and hierarchical connections with previously identified entities, e.g., instance [1]. On the right: We introduce the MRSeg, new multi-round image referring segmentation benchmark. As the rounds progress, the complexity of interaction and memory retention increases. However, SegLLM consistently surpasses the previous SoTA method LISA (Lai al., 2024), with a significant margin across all conversational rounds.
left: SegLLM can infer intricate
agents due to their inability handle multi-round, interactive conversations. For instance, after obtaining mask of 'person in black hoodie' in Fig. 1, a user might want to perform additional queries based on this mask output-such as segmenting the 'ski he holding', segmenting the 'man standing to the right of him', or segmenting a different person if the output is incorrect. Existing models struggle with these complex queries because there no "communication" between the large language models (LLMs) and the vision encoders. Information flows only from the LLMs to the mask decoder, not vice versa, preventing the L.MM from being aware of the output mask and making it difficult to reason about complex queries involving previous mask outputs.
To address this issue, we propose SegLLM. Unlike existing LLM segmentation models that naively assemble a mask decoder with an LLM, we introduce novel communication protocol that feeds segmentation outputs of the mask decoder back into the input stream the LLMs, and past conversation context into the input query of the mask decoder. This design allows the LLMs to "see" past mask outputs and the mask decoder to "see" the past conversation context, enabling it to handle complex queries like 'segment the helmet of the previously segmented person', shown in Fig. 1. Concretely, we introduce a Mask-Encodee Reference Mask-Decoding scheme to make the segmentation head context-aware. To fully explore the capabilities of these novel designs, we curated multiple high-quality multi-round interactive segmentation datasets, named MRSeg. The new dataset consists of complex object queries involving existing mask outputs, formulated in seamless multi-round natural language conversations.
scheme to make the LLM mask-aware and a
Through extensive experiments, we demonstrate that SegLLM outperforms previous state-of-the-as models by 18~30% on our multi-round reasoning segmentation benchmarks, MRSeg. Additionally, SegLLM surpasses prior state-of-the-ae performance on the single-round referring segmentation and detection benchmark, RefCOCO, with over 5.5% improvement in segmentation (cloU) and 4.5% increase in detection accuracy (Acc@0.5), SegLLM also exhibits greater robustness to various question templates, achieving 9.6% performance gains on RefCOCO with diverse query formats.
2 RELATED WORKS
2.1 MULTI-MODAL. LARGE LANGUAGE MODELS
To leverage the advancements in language models (Brown et al., 2020; Touvron et al., 2023; Chowdh- ery al., 2023; Le Scao al., 2023; Hoffmann al., 2022) across various modalities, Multi-modal
2

---

Published as a conference paper at ICLR 2025
Large Language Models (MLLMs) have been developed to combine language and vision (Yin et al., 2023; Liu et al., 2024; Zhu et al., 2023; Alayrac et al., 2022). Flamingo was one of the first unified architectures to align image and text pairs in context learning through gated cross-attention blocks (Alayrac et al., 2022). End-to-end MI.LMs typically require a finetuning process where an intermedi- network (l.i et al., 2024; Zhang et al., 2023a) and/or sampler module (You et al., 2023) is used to map the vision features into the language space. BLIP-2 bridges the modality gap with a querying transformer and a two-stage training process. which involves pretraining on a trainable LLM and instruction tuning on a frozen one (Li et al., 2023b). Models like MiniGPT-4 (Zhu et al., 2023) and LLava (Liu et al., 2024) follow similar training paradigm, with Vicuna 18 as a language decoder and GPT-4 designed prompts. Other notable models in instruction tuning include Otter (Li et al., 2023a) that is based on (Awadalla et al., 2023), mPLUG-Owl (Ye et al., 2023) with novel modular architecture, and InstructBIIP (Dai et al., 2024) which features an instruction aware Q-former.
2.2 MULTI-ROUND CONVERSATIONAL MILMS
Recent advancements in MLLMs have focused on enhancing interactive capabilities, Models like Kosmos-2 (Peng et al., 2023) and Shikra (Chen et al., 2023) use visual grounding and referring to provide the L.M with detailed location information of the objects, which enables the user to point out specific areas in the image. Various works aim to improve local information, such as Ferret (You et al., 2023) and PerceptionGPP (Pi et al., 2023b) which employ flexible continuous representations to handle different shapes. Other approaches (Yang et al., 2023a;b; Zeng et al., 2022) utilize prompt engineering and APIs to facilitate interaction, instead of relying on end-to-end models.
More recent approaches introduce the concept of reasoning, leveraging LLMs to provide a visual answer based on implied information. DetGPT (Pi et al., 2023a) performs object detection using high-level instructions rather than distinct classes. GPT4Rol (Zhang et al., 2023b) receives spatial boxes input to focus on specific regions and better align vision and text. LISA (Lai et al., 2024) adds a new embedding prompt to the mask decoder of the SAM (Kirillov et al., 2023) guiding segmentation, which is then processed by LLaVA (Liu al., 2024) to perform high-level reasoning. NExT-Chat (Zhang et al., 2023a) expands on LISA by using embeddings instead of tokens for location information and adding a decoder with a joint loss to facilitate object detection.
While some methods support multi-round conversations, they often lack mechanisms to maintain lo- calization performance over successive rounds. leading to degradation and information loss. SegLLM improves the multi-round interactive segmentation by leveraging the text and segmentation results from previous rounds, thereby generating reñned masks and supporting hierarchical representations to enhance performance in multi-round interactions.
BACKGROUND: REASONING SEGMENTATION
Task definition. The reasoning segmentation task (Lai et al., 2024) involves generating binary segmentation masks based on an image and descriptive, free-form text prompts. This task requires the model possess cross-modality comprehension, understanding both the complex visual scenes, as well as the natural- -aauuag- signals in the text prompt. Specifically, the model must interpret complex user text prompts that go beyond simple class names to include implicit descriptions that require general world knowledge, such as "the device that can illuminate dark room"..
Overall pipeline. To achieve such capabilities, reasoning segmentation model typically first employs pre-traaiee treittr large multimodal models (VLMs), FMM, which is capable of comprehending both visual and textual information simultaneously (Lai et al., 2024). new [SEG] token is then added to the VLMs's 's vocabulary. Given an input image Timg and input text prompt Tix, the VL.Ms generates an output text response Ihs,, which includes the [SEG] token to request the generation for a segmentation mask. Finally, the segmentor FSEG uses the last layer's 's hidden state, hegg corresponding to the [SEG] token along with the input image Timg to generate the segmentation mask JSBG.
Model architecture. An image reasoning segmentation model, FMM. typically consists of three key components (Lai et al., 2024): an image encoder εMM (e.g., CLIP (Radford et al., 2021) and DINOv2 (Oquab et al., 2023)), a base language model L (e.g.. Llama (Touvron et al., 2023)), and vision-to- laaiou-t- projection layer fyast., which is typically an MLP layer. Given a pair of input image and text prompt (Zimg, P, the image encoder first encodes the input image into patch embeddings
3

---

Published as a conference paper at ICLR 2025
1. Select Instances and Relationships
2. Fit Annotations to Templates
3. Refine Conversational Data with LLM (optional)
Dataset has relationahip annotations (Vausl Genome, PACO, Pascal)
serected instencee and relationsiis appropriate templates
Ask an LLM to refon Fine comvorsttions ensurin correct grammer
RegllM:: "Can you angaent the child shich zita sn the proviously segrented abjectt
Pandiaily select soae
relstionships
coresponding
Dataset does have retationship anmotations (Ru/COCO(+sg) LVIS)
Ueec:
Ita looking for the auganttatios naak of the Eouch an the image.
Rancomty solect some number Instances
Muti mund
SugtaM:: Daze. the seprentation reants 1n [1]
interactiv
conversation
tawe: "Cas you augoect the Ehild stich aiia on the powviously segeented object?"
Muli-ound convenattios
ongnent previsuaty cnl objocet"
SegiiM:
Caloulate spatial milioonntip betweeen the solectid instances
SeytaM: Bore. 123.
Figure 2: Pipeline for generating our multi-round conversational dataset MRSeg. The workflow involves selecting instances, generating relationships, fitting the instances and relationships into conversational templates. and refining the conversations using a language model for improved accuracy.
him.. which are then projected into the text embedding space via f The resulting visual tokens are concatenated with the sequence of text tokens hxs- Finally, taking both visual and language tokens as inputs, the language model L produces the output response Us containing the [SEG] token: Ai = F Trxt) [(cat([fva. (EMM (Timg)) hmx))). The [SEG] token in the output responses is then decoded into the segmentation mask using the mask decoder FSEG of a pre-trained segmentation model, SAM (Kirillov et al., 2023): ÚSEG FsEc (Timg, haEa) DsEu (Esua (Fmmg) hsgg).
4 MULTI-ROUND REASONING SEGMENTATION
The success of our SegLLM method relies on two essential components: comprehensive dataset MRSeg that has an extensive collection of Multi-Round interactive Segmentation instructions, and a mask-aware VLMs specifically designed to reason about the conversational history, with particular focus on the segmentation masks generated in previous interactions.
4.1 DATA PIPELINE
Data sources. We constructed multi-round image reasoning segmentation dataset (MRSeg) based on several widely utilized datasets, and include data from the following sources: RefCOCO(+/g) (Yu et al., 2016; Kazemzadeh et al., 2014), Visual Genome (Krishna et al., 2017), PACO-LVIS (Ra- manathan et al., 2023), LVIS (Gupta et al., 2019), Pascal Panoptic Part (de Geus et al., 2021), ADE20K(Zhou et al., 2017), COCO-Stuff(Cassar et al., 2016) and MSCOCO(Lin et al., 2014b). We used bounding box or segmentation annotations from these datasets to generate natural language conversations, applying a template-based approach as detailed in subsequent sections. The overall pipeline can be seen in Fig. 2 and we provide the statistics and some sample data for MRSeg in Fig. 3.
Multi-round conversation generation. We design various pipelines for generating multi-round conversations, tailored to the types of data and inter-instance relationships they support:
Hierarchical Relationships (PACO-LVIS, Pascal Panoptic Part): In these queries, the model is tasked with segmenting objects that are sub-parts of previously segmented instances. The queries start by asking about the instance, followed by questions about its parts. Example query: "Can you segment the <part> of the <object>?" ?"
Positional Relationships (RefCOCO(+gg), LVIS): These queries require the model to segment objects based on their positional relationships to previous outputs. An example query is: "Can you segment the <cclas>> that is <relationship> the output from round <i> ?" We refine these conversations using GPT-4 (our full prompt to GPT-4 can be found in Table A3) to ensure natural language fluency. Details on the RefCOCO(+/g) pipeline are in Fig. A2. Additionally, introduce challenging variant called MRSeg (hard), where understanding previous round information is necessary to correctly segment the current instance (details in Appendix A.1).
Interactional Relationships (Visual Genome): Utilizing Visual Genome (VG) relationship annotations, construct conversations that focus on interactional dynamics, rather than just.
4

---

Published as a conference paper at ICLR 2025
Number of Images per Dataset (Training)
Semantic Segmentation
Hierarchical Segmentation (PACO-LVIS, Pascal Part)
(ADE20K, COCO-Stuff)
MSCOCO
94260
ADE20K
Bottie
Label (1) Translucent body [I]
Pascal
Interactional Segmentation (Visual Genome)
RessonSeg
20190
Round bail
Bay that is haiding Bt
Positional Relationships (Refcoco+/g. LVIS)
Buskding
Tallaed gisfle on The
giatt right rint
Citafe facing the appostt directin to the right MI
Third gialfs from the left, seme su [
Figure 3: Statistics and sample conversations for the Multi-Round Referring Segmentation dataset (MRSeg). We provide more details for MRSeg Appendix A.1.
positional relationships. Each conversation has two rounds: the first round segments the subject, and the second round segments an object based on its relationship to the subject.
Attribate-oriented Queries (MSCOCO): These queries ask the model to segment objects based on their attributes usage rather than class names. An example query is: Q: Outline and extract the object that has a tall, slender neck covered with a distinct pattern of patches. A: Yes, the figure you specified for segmentation is giraffe. We generate captions by cropping MSCOCO instances and using GPT-4V prompts (details in Table A2).
Single-Round Semantic Segmentation is based on ADE20K and COCO-Stuff datasets. We construct single-round conversations by fitting class labels into various query templates.
Additional details the multi-round data pre-processing for MRSeg are provided in Appendix A.I.
Conversation templates. We observed that current state-of-the-art chat-based image segmentation models, such as LISA (Lai et al., 2024), tend to rely heavily fixed of question templates. This leads to fluctuations and instability in segmentation quality when user prompts are phrased differently, suggesting potential overfitting to specific language prompts. To address this, we leveraged the web- version of GPT-4 (Achiam et al., 2023) to generate diverse templates, creating more natural language conversations from dataset annotations. We generated templates for direct referring segmentation queries, relational queries, and hierarchical queries. For each query type, created 100~2000 templates for training and 50~100 different templates for validation.
4.2 SEGLLM FOR MULTI-ROUND IMAGE REASONING SEGMENTATION
Overall Pipeline. We introduce SegLLM to ensure that the VLMs's next token predictions incorporate the conversational memory from previous interactions, including the visual outputs, ie., segmented masks, and the text conversations. The architecture of model is illustrated in Fig. 4. SegLLM consists of two key components: 1) Mask-Encoding Module: This module feeds the output masks back into the input stream of the LLM, enabling it to reason about segmented masks from previous rounds. 2) Mask-Award Decoding Module: This module allows the mask decoder generate new masks based both the visual and textual conversational history, enhancing its contextual understanding. For example, when user requests segmentation of a part of an object identified in previous round (e.g., the ear of man). the model's ability to access prior mask data enables the decoder to more precisely localize and segment the specified object.
Mask-Encode Bask-Encoding Scheme. For each mask generated by the decoder, we compute two types of embeddings: mask embedding and bounding box embedding. The mask embedding captures the semantic information the masked object, and the bounding box embedding captures its location within the original image. Employing mask embeddings (one token per mask) and box tokens instead of new set of patch tokens for each mask substantially reduces the number visual tokens required for multi-round conversations. Furthermore, since the visual patch tokens for the entire image have already been utilized in previous conversations, this design does not compromise the richness of information necessary for accurate image segmentation and visual question answering.
5

---

Published as a conference paper at ICLR 2025
Conversation History
Bruod:: horse
Mask Decoder
jieri
Image
MMG EMEED
PPA
Soundl Seiinant portenttuua the horow thet right
Decoded Reference Mask
Mask +
VARKEVIRD
(RER]
Crasped Image
LLMS
[S80]
Fitos Cooncrrato y1,
Box Encoder
BON eVEEO
Round riding the ebessi (1]
TPSTIPARRO
Bounsid could you segiment top object (2]?
Decoded Target Mask
Figure Model architecture SegLLM for multi-round interactive image reasoning segmentation, which can understand complex user intentions and segment entities based on their relationships with previously identified ones. To facilitate this, first, we implement a mask encoding scheme that reincorporates the reference mask information back into the input stream of the LLMs. This enables the LLMs reason about segmented masks from previous rounds. Second, we develop mask-aware decoding scheme that allows the mask decoder generate new masks based on both the output from the LLMs the historical memory of output masks. The model uses the last layer hidden states associated with the [REF] and [SEG] tokens to generate both the reference mask and the target mask, seamlessly integrating past and current segmentation results.
To obtain the mask embedding, we first set the pixels outside the reference mask as black, then we crop the image according the bounding box of the reference mask. This yields an object-centric image of the masked object. We pass this image to CLIP encoder (Radford et al., 2021) to get the raw mask embedding. We use an MLP layer to map this embedding the input dimension of LLMs.
To obtain the bounding box embedding, we first compute the bounding box coordinates using the generated mask, then we create a positional embedding whose dimension matches the input dimension of L.M.M We use this generated positional embedding as the final bounding box embedding.
For each mask, we obtain the two embeddings and feed them sequentially back the input stream of LLMs. Following LISA, we use [SEG] token to generate the masks. During the training process, employ the teacher enforcing (Williams Zipser, 1989) and directly append the ground truth mask and bounding box embedding after each [SEG] token. At the inference time, compute the two embeddings for each mask generate and insert the embeddings before the input for the next round.
Mask-Aware Decoding. To facilitate the decoding process, generate two tokens and [SEG] the mask decoder. The [REF] token contains information about the referenced mask and [SEG] token contains information about the relational query. For example, in the query "segment the head of finstance 1J" where "finstance 1J" is a previously segmented person, the [ref] token should encode the previous mask finstance while [SEG] should encode the target mask. In the training process, construct two queries. We match first query "[REF). [PAD] " to the referenced mask Miet (finstance 1] in the previous example). and match the second query "[REF] [SEG] the desired mask Mgr (the head of the person in the previous example). The final loss is formulated as:
Tymask Luog ( ((REF). [PAD] Mer)) Lo (F((EFF) [SEG] M
(1)
where Luzg = Loe + ALPICE We apply cross entropy loss and DICE loss to the target mask and reference mask predictions. We set À as by default.
EXPERIMENTS
5.1 IMPLEMENTATION DETAILS
We use a pretrained CLIP- ViT-Large (Radford et al., 2021) with a patch size of 14 as the image encoder, HIPIE-R50 (Wang et al., 2024b) as the mask encoder and LLaVA-v1.5-7B (Liu al., 2024) as the base language model. Compared with LISA, which has exactly one mask per training sample, Le'L setup contains multiple masks per conversation. Hence, replaced the SAM

---

Published as a conference paper at ICLR 2025
MR-RefCOCO GLaMM
MR-ReICOCO+
MR-RefCOCOg GLaMM SegLLM
Rounds
T.SA
S
A
LISA
GLaMM SegLLM
A
LISA
A
#2
60.6
59.5
81.9
+21.3
51.4
$2.9
78.0
+25.1
61.3
65.4
79.2
+13,8
#3
58.9
61.6
81.7
+20.0
51.2
58.3
78.5
+20.2
52.1
57.8
76.0
+18.2
#4
61.3
59.3
78.4
+17.1
49.0
54.2
74.3
+20.1
56.0
55.4
77.1
+15.0
#5 61.0
62.6
80.3
+17.6
48.5
50.5
76.5
+26.0
47.5
49.4
66.9
+14.0
#6 60.7
62.6
74.5
+11.9
45.6
54.8
73.4
+18.6
39.9
40.8
68.9
+24.8
#7
54.4
52.1
69.3
+14.9
42.8
48.4
64.0
+15.6
55.1
57.8
71.0
+13.3
#8
51.9
50.7
70.5
+18.7
36.9
43.6
59.0
+15.4
36.3
38.4
54.9
+16.5
Table 1: Multi-round referring segmentation the proposed multi-round RefCOCO/+/ benchmarks. As
the rounds progress, becomes harder to interact and retain all relevant information, causing the performance measured in cloU to drop. SegLLM can consistently outperform LISA (Lai et al., 2024) and GLaMM (Rasheed et al., 2024), across series of rounds by a significant margin on the MR-RefCOCO/+/g benchmarks.
Methods
RefCOCO
RefCOCO+
RefCOCOg
val
testA
testB
val
lestA
testB
val
test
VLT (Ding al., 2021)
67.5
70.5
65.2
56.3
61.0
50.1
55.0
57.7
LAVT (Yang et al., 2022)
72.7
75.8
68.8
62.1
68.4
55.1
61.2
62.1
LISA-7B (Lai et al., 2024)
74.1
76.5
71.1
62.4
67.4
56.5
66.4
68.5
NExT-Chat (Zhang et al., 2024)
74.7
78.9
69.5
65.1
71.9
56.7
67.0
67.0
SegLLM (ours)
80.2
81.5
75.4
70.3
73.0
62.5
72.6
73.6
Table 2: Comparison between SegLLM and baseline methods on referring segmentation. Although not specifically designed for single-round referring segmentation, the diverse and challenging multi-round referring segmentation tasks and training data enable SegLLLM significantly outperforms previous state-of-the-art methods on standard referring segmentation tasks by substantial margin. We use cloU the main evaluation metric.
ViT-H mask decoder (Kirillov al., 2023) with a smaller HIPIE-R50 (Wang et al., 2024b) to reduce the computation overhead during the training, We then fine-tune the LLM model and the projector weights fvZL using the training set of our own multi-round instruction segmentation dataset MRSeg, while keeping the weights of the CLIP image encoder and the HIPIE mask decoder frozen.
instruction-semmentttoo
We use NVIDIA A100 GPUs for model training. We fine-tune our model with a total batch size of 16 (a per-device batch size of 2) using the Adam W optimizer (Loshchilov & Hutter, 2017) with learning rate of 2e- Furthermore, we utilize stage-2 DeepSpeed accelerator (Rasley et al., 2020) and 6 floating point precision to enhance training efficiency and reduce memory consumption.
5.2 EVALUATION
Evaluation benchmarks. For standard single-round image reasoning segmentation and detection tasks, we evaluate our model on the widely used referring segmentation and comprehension bench- marks. RefCOCO/+/g (Yu et al., 2016). We also conduct qualitative and quantitative comparisons with previous SOTA models on our multi-round referring segmentation henchmarks, based on MSCOCO (Lin et al., 2014a), PACO (Ramanathan al., 2023) and LVIS (Gupta et al., 2019), which assess performance based on positional, interactional or hierarchical relationship queries.
Evaluation metries. We use mean Pinrrectioo-OOer-unne (mIoU) and cumulative Intersection- Over-Union (cloU) as our main evaluation metrics. To assess the model's performance across multiple rounds of conversation, track the mloU and cloU scores for each round's segmentation outputs.
Baseline. Since some baseline models, e.g., LISA, do not natively support multi-round interactive segmentation, for comparisons, we adapt our multi-round validation data into their supported single- turn format by converting the N-turn data into N single-turn instruction segmentation tasks.
5.3 EVALUATION RESULTS
Mutli-round referring segmentation. We compare the performance of SegLLM and LISA on our multi-round referring segmentation benchmarks, MR-ReICOCO/#/g. As shown in Table 1, compared to LISA (Lai et al., 2024) and GLaMM (Rasheed et al., 2024), SegLLM not only achieves 14-26% % higher cloU score across all conversation rounds but also stays stable, whereas LISA and GLaMM's performance tends to degrade in the later turns of the conversation. For example, by round 5, the performance gap between SegLLM and GLaMM widens significantly, reaching over 17.6%, 26.0%, and 14.0% nR-RefCOCO, MR-RefCOCO+, and MR-RefCOCOg. respectively- double
7

---

Published as a conference paper at ICLR 2025
Round
Round
Round 3
Round
Round
Segment. John Mayer.
Sugment the famous British
Sigment the played instance
Ret
Segment the gullarr played by instance
Segment the puitar
Ref-
Instance
SegLiM
X
Round
Round
Round
Round
User
Shementthe
Segment the object Instanesance
Segment ofject instanc standingos on.
Can segment dog?
Can segment trisoee caught instancee
SegLiM
Figure 5: Side-by-side qualitative comparison with LISA's (Lai et al., 2024) on multi-round interactive segmentation. SegLLM not only excels in reasoning segmentation, demonstrating an understanding of world knowledge including recognition of famous individuals, illustrated in the round and round 4 results of the first demo in row one, but also efficiently responds questions that reference previous rounds. Ref indicates the referenced output from previous round.
RefCOCO
RefCOCO+
RefCOCOg
Methods
testA
testB
val
testA
testB
val
test
Shikra-I3B (Chen et al., 2023)
87.8
91.1
81.8
82.9
87.8
74.4
82.6
83.2
VisionLLMMH (Wang et al., 2024a)
86.7
NExT-Chat-7B (Zhang et al., 2024)
85.5
90.0
77.9
77.2
84.5
80.1
79.8
SegLLM-7B (ours)
90.0
92.1
86.2
82.2
85.5
76.1
83.9
85.9
Table 3: Comparison between SegLLM and baseline models on referring expression comprehension (REC). SegLLM not only sets a new SOTA result in referring segmentation (Table 2). but also surpasses baseline models detection tasks, including those specifically optimized for these tasks, such as NExT-Chat-7B (Zhang al., 2024), or models with larger LLMs like Shikra- 13B (Chen al., 2023). The evaluation metric used is the standard detection metric for REC, A0.
Val
Test
Test (long query)
Table 4: Result comparison on the ReasonSeg dataset. SegLLM demon-
Method
mloU cloU mloU cloU
mloU
cloU
LISA (Lai et al., 2024)
53.6
52.3
48.7
48.8
49.2
48.9
strates superior performance, particu-
SegLLM
57.2
54.3
52.4
48.4
55.9
54.2
larly on the long query subset.
the gap observed round (Table 2). For details on the evaluation protocol used to evaluate LISA on our multi-round dataset, please see Appendix B. Besides the quantitative results, Fig. 5 presents the qualitative results comparing SegLLM with LISA (Lai et al., 2024).
Why does SegLLM's performance in later rounds sometimes exceed the first round by 2~44 (Table 2)? This improvement attributed to earlier rounds helping to narrow down the search space in the image, thus enhancing the model's accuracy in subsequent queries.
Single-round referring segmentation and expression comprehension. As shown in Tabs. to 4, SegLLM consistently exceeds previous SOTA methods, such LISA (Lai et al., 2024), NExT- Chat (Zhang et al., 2024) and Shikra-13B (Chen et al., 2023), in the standard single-round referring segmentation and expression comprehension tasks, despite not being specifically designed for these
8

---

Published as a conference paper at ICLR 2025
Simple
Complex
Segment the white chair by the desk.
Gan you help me identily the white chair by the desk in this image. gonerating segmoniation mask?
Please provide segmentation mask bring out the white chair the desk prominentl in this visual
I'm inferested in the white chair by the desk this image Can hwlp me locale by genwrating segmentatonn mask?
goranriinng the
SegLIM
LISA
SegLLM
LISA
SegLLM
LISA
SegLLM
LISA
Figure 6: Demo results that demonstrate SegLLM''s robustness against varying question queries, in contrast to LISA (Lai et al., 2024), which is sensitive to prompt phrasing. Even with simple questions presented in different templates, LISA's performance significantly declines, frequently failing to deliver correct segmentation results for most test templates. This limitation forces users to adhere to specific phrasing, such as "Segment [object descriptions"". substantially restricting the model's real-world applicability.
Multi-Round PACO (w/ LVIS) (mloU))
Multi-RRund PACO (w/ LVIS) (cloU)
Models
LISA
ScgLLM
Absolute A
LISA
SegLLM
Absolute A
round
34.7
54.9
+20.2
45.6
65.3
+19.7
round2
10.6
37.6
+27.0
13.5
49.7
+34.2
round
13.7
32.9
+19.1
21.3
40.9
+19.6
round
11.5
33.3
+21.7
18.7
39.4
+20,7
round 11.6
31.6
+20.0
20.5
41.9
+21.4
Table 5: Single-round referring segmentation and multi-round hicrarchical image segmentation. The
Multi-Round PACO (MR- MMRPAACO) benchmark presents a significant challenge as it demands a good hierarchical understanding and the capability to precisely segment tiny masks representing parts or subparts of an object (refer to hierarchical query demos in Fig. 1). LM significantly improve performance over LISA (Lai et al., 2024), demonstrating substantial improvements in both mlol and cloU metrics across conversation rounds.
tasks. We hypothesize that SegLLM's ability to understand the relative relationships among objects or parts within images in multi-round tasks significantly enhances its overall visual comprehension. This enhanced visual understanding capability transfers to superior performance in single-round tasks as well. However, it is worth noting that while SegLLM shows improved performance single-round tasks, the performance gap is smaller compared to the improvements observed in multi-round tasks.
Multi-round hierarchical segmentation result comparison between ScgLLM and LISA is conducted with our MR-PACO. As detailed in Sec. 4.1, each subsequent round may query part subpart of a whole object from a previous round of conversation. As shown in Table 5, compared to LISA, higher mloU and 13.2~~7..1 1 % higher cloU across all rounds. It is
SegLLM obtains 10.7%~ observed that the absolute model performance typically decreases in later rounds. This decline is primarily due to the progressively smaller object sizes (segment parts of an instance) in later rounds of the multi-round hierarchical segmentation task, as shown in ??. As shown in Fig. 5, SegLLM leverages multi-round segmentation, using previous outputs to accurately identify the necktie of the person in the gray suit in round 2, as requested by the user. In contrast, LISA, lacking this contextual awareness, fails correctly identify the person. This is further demonstrated in round 5, where SegLLM successfully segments Barack Obama's 's necktic from round 4, while LISA fails again.
Robustness against question templates. We observed that many previous studies in image reasoning segmentation, such as LISA (Lai et al., 2024) and SES AME (Wu et al., 2024), tend to overfit to the specific question templates used during training. Consequently, when these models are evaluated with diverse question templates not encountered during training. performance often significantly declines. For example, shown in Table 6. the performance of LISA and SESAME drops by approximately 7% and 13% respectively, when assessed using our varied templates.
To mitigate this, we intentionally diversified our question templates during the dataset generation process. As result, our SegLLMM model not only demonstrates consistent segmentation performance across diverse templates but also achieves a 5.5% higher cumulative Intersection- Siriiinn--ve--nniio (cIoU), Notably, this performance gain occurs on the single-round referring segmentation benchmark, which these prior studies were specifically optimized for. SegLLM even achieves 2.3% higher cloU on RefCOCO+/g than (Wu et al., 2024) on its own question templates. Fig. 6 shows that LISA's performance significantly drops when asked with simple questions presented in various templates, frequently failing to produce correct segmentation results for most test templates.

---

Published as a conference paper at ICLR 2025
Methods
Averaged
Diverse
LISA
SESAME
RC RC+ RCg
RC
RC+
RCg
RC
RC+
RCg
RC RC+
RCg
SESAME (Wu et al., 2024)
67.4 57.9 61.4
66.0
56.9
60.6
61.4
51.6 55.6
74,9
65.1 67.9
LISA-7B (ft) (Lai et al., 2024)
70.1 61.0 63.0
67.8
59.0
62.4
74.7
64.9
66.1
67.8
59.2 60.6
SegLLM (ours)
79.7 70.0 72.2
80.2
70.3
72.6
80.4
70.7
72.3
78.6
69.0 71.6
vs. prev. SOTA
+9.6 +9.0 49.1
+12.4
+11.3
+10.2
+5.7 +5.8 +6.2
+3.7
+3.9 +3.7
Table 6: SegLLM Exhibits greater robustness to variety of question templates in image reasoning segmentation. Unlike previous models such as LISA (Lai et al., 2024) and SESAME (Wu et al., 2024). which tend overfit to specific question templates encountered during training. S demonstrates improved robustness. We assess performance using the single-round RefCOCO dataset with cumulative cloU as the evaluation metric. Notably, the templates used for evaluation were not utilized during the model training process.
Single-Round ReiCOC///
Multi-Round RefCOCO///g
Multi Round (hard)
Mask-enc
Box-enc
Ref' Loss
RefCOCO/+
X
80.2/ / 70.8
59.6753.9755. 55.2
32.4/32.3 34.1 67.6 67.3/6../// 62.8
X
/2.3/ / 77.8 75.7 / 71.3 68.6
/2.5 76.7
74.0 70.1 / 65.8
69.6 / 69.4 63.7
Table 7: Ablation study on the effectiveness of proposed components. Model performance evaluated on three benchmarks: (1) Single-Round Referring segmentation in single round using standard RefCOCO, Seitt--ooundReeeeeeeee
segmentation over multiple rounds, based on our custom benehmarks from RefCOCO, RefCOCO+, and RefCOCOg datasets (results show a weighted average of standard and hard subsets). (3) MRSeg (Hard): Multi-Round (Hard) (Hard)-Focuses exclusively on the hard subset of the multi-round segmentation benchmarks. The evaluation metric used is CloU.
ReiCOCO+, and RefCOCOg datasets. (2) MRSeg:
5.4 ABLATION STUDY
Ablation study on the effectiveness of proposed components. We conduct an ablation study (Table 7) Multi-Round RefCOCO benchmark evaluate the effectiveness of the three components we introduced in Sec. 4.2. We assess model performance across three subsets of the MR-ReíCOCO dataset: 1) Single-round: single round referring segmentation using the standard RefCOCO/+/g datasets. 2) MRSeg: multi-round referring segmentation based on our MRSeg. 63) MRSeg (Hard): this subset focuses exclusively on the hard subset of the MRSeg benchmarks, where understanding the reference mask is crucial for accurately segmenting the correct object. We provide more details for MRSeg (hard) in Appendix A.
Overall, in the MRSeg task, our proposed components lead to significant 20% performance improvement over the baseline. In the MRSeg (hard) subset, our mask-encoding scheme achieves over 30 points higher cloU compared to the baseline. These results highlight the effectiveness of our approach in enabling the model to interpret visual cues from user instructions and perform mask-conditioned segmentation critical for handling complex tasks where reference masks, rather than text- based instructions, provide key information.
The proposed components also improve results single-round referring segmentation shown in Table 7. This achievement is noteworthy as the task does not explicitly require box-encoding and mask-encoding. We hypothesize that the absence of these encoding modules complicates the learning of segmentation from multi-round instructions, resulting in less stable training dynamics that affect performance even in single-round tasks.
CONCLUSIONS
We introduce SegLLM, a novel multi-round interactive reasoning segmentation model that enhances traditional segmentation models by retaining conversational memory of visual, not just textual, results. Utilizing a mask-aware multimodal large language model, SegLLM integrates previous segmentation outputs back into its input stream, allowing it to handle complex queries about relationships between objects across multiple interactions. Tested on the newly curated MRSeg, SegLLM significantly outperforms existing benchmarks in multi-round interactive segmentation by over 20% and shows a 4.7% improvement single-round referring segmentation. These results demonstrate LegL.LM's capability as a versatile model for broad range of caarruction-ollowiiin segmentation tasks.
Acknowledgment. We thank helpful discussions with Tsung- han Wu, Xingyi Zhou, Alireza Fathi, and Cordelia Schmid. XuDong Wang and Trevor Darrell were funded by DARPA and the Berkeley AI Research (BAIR) industrial alliance programs..
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023. 1,5
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning, Advances in neural information processing systems, 35:23716 23736, 2022. 1,3
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01399, 2023. 1,
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33: 13:1877-1901, 2020. 2
Holger Caesar, Jasper R. R. Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. 2018 JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1209-1218, 2016. URL https: /ppi. www.et ics.chlarcccrrr org Corpus ID: 4 4
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal 1lm's referential dialogue magic. arXiv preprini arXiv:2306. /5/95, 2023. 3,8
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1290-1299 2022. 1
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 240240) 2023. 2
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024. 3,3
Daan de Geus, Panagiotis Meletis, Chenyang Lu, Xiaoxiao Wen, and Gijs Dubbelman. Part-aware panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5-85-548999 2021. 4
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04055 2018.
Henghui Ding, Chang Liu, Suchen Wang. and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 16321-16330, 2021.
Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary universal image segmentation with maskclip. In Proceedings of the 40th International Conference on Machine Leurning, pp. 8090- 8102, 2023. 1
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 5356-5364, 2019. 4,7
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision. pp. 2961-2969, 2017.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
11

---

Published as a conference paper at ICLR 2025
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682) 2023.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pp- 787-798, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan- Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2300. 02643, 2023. 3, 4,
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.4 4
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9579-9588 2024. 1, 2, 3, 5, 7, 8, 9, 10
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. 2023. 2
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023b. 3
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014a.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014b. URL pittps://apte wemanticscholar..o.
CorpusID:1411377
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 3,
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17Xiv:1711.05001 2017.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini arXiv:2304.07192. 200.13111..2.11331..222222
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models the world. arXiv preprint arXiv:2306. 14824, 2023. 3
Renjie Pi. Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong. Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, and Lingpeng Kong Tong Zhang. Detgpt: Detect what you need via reasoning. arXiv preprint arXiv:2305. 14167, 2023a. 1,3
Renjie Pi, Lewei Yao, Jiahui Gao, Jipeng Zhang, and Tong Zhang. Perceptiongpt: Effectively fusing visual perception into Umm. arXiv preprint arXiv:231 1, 2023b.
12

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021. 1, 6
Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7141-7151, 2023. 4,7
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuue Yang, and Fahad Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13009-13018, 2024.
Jeff Rasley, Samyam Rajbhandari. Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp- 3505-3506, 2020.7 7
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023. 1
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023. 1, 2,
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024a. 8
Xudong Wang, Shufan Li, Konstantinos Kallidromitis, Yusuke Kato, Kazuki Kozuka, and Trevor Dar- rell. Hierarchical open-vocabulary universal image segmentation. Advances in Neural Information Processing Systems, 36, 2024b. 6, 7
Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280, 1989, 6
Tsung-Has Han Wu, Giscard Biamby, David Chan, Lisa Dunlap, Ritwik Gupta, Xudong Wang, Joseph E Gonzalez, and Trevor Darrell. See say and segment: Teaching Imms to overcome false premises. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13459, 2024. 1,9, 10
Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mart prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint vrXiv:2310.11441 2023a. 3
Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18155-18165, 2022.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Li- juan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.1742.1 9(1):1, 2023b. 3
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14172 2023. 3
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint ar3iv:2306. 13549, 2023. 3
13

---

Published as a conference paper at ICLR 2025
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXhv:2310.07706 2023. 3
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg. and Tamara Berg. Modeling context in referring expressions. In Computer Vision-ECCH 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 69-85. Springer. 2016. 4,
Andy Zeng. Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00595 2022.
Zhang, Liming Zhao, Chen-Wa Xie, Yun Zheng, Wei Ji, and Tat-Seng Chua. Next-chat: An Imm for chat, detection and segmentation. arXiv preprint arXivv231 10311.00495 2023a. 1,3
Ao Zhang, Yuan Yao, Wei Ji, Zhiyuan Liu, and Tat-Seng Chua. NExt-chat: An LMM for chat, detection and segmentation. In Forty-first International Conference on Machine Learning, 2024. 7,
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of region-of-interest. interesst arXiv preprint arXiv:2307.03601 2023b.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene pars- ing through ade20k dataset. 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 5122-5130, 2017. URL httphttt///// / /app. semant seeanticccholaaee org CorpusID: 5636055. 4
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 3
A APPENDIX
A.1
DATASET DETAILS
In the section, we document further details on the dataset construction process. We also provide some statistics about our dataset.
A.1.1 DATASET SIZE
We document the number of images sampled from each source dataset and the number of conversations generated in Table Al. Additionally, we visualize the distribution of the number of rounds for each dataset Fig. A1.
Datasets
Training Set
Validation Set
# of Convs
of Images
Max Rounds
# of Convs
# of Images
Max Rounds
Re(COCO(+/g)
55188
27674
4263
2701
17
Visual Genome
367674
94221
40980
10524
PACO-LVIS
40827
40827
19
2178
2178
16
LVIS
71388
71255
17
13898
13898
18
Pascal Panoptic Part
4577
4577
17
4690
4690
18
ADE20K
59784
20196
$943
200
COCO-Stuff
340127
118205
14461
4999
Ciabutes-COCCO
49036
36413
5000
2566
ReasonSeg
1326
239
200
200
1
MRSeg (hard)
22470
22470
1988
1988
Table A1: Statistics of our MRSeg dataset, including the number of overall conversations, number of images, the maximum rounds of conversations for each dataset after processing through our dataset pipeline.
14

---

Published as a conference paper at ICLR 2025
Number harnnn Comversbbors found Count
Number velldation Conversation per Round Count
3000
4000
3000
2900
FNOOUSS
Prac Farcat farts
1r0000/0g
Figure A1: Bar-plot visualization for training and validation conversations count different number of rounds for multi-roum datasets. There very conversations with large number of rounds.
A.1.2 CONVERSATION GENERATION PIPELINE
We employ different strategies to generate natural- language conversation for different source datasets. Specifically, our dataset is generated using combination of the following methods:
Hierarchical relationships based on PACO-LVIS and Pascal Panoptic Part: In these queries, the model is asked to segment objects which are sub-pan of some output of a previous round. From each image, we randomly sample between one and four instances, and for each instance, randomly sample between by questions targeting the parts of each respective instance. For Pascal Panoptic Part, we only use objects and their parts instance level and not semantic segmentation level to avoid ambiguity. For both PACO-LVIS and Pascal Panoptic Part, we refer to previous round outputs with it's actual caption, e.g. the knife' with probablility 50%. With the other 50% refer to the previous round output as "Kinstance i>' or <<the out of round i>
and four parts. We initiate queries about the instance followed
Based on certain probabilty, we other ask about the ournet eaplion or select a another caption iquurn about.
Randomly select between saptions
Single Image
Captions
Solected Captions
Selected Captions Relativa
captions accay ["right teay pizza
relative, array [-1, D,
aliows "luft "pieza quy
Index represente indax of the corsesponding object in captions aceay.
glasses left pizza slice")
Multi-Round
Caloulate spotial melationsiip to chosen related objects
Deee:: I'me looking for the eegrentation wask
Septia:: alices the image.
reladoonaiip and insancce annolations to somplates
Could high cusult
Captions, Objects and Rolationships
quy glassen zub subject aegrentation?
left"
"balow"
SegliM [3]
Neee: Can you Find and segeens the Jait piees not
the alice that in below the cutgut of cound 27"
Generated Templates
ReglaN Ware,
Figure A2: Pipeline for generating multi-round conversational data for ReSCOCO(+/g) in MRSeg.
These conversations task the system with segmenting objects based on their positional relationships to the outputs from previous rounds. We randomly sample between 2 to 18 annotations per image. For each selected annotation. we either generate a query about the object itself or generate a query involving an object from previously processed instances, focusing on their relative positions calculated from their bounding box coordinates. For RefCOCO(+/g) multiple annotations may be selected for the same instance due to multiple captions available per instance. For LVIS, we select annotations where only one or two objects of that class appear in the image. When two objects of the same class are present, we detail their relative positions and add location descriptions to their captions to prevent ambiguity. We specifically choose instances not categorized under COCO classes to diversify the dataset's class variety. The probability for each round to query about an object itself is 1 1/3, otherwise, query about the current object with reference previous round's 's output and their relative position. To assign the positional relationships, we use compare the edge and center position of the bounding boxes for the two instance
Positional relationships based Refcoca(+/g) and LVIS:
trying to assign relationship to.
15

---

Published as a conference paper at ICLR 2025
There 9 total possible positions two instances can have (the same as, overlapping with, the left/right, above/below, to the top/bottom left/right of). Similar to Hierarchical Queries we refer to previous round outputs with it's actual caption, e.g. 'the woman on the left' r with probablility 50%. With the other 50% we refer to the previous round output as >><instance
i>'r <<the output of round i>' A detailed pipeline for how RefCOCO(+/g) dataset is sampled can be see in Fig. A2
MR Seg(hard): For each RefCOCO image, we identify cases where there are two instances of the same class within the image. From these, select pair of instances and construct two single-round conversations. Given two instances, X and Y, of the same class in the image, we create the following conversations:
Conv 1:
[IMAGE] [ENCODE X] Please segment the other <class
name> + Sure, [DECODE Y]
-Covv 2: [IMAGE] [ENCODE Y] Please segment the other <class
name> - Sure, (DECODE x]
We have 10 different templates for the training and 5 templates validation/test for MR Seg(hard). Interactional relationships based on Visual Genome: We adopt Visual Genome (VG), utilizing its relationship annotations construct conversations that emphasize interactional dynamics rather than merely positional relationships. We sample up to four relationships per image. Each relationship prompts a two-round conversation: the first round involves segmenting the subject, and the second round involves segmenting an object based on its relationship to the subject. Since VG also only provides bounding box labels, we generate masks for selected instances using SAM.
A.1.3 DETAILS OF GPT4 USAGE
We prompt GPT-4 models for generating captions for attribute- based descriptions as well as for cleaning grammar errors in our dataset. The detailed instructions and specific model we used can be found in Table A2 and Table A3. For the attribute-based description, we crop COCO images to only contain the specified instance, feeding the cropped image and it's class name to GPT to generate a description. For language correction, we found that grammar correction is often erroneous but can be a lot of accurate if we go through the data twice to double check.
A.1.4 MORE DISCUSSIONS ON DEMO OUTPUTS
In Fig. A3, example A illustrate the necessity of our Mask-Encoding Scheme, to avoid the ambiguity that may arise in cases where multiple instances of the same class are present in the image. Round and round 3 in example A show that without our mask encoding mechanism to supply information about the person segmented from round 1, since there are multiple laptops and chairs present in the image, confusion arises as to which specific laptop or chair the user is referring to in the query prompt. Therefore, without the guiding information from the mask encoding, LISA seems to naively guess the incorrect laptop in round 2, and does not generate a comprehensible segmentation mask in round 3. In contrast, the mask encoding guides our model to correctly segment the requested objects. Similarly, in round and round 6, our model was able to successfully segment the keyboard of the laptop from round and the person setting on the chair from round 5.
This phenomenon is again demonstrated in B in Fig. A3, Since there are two women, both carrying bags and holding an umbrella in the image, our Mask-Encoding Scheme again resolves this the ambiguity and allows the user to conveniently specify the bag and the umbrella requested in round and round 3 are carried and held by the person from round 1. As before, the awareness of previous round outputs enables our model to segment the correct objects, whereas LISA guesses the incorrect objects due to the lack of this awareness.
Example C demonstrates that our model is not limited to multi-round prompting, and can produce accurate segmentation results via direct, single-round prompts as well. In the indirect case, we first ask the model to segment the dog during the first round of the conversation. Then, in the second round, we ask a follow up question to guide the model to segment the Frisbee that is caught by the dog from round 1. However, tin the direct case, we straight away ask for the Frisbee that is caught by the dog. In comparison, our model succeeds in both the direct and indirect case, whereas LISA fails to segment the correct Frisbee instance in either cases. This shows that our multi-round comprehension capability is not limitation but addition.
16

---

Published as a conference paper at ICLR 2025
payload
moodel" -ottt-4--uooo2222---0
messaggss:
"role"; "user",
"content";
"type": "text",
"text"; f"Can you focus on describing the [class_name]
the image? Can you format your output a two item array, such that the first index is an abstract
description without any class name, such as "has pizza sitting top of it' or 'is wearing beige t-shirt' and the second index is the exact classnane for the object, such as 'a dining table' or 'a man'."
b,
"type": "image_ur1", "image_uri"" uri": "url": f"data:image ipeg; base64, 1base64 image)", "detail"; "low"
]
"max max__tokens"; 200
Table A2: Our full prompt to the GPT-4-urbo-2024-00000 model for generating abstract descriptions
Lastly, note that round 3 and round 6 of example A, round 2 and round 3 of example B and round 2 of example C demonstrate our model's understanding of interactional relationships as introduced Sec. 4.1 and round demonstrates the hierarchical relationship introduced in Sec. 4.1.
B DETAILS OF COMPARISON WITH LISA
Since Lisa does not naively support multi-round training, ensure fairness, employed two different approaches:
Approach One: We substitute the mask and bounding box encoding tokens of the reference instance with the word "mask". For example, a query in MR-RefCOCO dataset "Segment the person to the left of <mask> <box>." would be converted to "Segment the person left to the mask."
Approach Two: We substitute the mask and bounding box encoding tokens with the descrip- tion of the reference instance. For example, query in MR-RefCOCO dataset "Segment the person to the left <mask> <box>." would be converted to "Segment the person left the dog chasing after butterfly." (where <mask> <box>> encoding tokens of the reference instance "the dog chasing after a butterfly")
We report results on MR-RefCOCO in Table A4. SegLLM outperforms both alternative approaches 1 and 2. Furthermore, we find that LISA performs worse using approach compared to approach despite the inclusion of the description of the reference instance. We suspect that this may be due to LISA being trained on data that focuses on instance, hence the presence of description for two instances, the target and the reference instance, may cause confusion than guidance. Regardless, in our main table Table 1, we report LISA's performance on our MR RefCOCO/+// benchmark using the best approach for LISA, approach 2.
17

---

Published as a conference paper at ICLR 2025
Round response
cl cht.tttccattccmmllee...... create(
response_format- "type": "json_object" },
messages=1
I"role": "system", "content"; "You are a helpful assistant designed to output JSON."I, ("role": "user", "centent"; f"Can you fix any errors and make the sentence sound like natural English, and dictionary of format
provide our output SENTENCE? here is the sentence worrected'=CoRREN =CRRECTSSENEEECEE want you to correct, [sent)"")
Round
response = client. chatentccha.ccooo www.:hht.ctmmmmciccccccooomm create 10-000----222---1113
response_format-f "type": "json_object" }, messages-
|"role": "system", "content"; "You are a helpful assistant designed to output JSON",, "rrole:: "user", "content": f"Here is the original
sentence: (sent)'. Here is the corrected sentence: [correctee_sentt]' Does the corrected sentence have the sane meaning as the original? If yes, please output ['Same', 'None'). If no, please output ['Different',
"ccorrected with Baath_ssme__eeate same wtw.eeeaannngg_sooiiiiiiii
Table A3: Out full prompt to the gpt-4o-2024-05- 13 model for grammar correction. We use two-round approach, feeding GPT's first round answer back to itself be self-correeted.
Table A4: Comparison of SegLLM, Lisa(Approach 1), and Lisa 2) MR-ReICOCO evaluation
SegLLM
Approach
Approach 2
round
81.9
60.6
55.9
round 81.7
58.9
54.7
round
78.4
61.3
56.7
round
80.3
61.0
57.8
round 6
74.5
60.7
57.7
round
69.3
54.4
45.6
round 8
70.5
51.9
50.3
C LICENSE
We makes use the following models: CLIP (MIT license), LLAMA (Llama Community License Agreement), Vicuna (Apache2 license). BLIP-2 BSD-3-Clause license)
We use the following dataset COCO crabbbiion-NonCCmmmrrialll RefCOCO (Apache-2.0 license), Visual Genome (Creative Commons Attribution 4.0 International License.), PACO (MIT License), Pascal-Panoptic-Parts Apache-220 license), LIVIS (CC BY 4.0 + COCO license).
4.0 Internationa),
D LIMITATIONS
Although have shown promising quantitative results in the novel multi-round reasoning segmentation task, our method exhibits several limitations upon qualitative examination, as shown in Fig. A4 and discussed in detail below. In addition to revealing potential weaknesses within
18

---

Published as a conference paper at ICLR 2025
SegLLM
LISA
SegLLM
LISA
Rnd 1: Segment the man
Rnd 1: Segment the batter.
Ref:
Rnd 2: Segment the person crouching down with the glove to the right of instance 1.
Ref:
Rnd 2: Segment the other person.
Ref:
Rnd Segment the object that instance is holding.
Rnd 3: Segment helmet instance
Ref:
Rnd 5: Segment the arm of instance
Rnd 1: Segment the TV.
1.
Ref:
Rnd 2: Segment the object need to control it.
Rnd 6: Segment the hair tie of instance 2.
Figure A3: Additional bide-by-side comparison with LISA. This shows that without of segmentation outputs from previous rounds, LISA struggles to identify correct instance requested by the user, when there ambiguity. Rof indicates the referenced output from previous round.
our proposed model components, perhaps the existence of failure cases in qualitative evaluation despite the impressive performance in quantitative evaluation indicates that our dataset construction methodology also requires improvement such as including harder test samples. It is our hope that these findings will encourage further research along the direction of multi-round segmentation,
19

---

Published as a conference paper at ICLR 2025
aiming to improve the incorporation of conversation and segmentation history, address some of these limitations and extend upon our initial proposals and approaches.
Sensitivity to conversation history order. Given fixed input queries for subsequent rounds, when the order of the previous rounds in the conversation history are permuted, the model's output is not consistent. As shown by conversation 1 in Fig. A4, in rounds 1 and 2, version A first asks for the left paddle board and then the right paddle board, whereas version B first asks for the right paddle board and then the left paddle board. However, in rounds 3 and 4, both version have the same queries, first asking for the person standing on top of the paddle board from round I followed by the person standing on top of the paddle board from round 2. Despite having the same imput queries for rounds and 4, the model only succeeds in segmenting the correct instance in version A and fails to do so in version B. From the user's standpoint, the two versions are equivalent, since the user is equally likely to start the conversation by asking for the left paddle board, as for the right paddle board. However, the model's s behavior not invariant under the permutation of the conversation history, suggesting that the robustness of our model can be improved.
Sensitivity to input query order. Symmetric to the previous case, given a fixed conversation history, if the order of the queries in subsequent rounds are permuted, then the model's output is also no consistent. As shown by conversation 2 in Fig. A4, in rounds and 2, both version A and version B first asks for the person wearing red followed by the person wearing blue. Then, in rounds 3 and 4, version A first asks for the chair that the person from round 1 is sitting on (query 1), followed by the chair that the person from round 2 is sitting on (query 2), and version B asks the same queries but in the opposite order. However, despite the queries being the exact same, simply switching the order of these two queries causes the model to only succeed in segmenting the correct instance in version A and fails to do so in version B. Again, from the user's perspective, the two versions are equivalent, since the user is equally likely to first request an instance related the output from round 1, as to first request an instance related the output from round 2. However, the model's behavior is not invariant under the permutation of the input queries in subsequent rounds, suggesting that the robustness of our model can be improved.
Independence of encoding information. Although we quantitatively showed that using our proposed mask encoding component to re-introduce past segmentation outputs into the model's input stream can surpass the performance of other models without this component on our multi-round segmentation benchmarks in Sec. 5.3, well verified its effectiveness through conducting ablation study in Sec. 5.4, qualitative evaluations show that the mask encoding information provided by this component may be underutilized by the model. As shown by conversation 3 in Fig. A4, in rounds 3 and 4, the helmet belonging to the kid from round 1 and round 2 are asked, respectively. In version A, the mask encodings corresponding to the reference instance (the kid from round round 2) are supplied to the model, whereas in version B, the relevant encoding information were not provided. However, despite this lack of encoding information, the model is able to successfully segment the correct instance in version B as well as in version A. This suggest that perhaps the textual information of "instance 1" is sufficient for the model to reason that the helmet requested in round 3 corresponds to the one belonging to kid from round 1. A Alternatively, the model may just be randomly guessing, which in this example has a success probability of 0.5. Regardless, it may be worthwhile to re-think the design of our multi-round dataset. Perhaps increasing the level of ambiguity by filtering for images where > 2 instances of the same class are present can force the model to rely on mask encoding information to reason about the requested instance.
Sensitivity to positional keywords. Lastly, during qualitative evaluation, we also observed that our model can be highly sensitive and reliant positional keywords, such as "left" or "right". As shown in single round conversations 1A and IB in Fig. A4, the input queries are purposefully constructed to include the positional word "left" whilst requesting for the person the right (1,), and including word "right" whilst asking for the person on the left, respectively. Indeed, the model falls for this trick, segmenting the person on the left and on the right, whenever the word "left" or "right" is present in the prompt, respectively, failing to comprehend the referring expression. This indicates that perhaps the model is paying the most attention such positional keywords, instead of understanding the entire expression. Moreover, as shown in single round conversations 2A and 2B in Fig. A4, the model able to correctly segment the request instance when given a referring expression that includes a positional keywords "right" (2A), but fails to do so when given an alternative referring expression that doesn't 't include any positional keywords. This further indicates that currently our model relies heavily on positional keywords that appear in the referring expression for the request instance, thus
20

---

Published as a conference paper at ICLR 2025
limiting its accuracy and generalization beyond expression that do not contain such keywords. As mentioned previously, perhaps re-designing the construction of our dataset, such as by reducing the number of samples involving Positional Relationships (see Sec. 4.1) and introducing more complex relationships between instances may alleviate this dependency on positional keywords.
21

---

Published as a conference paper at ICLR 2025
Round
Round 2
Round
Sogment the paddle the left.
Segrnent paddle the right
Segment person standing on top of inscancee 1.
Segment person standing on top instance
Conv.
Segment the poddle board thert
Segment the paodle board on left
Segment the standingi instance 1.
Segment the person standingon top of instance
GT
Conv. ver.
Segment wearing
Sepment che
chair sitting
Rwe.
chair
sitting
Conv. ver. A
Segment wearing
Segment the
Segment the chair instance is sitting
Ret
Segment chair sitting
Ree.
Conv. ver. B
Segment the kid riding the bike.
Sogment the kid kiddidingg the skateboard
Sogment the helmet of Instance
Segment the helmet of instance 2
Conv. 3 Ver. A
riding
Sogment kid the skateboard.
Segnent the helmet
Segment the helmet instance 2
bike
Conv. ver B
Single Round 1A
Single Round 1B
Single Round
Single Round 2B
Segment the penson looking left.
Sogment the porson leaning the right.
Sepment the person right
Segment the person holding the black umbrella.
GT
Figure A4: Limitations exhibited by model during qualitative evaluation. (Top row) Sensitivity order of conversation history, given fixed queries in subsequent round. (2nd row) Sensitivity to order of queries in subsequent rounds, given fixed conversation history. (3rd row) Independence of mask encoding information. (Bottom row) Reliance positional keywords referring expression of query.
22


Reviews:
[{"review_id": "Z5hcV7uGVf", "reviewer": "Reviewer_rtU2", "summary": "This paper proposes SegLLM a new Multimodal Language Model capable of performing multi-turn segmentation tasks based on language inputs. The paper introduces two new mechanisms to ease the multi-turn segmentation by allowing the LLM to get mask and bounding box tokens as inputs. This mechanisms allows the LLM to understand multi-turn instructions that can refer to previously predicted masks. Along with the method, the paper is paired with a new dataset to train multi-turn segmentation tasks. The dataset was created automatically by using current single-turn datasets and leveraging the capabilities of Language Models to adjust input prompts.\nThe results show that SegLLM is capable of solving multi-turn segmentation better than other alternatives, while also improving at single-turn segmentation benchmarks.", "questions": "There are a few aspects of the paper that were not entirely clear to me, and I would appreciate some clarifications:\n\n1. **Metric for Multi-turn Segmentation**:\n   - Is the goal of multi-turn segmentation to segment only the object referred to in the final turn, or is the model expected to get each intermediate step correct as well?\n   - Additionally, does each turn in the multi-turn setup refer to only one instance at a time?\n\n2. **Evaluation of Baselines on MR-RefCOCO Dataset**:\n   - How were the baselines evaluated on the proposed MR-RefCOCO dataset? I did not see details on whether the baselines were trained using the same training split as SegLLM.\n   - If the training split was not used to refine these methods, could this weaken the strength of some conclusions? For instance, if SegLLM’s robustness to query diversity is largely due to the dataset rather than the model itself, wouldn't training LISA on this dataset also equip it with similar robustness?\n   - This additional context would also clarify the performance gap observed between previous methods and the proposed one.\n\n3. **Ablation Study Execution**:\n   - Although the ablation study results look good numerically, I find some of the details around execution unclear. For example, how does SegLLM perform multi-turn segmentation without the module that enables it to \"see\" previously segmented masks?\n   - Without access to these masks, how would the model handle queries that refer back to prior segmentations? More details on how the model operates without these critical components would make this section much clearer.\n\n4. **Multi-turn Segmentation by Other Methods**:\n   - Related to the previous question, how do other methods in this study perform the task without mechanisms to refer to previously segmented masks?\n   - Do you feed an image back into the model with the segmented instance painted in? If so, could you provide more details on this approach?\n\n\nI would love to hear these clarifications from the authors. I believe the paper is already very good, and making it crystal clear for the reader would make it a great work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is really well presented and written. The ideas are clear and the problem that the paper is trying to solve is clear from the get go. \nThe method is also clear and simple to understand, while still being technically sound. The validation of the method using the self-created dataset and benchmark is also a great contribution for the community. The ablation studies are solid and conclusive and validate the design choices.", "weaknesses": "There are a few of the paper that were not 100% clear to me and I would appreciate some clarifications.\n1. The metric for the multi-turn segmentation is not clear to me. Is the goal of multiple turn to simply segment correctly the object that is referred in the last turn? Or does the model have to get every single middle step correctly as well? Additionally, is every turn referring to only one instance at a time?\n\n2. Is is not clear how were the baselines evaluated on the proposed MR-RefCOCO dataset. I did not see the details on whether the baselines were trained using the same training split as SegLLM or not. If the training split was not used to further refine these method then some conclusions become less strong. For instance, the fact that SegLLM is more robust to query diversity is mostly due to the dataset but not the method itself, then just training LISA on this dataset would equip it with similar robustness to query diversity. Additionally, that would explain the big gap between the previous methods and the proposed one. This needs a more in-detail explanation to understand the results of the paper in detail.\n\n3.  The ablation study although looks good in terms of numbers, is not clear in terms of excecution. It is not clear to me how can SegLLM perform multi-turn segmentation without having the module that allows it to \"see\" the previously segmented masks.  How would the model be able to understand queries that refer to previously segmented masks? This whole table needs more details on how would the model operate without some of these critical components. \n\n4. Related to the previous question, it is also not clear how other method can perform the task without the mechanisms to refer to previously segmented masks. Do you feed an image with the painted instance back to the model? How is it performed?\n\n\n\n\nAdditional notes:\n\nWhen showing the queries to ChatGPT API in the supplementary materials, I would suggest to simply put the question/template and answers instead of the whole command used. It is not needed and it makes it really hard to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T7TVP03KxH", "reviewer": "Reviewer_PsJ9", "summary": "This paper introduces SegLLM, an advanced multi-round interactive segmentation model that improves LLM-based segmentation by incorporating conversational memory of past visual and textual outputs. Utilizing a mask-aware multimodal LLM, SegLLM refines its inputs with previous segmentation results, allowing it to handle complex queries about object relationships across multiple interactions. Testing shows SegLLM outperforms existing methods in both interactive and single-round segmentation tasks.", "questions": "Will the code, dataset, and details on data processing be made open-source?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper proposes a multi-round reasoning segmentation task, which extends the boundaries of traditional reasoning segmentation and introduces new avenues for exploration within the community.\n2. The paper presents the SegLLM model, which supports both bounding box and mask inputs, expanding the capabilities of large segmentation models. For the multi-round reasoning segmentation task, the authors carefully designed special tokens and a corresponding mask decoder. These modifications, in my view, represent an extension of the LISA paradigm with practical significance.\n3. The paper constructs a multi-round reasoning segmentation dataset, drawing from diverse sources and following a well-designed pipeline, thus advancing research in this area through a robust data contribution.\n4. The experiments are thorough, showcasing the task’s difficulty while effectively validating the proposed modules.", "weaknesses": "1. Methods like LISA employ LoRA for fine-tuning; does this paper adopt a similar approach? If not, could the observed performance improvement over LISA be attributed to a more advanced fine-tuning strategy?\n2. The paper lacks an ablation study with masks and bounding boxes separately as inputs, and it is unclear whether using points or drawing trajectories as prompts would be viable.\n3. There are minor errors in the text, such as on lines 427–428 and line 470.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nDymr3lNEk", "reviewer": "Reviewer_g3qz", "summary": "This paper introduces SegLLM, an advanced interactive reasoning segmentation model that leverages **multi-round dialogue capabilities** in Large Language Models. SegLLM outperforms existing models by enhancing interactional reasoning and memory retention, demonstrated on a new benchmark, MRSeg, with substantial gains in accuracy metrics.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a novel integration of mask-aware LLMs for segmentation, enabling iterative, memory-based multi-round interactions.\n\n2. SegLLM outperforms state-of-the-art segmentation models like LISA with significant performance margins across single and multi-round tasks.\n\n3. Paper provides a comprehensive MRSeg dataset, structured for interactional and hierarchical segmentation tasks.\n\n4. Paper demonstrates robustness against diverse question templates, improving the practical applicability of multi-modal conversational AI.", "weaknesses": "1. Using vision prompts for referring segmentation is a good approach. However, I would like to see a more comprehensive ablation study in Table 7 of the main paper, exploring the effects of different types of vision prompts, such as using only the mask encoder or only the bounding box encoder.\n\n2. It would be valuable to understand the performance impact of directly using the `[SEG]` token from the previous round as the visual prompt for the current round. This approach might reduce the computational overhead introduced by the mask and bounding box encoders.\n\nMinor Comment: Typically, table captions are placed at the top rather than the bottom of the table.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)