Text4Seg: Reimagining Image Segmentation as Text Generation
Authors: Mengcheng Lan
Chaofeng Chen
Yue Zhou
Jiaxing Xu
Yiping Ke
Xinjiang Wang
Litong Feng
Wayne Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal large language model
Image segmentation
Referring expression segmentation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Text4Seg introduces a novel text-as-mask paradigm that simplifies image segmentation by framing it as a text generation task, achieving state-of-the-art performance without architectural modifications to MLLMs.

Abstract: Multimodal Large Language Models (MLLMs) have shown exceptional capabilities in vision-language tasks; however, effectively integrating image segmentation into these models remains a significant challenge. In this paper, we introduce Text4Seg, a novel text-as-mask paradigm that casts image segmentation as a text generation problem, eliminating the need for additional decoders and significantly simplifying the segmentation process. Our key innovation is semantic descriptors, a new textual representation of segmentation masks where each image patch is mapped to its corresponding text label. This unified representation allows seamless integration into the auto-regressive training pipeline of MLLMs for easier optimization. We demonstrate that representing an image with $16\times16$ semantic descriptors yields competitive segmentation performance. To enhance efficiency, we introduce the Row-wise Run-Length Encoding (R-RLE), which compresses redundant text sequences, reducing the length of semantic descriptors by 74\% and accelerating inference by $3\times$, without compromising performance. Extensive experiments across various vision tasks, such as referring expression segmentation and comprehension, show that Text4Seg achieves state-of-the-art performance on multiple datasets by fine-tuning different MLLM backbones. Our approach provides an efficient, scalable solution for vision-centric tasks within the MLLM framework.

Published as a conference paper at ICLR 2025
TEXT4SEG: REIMAGINING IMAGE SEGMENTATION AS TEXT GENERATION
Mengcheng Lan, Chaofeng Chen, Yue Zhou S-L.ab, Nanyang Technological University lamm00020e .nt..e.ee.................... {chaofeng chen, yue. zhou (chou)ent.... entueenu..uuuu...... edu.sg
Jiaxing Xu, Yiping Ke CCDS, Nanyang Technological University jiaxing0030 ntu. edu.sg ypke@ntu.. edu.sg
ntu. edu sg
Xinjiang Wang, Litong Feng', WWyne Zhang SenseTime Research Iwangxinjiand feng] i itogg, wayne. 2hangg
ABSTRACT
Multimodal Large Language Models (MLLMs) have shown exceptional capabili- ties in vision-language tasks; however, effectively integrating image segmentation into these models remains a significant challenge. In this paper, we introduce Text4Seg, novel text-as-mask paradigm that casts image segmentation generation problem, eliminating the need for additional decoders and significantly simplifying the segmentation process. Our key innovation is semantic dexcriptors, a new textual representation of segmentation masks where each image patch is mapped its corresponding text label. This unified representation allows seam- less integration into the auto-regressive training pipeline of MLLMs for easier optimization. We demonstrate that representing an image with 16 16 semantic descriptors yields competitive segmentation performance. To enhance efficiency. we introduce the Row-wise Run-Length Encoding (R-RLE), which compresses redundant text sequences, reducing the length of semantic descriptors by 74% and accelerating inference by 3x, without compromising performance. Extensive ex- periments across various vision tasks, such referring expression segmentation and comprehension, show that Text4Seg achieves state-of-the-ar performance on multiple datasets by fine-tuning different MILM backbones. Our approach pro- vides an efficient, scalable solution for vision-centric tasks within the MLLM framework. https: /tttps://githieee cwm/com////oo/ooo Lan/Text 4Seg
text
INTRODUCTION
Multimodal Large Language Models (MLLMs) (Yin et al., 2023) have successfully extended the ca- pabilities of powerful Large Language Models (L.LMs) into the visual domain. Recent advancements demonstrate the remarkable ability of these models to engage in natural language-based human- computer interaction and text-based reasoning over visual inputs (Liu et al., 2024c; Lu et al., 2024; Liu et al., 2024a; Bai et al., 2023; Chen et al., 2024). MLLMs have emerged as powerful tools for vision-centric tasks, including image generation (Song et al., 2024; Wang et al., 2024d), object detection (Wang et al., 2024a; Ma al., 2024; Zhang et al., 2023) and semantic segmentation (Lai et al., 2024; Zhang et al., 2024b; Lan et al., 2024c). However, seamlessly integrating MLLMs with these tasks, particularly in dense prediction tasks like semantic segmentation, remains challenging due to the intrinsic differences between language and visual modalities.
A straightforward approach adopted by most existing works (Lai et al., 2024; Xia et al., 2024; Zhang et al., 2024b; He et al., 2024; Ren et al., 2024; Rasheed et al., 2024; Wang et al., 2024c; Zhang et al., 2023; Wu al., 2024) involves appending additional visual decoders (e.g. SAM (Kirillov al., 2023)) to MLLMs, as illustrated in Fig. ((a). While effective, this combination presents several limitations: complicates the end-to-end training pipeline with additional loss functions; 2) it re- quires careful modifications to MLLM architectures, leading to unexpected challenges when scaling
Coorreponndigg author.

---

Published as a conference paper at ICLR 2025
List + Lmask
Lixt
Liel
The black dog -880-
Decoder
The result is (o.x1, y1, x2, x2, x11,)1y111
Chen
Othe
Dackdog (leckckag...
MLLM
MLLM
MLLM
Please segment only the black dog image.
Please segrent the black dog in image.
Please segnent only the black
inage.
(a) embeddingg--a-mask
(b) polygon coordinates
(c) text-as-mask
Different paradigms of MLLMs based image segmentation: (a) embeddings-as-maat paradigm that relies on additional segmentation decoder and loss (e.g., LISA (Lai al., 2024));
Figure 1:
(b) polygon coordinates for instance segmentation (e.g., VisionL.MM (Wang et al., 2024a)); (c) our text-as-mask paradigm that relies on semantically consistent text sequences.
up the training. VisionLLM (Wang al., 2024a) attempts to convert segmentation masks into poly- gon coordinate sequences, shown in Fig. 1(b). However, the performance is often unsatisfactory, as LLMs may struggle to associate polygon coordinates with shapes, leading to the reintroduction of segmentation-sspecfii decoders in VisionLLMv2 (Jiannan et al., 2024). Finding more effective method to unlock the segmentation capabilities for MILMS remains crucial. Such method should adhere to the next-token prediction paradigm of MLLMs for easier optimization, require fewer chitectural changes for better scalability, and fully leverage text generation capabilities of LLMs.
In this paper, introduce novel text-as-mask paradigm that casts image segmentation a text generation problem, which significantly simplifies the segmentation process. We propose Text4Seg, decoder-free framework for MLLMs based image segmentation, as illustrated in Fig. (cc). Cen- tral to our method is a novel sequence representation of segmentation masks. Instead of using index masks or numerical coordinates, we map each flattened patch of the input image to its corresponding text description (e.g., semantic label, a short phrase, or a long sentence), forming a purely textual representation of images, named as semantic descriptors. This representation offers several advan- tages: 1) a unified sequence representation seamlessly integrated into the auto-regressive training pipeline, making joint optimization with text tasks easier; 2) no architectural changes are required, allowing full utilization of existing MLLM training infrastructure, making it ideal for scaling up; 3) support for large label vocabularies, equivalent to semantic words; and 4) flexible switching between referring expression segmentation, open- vocabulary segmentation, and other visual grounding tasks.
Inspired by VIT (Dosovitskiy et al., 2021), we demonstrate that representing an image with 16 x 16 semantic words, i.e., 256 length of semantic descriptors, is sufficient achieve satisfactory results. To improve efficiency, we introduce the Row-wise Run-Length Encoding (R-RLE), which com- presses the repeated descriptors within each image row while preserving the spatial structure. With- out compromising performance, R-RLE achieves a 74% reduction in semantic descriptors length and speeds up inference by 3x on average. To further enhance performance, we apply an off-the-shelf mask refiner, i.e., SAM, as a post-procsssing method to obtain pixel-level segmentation masks.
With the proposed semantic descriptors, training MLLMs for segmentation requires minimal ad- ditional effort. We begin by constructing instruction- -folowiinnnooooiingg data from existing segmentation datasets, transforming the vanilla semantic masks into the semantic descriptors format, and then fine-tuning the model using query-response conversations. This approach applies to a variety of vision-centria tasks, such as referring expression segmentation, open- vocabulary segmentation, and visual grounding tasks. Our experiments demonstrate that Text4Seg scamlessly integrate seg- mentation capabilities into existing MLLM architectures, such as LLaVA-I.5 (Li et al., 2024a), Qwen-V VL (Bai et al., 2023), DeepseekVL. (Lu et al., 2024), and Int (Chen et al., 2023b), without any architectural modifications. Without bells and whistles, Text4Seg consistently achieves superior comparable performance to previous models, highlighting its efficiency, flexibility, and robustness. In summary, our key contributions are as follows:
We propose Text4Seg, a novel text-as-mant as-akkk paradigm that redefines image segmentation as a text generation problem, fully leveraging the text generation capabilities of MI.LMs
2

---

Published as a conference paper at ICLR 2025
We introduce semantic descriptors, textual sequence representation of segmentation masks that seamlessly integrates with existing MLLMs for easier optimization. We demonstrate that 16 x 16 semantic descriptors are sufficient for achieving strong performance.
We develop Row-wise Run-Length Encoding (R-RLE) to compress semantic descriptors, signif- icantly reducing its length and inference costs without compromising performance.
We validate the effectiveness and robustness of Text4Seg based various MLLMs backbones
by achieving ofate-of-the-ar performance across various vision centric tasks.
2 RELATED WORK
Multimodal Large Language Models. MLLMs are typically developed by enhancing large lan- guage models (LLMs) with visual perception modules, which can generate coherent textual con- versations grounded in multimodal inputs. For instance, Flamingo (Alayrac et al., 2022) introduces the Perceiver Resampler, which connects pre-trained vision encoder with LLMs for effective few- shot learning. OpenFlamingo (Awadalla et al., 2023) and Otter (Li et al., 2023a) build upon this architecture with focus multi-modal in-context instruction tuning. BLIP-2 (Li et al., 2023b) and InstructBLIP (Dai et al., 2023) bridge the modality gap using a lightweight Querying Trans- former (Q-Former), demonstrating enhanced performance zero-shot lsion-to-language tasks. The LLaVA seires (Liu et al., 2024c;a) employs a linear layer or ML. as a modality connector, -muuuaeg-magge instruction-followin data generated with GPT-4, showcas- ing notable capabilities in multimodal chat interactions. They demonstrate impressive capabilities in multimodal chat interactions. Recent advancements (Liu al., 2024b; Xu al., 2024; Li al., 2024a;b;;; Lin et al., 2023) have focused on enhancing visual encoding through high-resolution in- puts. For example, LLaVA-UHD (Xu al., 2024) implements an image modularization strategy, segmenting native-resolution images into smaller, variable-sized slices to improve scalability and encoding efficiency. Similarly, LLaVA-NEXT (Liu et al., 2024b) and 2024a) utilize the AnyRes scheme to accommodate high-resolution image inputs. In this work, we present Text4Seg to endow existing MLLMs with image segmentation capabilities based on instruc- tion tuning, without necessitating any changes to their architecture
trained on multimodal
(Li al.,
Language-Guidee Guided Semantic Segmentation and Localization. Recent advancements have abled ML.LMs to incorporate task-specific modules for vision-centrie tasks. LISA (Lai et al., 2024) introduces the embedding-as-maate paradigm, utilizing a special <seg> token prompt segmen- tation mask decoder, such as SAM (Kirillov et al., 2023), thereby enhancing performance in reason- ing and referring expression segmentation. Building on this, GSVA (Xia al., 2024) employs mul- tiple <seg> tokens and <REJ> token to address cases where users reference multiple subjects or provide descriptions mismatched with image targets. Similarly, GLaMM (Rasheed et al., 2024) extends LISA's single-object focus by integrating natural language responses with corresponding object segmentation masks. They introduce a large-scale, densely annotated Grounding Dataset to train GLaMM, which significantly improves performance across various vision tasks. OMG-LLaVA (Zhang et al., 2024a) and PixeILM (Ren et al., 2024) are also capable of grounded conversation generation. PixeILM (Ren et al., 2024) advances LISA further by replacing SAM with a lightweight pixel decoder and introducing a comprehensive segmentation codebook for efficient multi-target reasoning and segmentation. In contrast, GROUNDHOG (Zhang et al., 2024b) proposes inputting visual entity tokens, rather than visual tokens, using their masked feature extractor, which enables fine-grained visual understanding. GROUNDHOG also curated grounded visual instruc- tion tuning dataset with Multi-Modal Multi-Grained Grounding, M3G2, to fully train the model. Recent studies (Zhang et al., 2023; Jiannan al.. 2024; Wu et al., 2024; Fei et al., 2024) extend MLLMs to vision-centric tasks like visual grounding (e.g., bounding boxes, masks) by integrating task-specific heads for different applications. While effective, these approaches increase training complexity and limit model scalability due to multiple decoders and loss functions. Other efforts (Chen et al., 2021; Peng et al., 2023; Wang et al., 2024a) have sought to simplify this process by learning coordinate sequences or location tokens. However, they tend to perform well only in object detection tasks with simple location coordinates, and struggle to achieve competitive results on more complex tasks such segmentation. In contrast, introduce general sequence representation for vision tasks without task-specific heads, enabling seamless integration with MLLMs and leveraging their text-generation capabilities for effective, versatile performance across applications.
3

---

Published as a conference paper at ICLR 2025
3 METHODOLOGY
3.1 PRELIMINARY
Multimodal Large Language Models (MLLMs) (Yin al., 2023) refer to the LLM-based models with the ability process, reason, and generate response from multimodal in- formation. Typically, as shown in Fig- 2, MLLM be abstracted into three main components: 1) a pre-trained vision encoder, which is responsible for extracting visual tokens from input images, 2) pre-trained large language model (LLM), which handles reasoning and generating out- puts, and 3) modality connector, which acts as bridge between the vision encoder and the LLM.
Response
LLM
Connector
Query
Vision
Encoder
Figure 2: MLLM architecture.
3.2 SEMANTIC DESCRIPTORS
Definition semantic descriptors. Our semantic descriptors are inspired by VIT (Dosovitskiy et al., 2021), which represents image as 16 x 16 visual tokens. As illustrated in Fig. 3, for simplicity, the example uses 6 x 6 visual tokens, the process begins by splitting the image into fixed-size patches and flattening them. Each patch is then represented by its corresponding semantic descriptor. A descriptor can be as simple as semantic label (e.g., "sky," "sand"), a phrase (e.g., "brown dog". "black dog"), or even intricate scenes. This approach encodes an image into a sequence of semantic descriptors of length 256, which meets the requirements for integrating image segmentation into MLLMs by:
complex textual description (e.g., "a dog in the left") for
Adhering the next-token prediction paradigm of MILMs, facilitating easier optimization.
Requiring architectural changes, ensuring seamless integration and scalability.
Adopting text-as-mask paradigm, using generation capabilities of LLMs for segmentation.
Image patches
Sementic descriptors
browndag blackdoo
Backdog sand Sennt seind, browndog
blackdog
send
Image-wise RLE
94, brown Backdoa brown
Beck
sand
browndogh
Blackdog '2 sand,
brownn
Blackdog 11
'2
Row-wise RLE
In
Blackdog
broundag
blncledon
Inisand
Blockediog
Sann
sane '2 browndog
Blackdog
soand 12'n
Figure 3: An illustration of semantic descriptors for images and two token compression techniques.
Row-wise RLE. One of the key limitations of full-length semantic descriptors the long token length due to the inherent spatial redundancy in images. For instance, the average token length of 256 semantic descriptors on the refCOCO (Kazemzadeh et al., 2014) dataset is 583, requiring ap- proximately 19s V100 GPU for single round of referring expression segmentation. To address this issue, we introduce the simple Run-Length Encoding (RLE) (Golomb, 1966) to compress the adjacent repeated texts in semantic descriptors.
A straight forward approach is to directly apply RLE to the whole semantic descriptors, referred as image-wise RLE (I-RLE). However, empirically found that it results in notable performance drop, suggesting that the compressed descriptors may lose crucial spatial information.
To mitigate this issue, propose novel Row-wise Run-Length Encoding (R-RLE) technique. As shown in Fig. 3, R-RLE operates at the row level, with each row separated by "\n"". This approach reduces the token length from 583 154 average while preserving spatial information.
4

---

Published as a conference paper at ICLR 2025
Query:
Car you segeent dog lo dovn the twage?
SAM Inferencca
The segnentatio nust for 'dog laring does is skown below tessegeatters *I6s others "16s where 11ii other 128 onhers "16a other "I6ls othon 120 others 160n wterr "Idle dog loying down *I others *15 dog lgying down *M] onhery "I2ia dog loytng down "*| ooherz "Is dog lising down '7] others "St other "18h olkes "Idix other "16wr-hay.
Respoase:
Semantic Descriptors
Traising
Lext
Query: Pleare segnent the isage and hightight behwa, grass"
Hive the orgurmiation muy forutiag on the seliva grase geggegggne *Idle triva *Y| grayy 12 zehra '3] gruss *yn zehra "6 gruse "En zetre 19 grass tis zolra *20 grass f6in zebra *10 gras 16ln aeber 1i2] grour 15x mobra "Iil goss tIn gras '6 nobon 12) grase *?] selva
Response:
MLLM
LoRA A
Please black
tii nit solve 55 *I] sebro *2) %oo "4 zelma "I] 15i *I] 13 gous
zelra
prous grass
image.
gouer 22] 12] gross grows
graer *5le grass "Nowrsieeg"
Figure 4: Visual instruction data.
Figure 5: Text4Seg.
Importantly, R-RLE demonstrates no performance degradation compared to the full-length semantic descriptors, and significantly enhances the inference speed.
3.3 VISUAL INSTRUCTION TUNING TEXT4SEG
Building on the proposed semantic descriptors, we construct visual instruction data by leveraging existing segmentation datasets. Fig. shows examples for referring expression segmentation and semantic segmentation. Given pair of <image, mask>, we resize the mask 16 x 16 resolution and flatten The indexes in the sequence are then replaced with their corresponding text labels to create full-length semantic descriptors. We further apply R-RLE to compress the sequence, with descriptors separated by "I" and rows separated by "\,n". Finally, the image, text labels, and semantic descriptors embedded into query-response template like
Query: <IMAGE> Can you segment the <text labels> in the image? Response: The result is :In <seg>semantis descriptors</Se /seg>.
Note that <seg>> and < /seq>> are start and end of semantic descriptors. With such pure text re- sponse, Text4Seg can be seamlessly integrated with existing MILMs without any architectural mod- ifications, as shown in Fig. 5. We use Low-Rand Adaptation (LoRA) (Hu et al., 2021), to fine-tune visual instruction data, using its original auto-regressive training objective Lex-- In contrast to existing models (Lai et al., 2024; Zhang et al., 2024b; Rasheed et al., 2024), which typically rely on Continued Pre-Training (CPT) with large, mixed datasets to fuse the architectures before fine-tuning specific downstream tasks, we apply Supervised Fine- Fine-Tuning (SFT) directly on the downstream tasks. During inference, the coarse masks generated by the MLLM demonstrate competitive performance compared to existing methods. To enhance the quality of pixel-level se- mantic masks, we optionally apply either Conditional Random Fields (CRF) (Krähenbühh & Koltun, 2011) the SAM as the mask refiner.
the MILMM
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Model architectures. Our method is built upon several open-source MILMSs. including LLaVA- 1.5 (Liu et al., 2024a), Deepseck (Lu et al., 2024), Intern VL2 (Chen et al., 2024), and Qwen-VL (Bai et al., 2023). The main experiments cover 6 MLMMs with model sizes ranging from 1.3B to 13B parameters, and 3 connectors, including MLP (LLaVA-1.5. Deepseek VL), Pixel Shuffle MLP (Intern and Cross-attention (Qwen-VLL All architectures were left unaltered during the experiments. Additionally, we employ the off-the-shelf SAM with ViT-H as our mask refiner.
Model training. Our method is implemented using SWIFT (Zhao et al., 2024). All models are trained on 8 Tesla A800 GPUs (40GB) with global batch size of 128. We use the AdamW opti- mizer (Loshchilov, 2017), starting with an initial learning rate of 2e-4, which follows a linear decay schedule after warm-up phase with ratio of 0.03. The weight decay is set to O, and gradient norms are clipped at 1.0. minimize GPU memory usage, we fine-tune all models using LoRA with rank of 64, along with ZeRO-2 stage memory optimization.
5

---

Published as a conference paper at ICLR 2025
Table 1: Referring Expression Segmentation results (cloU) on refCOCO (+/g) datasets
(Kazemzadeh et al., 2014; Mao et al., 2016). GLaMM is depicted in lighter color as it uses training dataset two orders of magnitude larger than ours. Model with CRF as the mask refiner. Model based on the x2x 32 semantic descriptors without the mask refiner.
Methods
LLM
refCOCO
TeCCOCO+
refCOCOg
Avg.
TestB
Specialised Segmentation Models
ReLA (Liu et al., 2023a)
73.8
76.5
70.2
66.0
71.0
57.7
65.0
66.0
68.3
HIPIE (Wang et al., 2024b)
78.3
66.2
69.8
PolyFormer-L (Liu et al., 2023b)
76.0
78.3
73.3
69.3
74.6
61.9
69.2
70.2
71.6
UNINEXT-L (Yan et ul., 2023)
80.3
82.6
77.8
70.0
74.9
62.6
73.4
73.7
74.4
Generalist
Segmentation Models (S88)
NEXT-Chat (Zhang 2023)
Vicuna-7B
74.7
78.9
69.5
65.1
71.9
56.7
67.0
67.0
68.9
LISA (1.ai et al., 2024)
Vicuna-71B
74.9
79.1
72.3
65.1
70.8
58.1
67.9
70.6
69.9
PixeILM (Ren al., 2024)
Vicuna-7B
73.0
76.5
68.2
66.3
71.7
58.3
69.3
70.5
69.2
AnyRef (Hl et al., 2024)
LLMMA2-7B
76.9
79.9
74.2
70.3
73,5
61.8
70.0
70.7
72.2
GSVA (Xia et al., 2024)
Vicuna-7B
77.2
78.9
73.5
65.9
69.6
59.8
72.7
73.3
71.4
LaSagnA (Wei ul., 2024)
Vicuna-7B
76.8
78.7
73.8
66,4
70.6
60.1
70.6
71.9
71.1
Groundhog (Zhang et al., 2024h)
LLaMA2-78
78.5
79.9
75.7
70.5
75.0
64.9
74.1
74.6
74.2
GLaMM Rasheed 2024)
Vieuna-7B
79.5
83.2
76.9
72.6
78.7
64.6
74.2
74.9
75.6
Text4Seg Ves-.as.VL.11388
DeepSeck-1.. 1.33
75.0
78.6
70.1
68.4
73.4
60.0
71.5
71.7
71.1
Text4Seg DespuedsVL.
DeepSeck-7B
72.6
74.8
70.0
67.2
71.5
62.2
69.1
69.4
69.6
Text4Seg D
DeepSeek-78
78.8
81.5
74.9
72.5
77.4
65.9
74.3
74.4
75.0
Text&Sog Qwen VL-
Qwen-7B
71.3
73.7
69.6
65.9
70.4
61.9
69.3
69.3
68.9
Text4Scg Qwcn-V1-78
Qwen-7B
78.0
80.9
74.6
71.6
77.3
66.0
74,8
74.7
74.7
Text4Seg
Vicuna-7B
73.2
75.7
71.4
67.0
71.9
62.4
67.3
68.9
69.7
Text4Seg 15-78
Vicuna-7B
79.3
81.9
76.2
72.1
77.6
66.1
72.1
73.9
74.9
Text4Scg ImscmV1...68
InterLLM2.5-78
73.0
75.2
70.7
67.6
72.1
62.6
68.9
70.3
70.1
Text 1m
IntemLLM2.55711
74.7
77.4
71.6
68.5
73.6
62.9
70.7
71.6
71.4
Text4Scg 1
M2.577L
79.2
81.7
75.6
72.8
77.9
66.5
74.0
75.3
75.4
Generalist Segmentation Models (T3B)
LISA (Lai el al., 2024)
Vicuma-1
76.0
78.8
72.9
65.0
70.2
58.1
69.5
70.5
70.1
GSVA (Xia 2024)
Vicuna- 13B
78.2
80.4
74.2
67.4
71.5
60.9
74.2
75.6
72.8
Text4SeguLava Text4Segruava.115
Vicuna- 13B
74.1
76.4
72.4
68.5
72.8
63.6
69.1
70.1
70.9
Vicuna- 13B
80.2
82.7
77.3
73.7
78.6
67.6
74.0
75.1
76.2
Table Generalized Referring Expression Segmentation results on the grefCOCO dataset (Liiu et al., 2023a). Model with CRF as the mask refiner. $ Model based on the 32x 32 semantic descriptors without the mask refiner.
Validation Set
Test Set
Test Set
Methods
LLM
Avg.
gloU
cloU
gloU
cloU
gloU
cloU
Specialised Segmentation Models
LAVT (Yang ct al., 2022)
58.4
57.6
65,9
65.3
55.8
55.0
59.7
ReLA (Liu et al., 2023a)
63.6
62.4
70.0
69.3
61.0
59.9
64.4
Generalist Segmentation Models
(S88)
LISA (Lai et al., 2024)
Vicuna-7B
61.6
61.8
66.3
68.5
58.8
60.6
62.9
GSVA (Xia al., 2024)
Vicuna-7B
66.5
63.3
71.1
69.9
62.2
60.5
65.6
Text4Seg DeepseckV1--1.39
DcepSeck-1.3B
69.9
63.2
69.7
67.5
62.3
59.8
65.4
Text4Seg DeepseekV1.-78
DeepSeck-7B
70.4
65.8
68.9
69.9
63.2
63.6
67.0
Text4Seg DeeeseekVV.-78
DeepSeek-7B
74.7
69.0
74.3
73.0
67.4
66.3
70.8
Text4Seg Qwen-V1-78
Qwen-7B
69.7
64.1
67.4
67.8
62.4
62.3
65.6
Text4Seg Qwen-VL--8
Qwen-7B
74.4
68.1
71.5
66.7
69.9
Text4Seg 1ext48cggua.v..1.-755 1.1-VA-1.5778
Vicuna-7B
69.1
64.7
69.9
70.8
62.1
62.3
66.5
1..
Vicuna-7B
73.6
67.9
74.1
72.8
66.1 1
64.8
69.9
Text4Seg JntemVL12-88
InternLM2.5-77
70.0
66.1
69.4
70.9
63.1
64.1
67.3
Text4Seg IntemV1.2-88
InternLM2.5-7
71.8
65.6
71.2
70.0
64.2
62.5
67.6
Text4Seg Jntem/11.-888
MnternL.2.5-775
74.4
69.1
75.1
73.8
67.3
66.6
71.1
Generalist Segmentation Models
(13B)
LISA (Lai et al., 2024)
Vicuna-13B
63.5
63.0
68.2
69.7
61.8
62.2
64.7
GSVA (Xia al., 2024)
Vicuna-
68.0
64.1
71.8
70.5
63.8
61.3
66.6
TexI4SegULLWA..5.IB
Vicuna-1 13B
70.3
66.9
69.8
71.4
63.8
64.4
67.8
Text4SeguL.v.11515
Vicuna-
74.8
69.8
75.1
74.3
68.0
67.1
71.5
4.2 REFERRING EXPRESSION SEGMENTATION
Settings. For referring expression segmentation (RES), we follow standard evaluation protocols (Lai et al., 2024; Xia et al., 2024) and assess method using the refCOCO series. We construct

---

Published as a conference paper at ICLR 2025
Table 3: Referring Expression Comprehension results (Acc@0.3) on RefcOco (+/g) datasets (Kazemzadeh et al., 2014; Mao et al., 2016). Model without the mask refiner.
refCOCO
refCOCO+
re/COCOg
Methods
LLM
Avg.
val
TestA
TesiB
val
IeSTA
TestB
val
Test
Specialised Segmentation Models
MDETR (Kumath al., 2021)
86.8
89.6
81.4
79.5
84.1
70.6
81.6
80.9
81.8
G-DINO (Liu al., 2023c)
90.6
93.2
88.2
82.8
89.0
75.9
86.1
87.0
86.6
PolyFormer-L, (Liu al., 2023b)
90.4
92.9
87.2
85.0
89.8
78.0
85.8
85.9
86.9
UNINEXT-L (Yan et al., 2023)
91.4
93.7
88.9
83.1
87.9
76.2
86.9
87.5
87.0
Generalist Segmentation Models (SBB]
Shikra (Chen al, 2023a)
Vicuna-7B
87.0
90.6
80.2
81.6
87.4
72.1
82.3
82.2
82.9
Ferret (You ct al., 2023)
Vicuna-7B
87.5
91.4
82.5
80.8
87.4
73.1
83.9
84.8
83.9
Qwen- VL (Bai al., 2023)
Qwen-7B
92.3
82.8
76.8
86.0
86.3
85.7
I1tem (Chen et al., 2024)
InternL.22.5785
87.1
91.1
80.7
79.8
87.9
71.4
82.7
82.7
82.9
LISA (lai et al. 2024)
Vicuma-74 7B
85.4
88.8
82.6
74.2
79.5
68.4
79.3
80.4
79.8
GSVA (Xia al., 2024)
Vicuna-7B
89.2
83.8
72.8
78.8
68.0
81.6
81.8
80.3
NEXT-Chat (Zhang et al., 2023)
Vicuna-74 7B
85.5
90.0
77.9
77.2
84.5
68.0
80.1
79.8
80.4
PixeLLM (Ren et al., 2024)
Vicuna-7B
89.8
92.2
86.4
83.2
87.0
78.9
84.6
86.0
86.0
Groma (Ma
2024)
Vicuna-7B
89.5
92.1
86.3
83.9
88.9
78.1
86.4
87.0
86.5
Text Despook91-11395
DeepSock- 1.3B
86.4
90.3
81.7
80.5
86.3
72.3
82.4
82.7
82.8
Text4Seg Dos-
DeepSeek-7B
90.8
83.4
82.1
76.8
81.1
81.0
83.8
Text4Seg Diepookk1-78
DeepSeck-7B
89.6
93.3
85.4
84.2
90.2
78.5
84.4
84.7
86.3
Text4Seg Qwes-VL.-78
Qwen-7B
87.2
90.1
83.6
82.1
87.4
76.6
81.5
81.3
83.7
Text4Scg quen VL.
Qwen-7B
89.7
93.0
85.8
84.6
90.1
78.6
85.0
85.1
86.5
Text4Seg 11.V-1..11
Vicuna-7B
89.2
92.0
86.4
83.4
88.6
78.0
81.7
82.4
85.2
Text&Seg LLaVA 1.577
Vicuna-7B
90.8
93.7
87.6
84.7
90.2
79.0
84.8
85.0
87.0
Text4Seg IneeaVVL2-4B
InternL.2.5571B
88.3
91.4
85.8
83.5
88.2
77.9
82.4
82.5
85.0
Text4Seg IneeVVL...0
InternL.M2.5778
90.3
93.4
87.5
85.2
89.9
79.5
85.4
85.4
87.1
Generalist Segmentation Models (13B)
Shikra (Chen LISA (Lai et al., 2024)
2023a)
Vicuna-13B
87.8
91.1
81.8
82.9
87.8
74.4
82.6
83.2
84.0
Vicuna-13B
85.9
89.1
83.2
74.9
81.1
68.9
80.1
81.5
80.6
GSVA (Xia et al., 2024)
Vicuna-131 13B
90.5
76.5
81.7
70.4
83.9
84.9
82.5
Text4Seg LLVV 138
Vicuna Meuna-1138
89.6
92.3
87.0
84.4
89.0
79.1
82.9
82.9
85.9
Text4Seg
Vicuna-131B
91.2
94.3
88.0
85.7
90.8
80.1
85.6
85.5
87.7
the referring segmentation dataset by combining the t iiainn n split of refCLEF, refCOCO, refCOCO+ (Kazemzadeh et al., 2014), and refCOCOg (Mao al., 2016), resulting dataset of 800k sam- ples. Our model is trained on this dataset for 5 epochs. Additionally, to evaluate the performance on a multi-objectnonnoojjee segmentation task, we construct a generalized referring expression seg- mentation dataset with 419k samples using the train split of grefCOCO (Liu et al., 2023a). We continue to fine-tune the model for 2 epochs.
Result of single object. As summarizze Tab. 1, our Text4Seg achieves the highest performance across all splits of the refCOCO (+/g) datasets. For 7B-scale MLLMs, Text4Seg DeepeekVi de- livers an impressive average cloU of 75.0, surpassing the closest competitor, Groundhog, which scores 74.2 cloU. Notably, Text4Seg 2. stands out with an average of 75.4 cloU. At the 13B parameter scale. Text4Seg LLa3A-1-5-13B achieves a marked improvement, with an average cloU of 76.2, significantly outperforming GSVA's 72.8 cloU. Even without using the SAM refiner, our method remains competitive. For instance, Text4Seg Vn refined with CRFs, and Text4Seg 10 based on 32 32 semantic descriptors, achieve results that rival or exceed existing methods.
Result of multi-/no object. As shown in Tab. 2, Text4Seg maintains competitive edge multi- object and no-object referring expression segmentation tasks. For instance, at the 7B scale, Text4Seg records average scores between 69.9 and 71.1, , a notable improvement over GSVA's 65.6 on the gRefCOCO dataset. At the 13B scale, Text4Seg LLaVA-1.5-138 further extends its lead, achieving an average score of 71.5, outperforming GSVA by 4.9 points. These outcomes highlight the robustness and versatility of Text4Seg in handling more complex segmentation challenges.
4.3 REFERRING EXPRESSION COMPREHENSION
Settings. Our Text4Seg can also be directly applied in object detection with simple mask2box paradigm, which first generates a segmentation mask based on the input and then derives the bound- ing box from the mask. We employ this method to evaluate the referring expression comprehension of our model using the same datasets as in RES. Specifically, a prediction is considered correct the loU between the predicted and ground truth bounding boxes exceeds 0.5.
7

---

Published as a conference paper at ICLR 2025
Table 4: Results on visual question answering and RES benchmarks. refC denotes refCOCO, Mix is combination of referring segmentation, semantic segmentation and VQA datasets from LISA.
VQA
RES (val)
Methods
Training Data
VQAv2
GQA
VisWiz
ScienceQA
TexiQA
POPE
reC
refC+
refCg
LISA
Mix
74.1
62.4
66.4
LLaVA-1.5
665k
78.0
61.7
50.6
68.4
55.0
85.4
Text4Seg
665k refseg
76.6
60.2
50.9
68.1
55.0
84.2
77.5
70.7
73.4
Results. As shown in Tab. 3, our Text4Seg achieves the best results on the refCOCO and re- fCOCO+ datasets, while Groma performs well refCOCOg. However, Text4Seg IntemVL2-88 delivers the highest overall accuracy, reaching 87.1%. Notably, both Text4Seg IntemVL2-8B and Text4Seg Qwen VL.7B surpass their respective MILMM baselines. In particular, Text4Seg ImemVL2 demonstrates a significant improvement over Intern VI.2-8B, increasing its average accuracy from 82.9% 87.1%. Additionally, our Text4Seg LLAVA-1.5-13 5-1333 outperforms previous SOTA, Shikra, by an average margin of 3.7%. It is worth noting that Text4Seg Lext LLaVA-1.5-7B and Text4Seg LLaVA-1 without a mask refiner, outperform their respective baseline counterparts. These results emphasize the superiority of Text4Seg in following instructions, leading to enhanced visual grounding ability.
4.4 VISUAL UNDERSTANDING
Settings. Our text-as-mask paradigm allows for seamless integration of downstream segmentation task into the pre-training of MLLMs. To evaluate its effectiveness, we assess the model's 's perfor- mance on various visual understanding benchmarks, using the LLaVA-1.5-77 model the bascline. Our method, Text4Seg, built upon the stage-2 of mix665k dataset and our referring segmentation datasets. For comprehensive comparison, we also report the performance of the LLaVA-1.5-78 model based on our implementation.
is trained on both the ALaVA-v1.5
Results. Table presents a comparison between LLaVA-1.5 and Text4Seg across various VQA and RES benchmarks. Notably, Text4Seg, trained on mixed dataset, achieves performance par with LLaVA-1.5 in visual question answering tasks while delivering strong results RES bench- marks. These results validate that our text generation based segmentation method acts seamless enhancement, offering a streamlined approach for pre- pre-training MLLMs. It successfully integrates robust segmentation functionality without compromising the model's conversational capabilities.
4.5 OPEN VOCABULARY SEGMENTATION
Settings. We follow LaSagnA (Wei et al., 2024) to eval- uate the performance of Text4Seg segmentation tasks. Our Text4Seg is built upon LLaVA- 1.5-7B and trained on the COCOStuff (Caesar al., 2018) for epoch. We evaluate the model's performance on ADE20K (A-150) (Zhou al., 2019), PASCAL Con- text 59 (PC-59) (Mottaghi et al., 2014), and PASCAL VOC 20 (PAS-20) (Everingham, 2009) datasets, using mloU as the evaluation metric.
Table 5: Open Vocabulary Segmenta- tion results (mloU) on various segmen- tation datasets.
apen-vocabulary
Methods
A-150
PC-59
PAS-20
Specialised Segmentation Models
ClearCLIP
16.7
35.9
80.9
ProxyCLIP
24.2
39.6
83.3
MaskCLIP
23.7
45.9
Group VIT
9.2
23.4
79.7
Results. As reported in the Tab. 5, it is expected that Text4Seg falls behind specialized segmentation models (e.g., ClearCLIP (Lan al., 2024a), ProxyCLIP (L.n et al., 2024b), MaskCLIP (Ding et al., 2022). GroupViT (Xu et al., 2022), OVSeg (Liang et al., 2023), and SAN (Xu et al., 2023)), because LLMs typically require quite large datasets to be sufficiently trained. However, Text4Seg still demonstrates competitive performance the PC-59 benchmark, underscoring its ef- ficiency. More importantly, significantly outperforms the MLLM-based LaSagnA, which uses an additional decoder, showcasing its strong potential for open-vocabulary segmentation.
OVSeg
24.8
53.3
92.6
SAN
27.5
53.8
94.0
Generalist Segmentation Models (7B)
LaSagnA
14.3
46.1
69.8
Text4Seg
16.5
52.5
76.5
8

---

Published as a conference paper at ICLR 2025
72
247
72.5
IeFCOCOg val
rePCOCOg teat
100
24
24
32
SAM
Figure 6: RES comparison across
Figure 7: Visualization of RES results across different resolu- tions, and with SAM as mask refiner.
different resolutions.
Table 6: Ablation study of mask refiner on refCOCO val.
Tolknns
Inference Time
Performance
20
80
HET
Count
400
Seconds
Method
Refiner
cloU
A.c@0.5
Time (s)
70
Text4Seg
None
73.5
89.3
5.34
200
65
Text4Seg SAM-B
75.5
89.9
5.54
HALE
RA
w/ RLE
ALE
RALE
weo RLE
60
LALE
RALE
w//o RLE
Text4Seg
SAM-L
79.1
90.6
5.73
Text4Seg
SAM-H
79.2
90.0
5.92
Figure 8: R-RLE is better than I-RLE.
4.6
ABLATION STUDY
Focusing on semantic descriptors for visual segmentation and grounding, we conducted ablation studies to evaluate its impact on performance using Intern VL2-8B (Chen et al., 2024) as the MLLM.
Resolution semantic descriptors. analyze the impact of varying the resolution of semantic descriptors RES performance, we create instruction-tunii datasets with different densities of semantic descriptors. Specifically, we represent each image with 16x 16, 24x24, and 32x32 se- mantic descriptors explore how finer coarser resolutions affect model accuracy. As shown in Fig. 6, the performance of Text4Seg without a mask refiner improves with higher resolution, from 67.5 cloU 162 to 71.4 cloU 32 average, surpassing LISA 69.9 cloU. Two examples illustrated in Fig. 7. Note that the improvement is achieved without increasing the feature resolution from the vision tower of MLLM. While higher-density semantic descriptors improve results, also significantly increases token length and computational cost. Therefore, we incorporate an off-the- shelf SAM refine the outputs. Experimental results show that using 162 semantic descriptors with SAM already achieves optimal performance.
Mask refiner with SAM variants. Tab. 6 compares the performance of various mask refiners, such SAM with different architectures, against refiner for semantic descriptors at a 16 x 16 resolution. SAM with the ViT-L architecture achieves similar performance to SAM with ViT-H while reducing inference time. Notably, Text4Seg with SAM-L increases the average performance on RES tasks from 73.5 to 79.1 cloU compared to Text4Seg without a mask refiner, with only a little increase in inference time.
I-RLE v.s. R-RLE. We investigate the impact of different encoding methods for semantic descrip- tors at a x resolution using the a Lnval splits the refCOCO and refCOCO+ datasets. As illustrated in Fig. 8, while full-length semantic descriptors achieve high performance, they suffer from significantly longer inference times (~ seconds) due to longer output tokens (~5590 on both datasets. Although the I-RLE method reduces both the number of tokens and inference time, it re- sults a notable performance drop, from 74.2 to 70.4 cloU on refCOCO and 68.0 to 64.7 cloU on refCOCO+. Our proposed R-RLE method strikes better balance, reducing the length of semantic descriptors by 74% and improving inference speed by an average of 3x, while still maintaining the same performance.

---

Published as a conference paper at ICLR 2025
GSVA
Text4Sig
young giraffe
ground pink hydrant
orange
green buy
person body
Figure 9: Visualizations Text4Seg and GSVA (Xia al., 2024) on the RES task. Our Text4Seg is based on Intern VL2 backbone. The corresponding referring expressions are displayed in the bottom.
GSVA
Text4eg
CT
Iaby wearing black shirt
the back with smoke
calcher and #1 batter
Iapanedd and partial
black in the mistlee black part body
holding black. umbretla (bot the umbrella
Figure 10: Visualizations of Text4Seg and GSVA (Xia et al., 2024) the GRES task.
4.7 VISUALIZATION EXAMPLES
We present qualitative comparisons between Text4Seg and GSVA in Figs. and 10. the single- object RES task, Text4Seg demonstrates a superior understanding of referring expressions, generat- ing more accurate and precise segmentation maps compared to GSVA. In the GRES task (Fig. 10), GSVA tends incorrectly segment empty objects despite the inclusion of a <REJ> token (as seen in the first two columns). In contrast, Text4Seg consistently avoids such mistakes by labeling them as "others" without special design. Furthermore, Text4Seg significantly outperforms GSVA in the multiple-object RES task, delivering more precise segmentation results with better grounding performance. These results fully validate the effectiveness of Text4Seg handling diverse and challenging visual grounding and segmentation tasks.
CONCLUSION
In this work, we present Text4Seg, a decoder-free framework that integrates seamlessly with exist- ing MLLMs for image segmentation using novel text-as-mask paradigm. With the novel semantic descriptors, Text4Seg achieves state-of-the-arr performance across various segmentation tasks, with- out requiring architecture modifications. We further introduce the Row-wise Run-Length Encoding (R-RLE) to compress semantic descriptors, which significantly improves the efficiency of Text4Seg while maintaining the performance. In summary, this work highlights the flexibility and effective- ness of Text4Seg in bridging the gap between MLLMs and vision-cent centric tasks, offering a scalable solution for future research multimodal learning.
10

---

Published as a conference paper at ICLR 2025
This study is supported under the RIE2020 Industry Alignment Fund Indus- try Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
Acknowledgment.
REFERENCES
Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.000 01300, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308. 2023. 5,
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209- 1218, 2018. 8
Keqin Chen, Zhao Zhang, Weili Zeng. Richong Zhang, Feng Zhu, and Rui Shioa Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint 1iXi::23006157955 2023a. 23
Ting Chen, Saurabh Saxena, Lala Li, David Fleet, and Geoffrey Hinton. Pix2seq: language modeling framework for object detection. arXiv preprint arXiv:2109.10852, 2021.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qing- long Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. In- ternvl: Scaling up vision foundation models and aligning for generic lingul-ii tasks. arXiv preprint arXiv:23 14238, 2023b. 2
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint 2rXiv:2404.1682 2024. 7,5,7,9
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In shirth-eeventh Conference Neural Information P Processing Systems, 2023. URL https:/
opw.reveeeeeeeeeeeoeeeeeee /tttps://openreviee
net/ forum?id=wvokk YZZJA. 3
Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary universal image segmentation with maskelip. arXiv preprint arXiv:2208. 08984, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: // //opnzeeview net wic.oummid.i.iic..yaaa.....a 2,
Mark Everingham. The pascal visual object classes challenge 2007. In http://www. pescal-network. aggchallnge///ccoee20///sshohogp
html, 2009. 8
Hao Fei, Shenggiong Wu, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Vitron: A unified pixel-level vision iim for understanding, generating, segmenting, editing, 2024.
Solomon Golomb. Run-length encodings (corresp.). IEEE transactions on information theory, 12 (3):3994401, 1966. 4
11

---

Published as a conference paper at ICLR 2025
Junwen He, Yifan Wang, Lijun Wang, Huchuan Lu, Jun Yan He, Jin-Peng Lan, Bin Luo, and Xuan- song Xie. Multi-modal instruction tuned Ilms with fine- fine-grained visual perception. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13980-13990, 2024. 1,6
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.cm 09685, 2021. 5
Wu Jiannan, Zhong Muyan, Xing Sen, Lai Zeqiang, Liu Zhaoyang, Chen Zhe, Wang Wenhai, Zhu Xizhou, Lu Lewei, Lu Tong, Luo Ping, Qiao Yu, and Dai Jifeng. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. arXiv preprint arXiv:2406.08394, 2024. 2,3
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1780-1790, 2021. 7
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014. 4, 6,
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023. 1,
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected erfs with gaussian edge potentials. Advances in neural information processing systems, 24, 2011.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579-9589, 2024. 1, 5, 6, 22, 23
Mengeheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Clearclip: Decomposing clip representations for dense vision-language inference. European Conference on Computer Vision, pp. 143-160. Springer, 2024a. 8
Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Prox- yclip: Proxy attention improves clip for apen-vocabuliy segmentation. In European Conference on Computer Vision, pp. 70-88. Springer, 2024b. 8
Mengcheng Lan, Xinjiang Wang, Yiping Ke, Jiaxing Xu, Litong Feng, and Wayne Zhang. Smooseg: smoothness prior for unsupervised semantic segmentation. Advances in Neural Information Pro- cessing Systems, 36, 2024c.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint xxXiv:2305.33726 2023a. 3
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.000 03326, 2024a. 3,3
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023b. 3
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modalle vision language models. arXiv preprint 2403 240...4844444 18814, 2024b.
12

---

Published as a conference paper at ICLR 2025
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang. Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi- modal models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26763-26773, 2024c. 3
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7061-7070, 2023. 8
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575, 2023.
Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 23592-2383 23601, 2023a. 6,7
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 16296-26300 2024a. 3,3,5
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Lexva-next: Improved reasoning, oct, and world knowledge, 2024b. 3
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024c. 1.3
Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and R Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2023b. 6,7
Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2023e. 7
I Loshchilov. Decoupled weight decay regularization. arXiv preprint
2017. 5
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vit vl: towards real-world vision-language understanding. arXiv preprint arXi:22003.05525, 2024. 2,
Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual tokenization for grounding multimodal large language models. arXiv preprint arXiv:2404. 13013, 2024. 1,
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan I. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 11 11-20, 2016. 6,7
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role context for object detection and semantic seg- mentation in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 891-898, 2014. 8
Seong-Whan
Zhiliang Peng, Wenhui Wang, Dong, Yaru Hao, Shaohan Huang. Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824. 2023.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming- Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13018, 2024. 1,
13

---

Published as a conference paper at ICLR 2025
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16374-26388 2024. 6,
Kunpeng Song, Yizhe Zhu, Bingchen Liu, Qing Yan, Ahmed Elgammal, and Xiao Yang. Moma: Multimodal Ilm adapter for fast personalized image generation. arXiv preprint arXiv:2404.05674 2024.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024a. 1,
Xudong Wang, Shufan Li, Konstantinos Kallidromitis, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. Hierarchical vecabblcyyylary universal image segmentation. Advances in Neural Infor- mation Processing Systems, 36, 2024b. 6
XuDong Wang,
Shaolun Zhang, Shufan Li, Konstantinos Kallidromitis, Kehan Li, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. Seglim: Multi-round reasoning segmentation. arXiv preprint arXiv:2410.1892. 18923, 2024c.
Zhenyu Wang. Aoxue Li, Zhenguo Li, and Xihui Liu. Genartist: Multimodal Ilm as an agent for unified image generation and editing. arXiv preprint arXiv:2407.05600, 2024d. 1
Cong Wei, Haoxian Tan, Yujie Zhong, Yujuu Yang, and Lin Ma. Lasagna: Language-bast based segmen- tation assistant for complex queries. arXiv preprint AXiv:2444008500 2024. 6, 8, 21
Shengqiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Towards semantic equivalence of tokenization in multimodal Ilm. arXiv preprint arXiv:4406.05127 2024. 1,3
Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Gen- cralized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3858-3888 3869,,96, 2024. 1,3, 6, 7, 10, 22, 23
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18134 18144, 2022. 8
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2945 2954, 2023.
Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an Imm perceiving any aspect ratio and high- resolution images. arXiv preprint 2rX 2024.
Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception object discovery and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15325-15336, 2023. 6,7
Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 18155-18165. 2022.
Shukang Yin. Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023. 1,4
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzii Du Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint 2rXiv:2310.07710 2023. 7
14

---

Published as a conference paper at ICLR 2025
Ao Zhang, Liming Zhao, Chen- Wei Xie, Yun Zheng, Wei Ji, and Tat-Seng Chua. Next-chat: An Imm for chat, detection and segmentation. arXiv preprint arXiv:2311.044939 2023. 1,3,6, 6,
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-Ilava: Bridging image-level, object-level, pixel-level reasoning and understanding. arXiv preprini arXiv:2406. /9389, 2024a. 3
Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. Groundhog: Grounding large language models to holistic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14227-14238, 2024b. 1, 1,3,66
Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swiftra scalable lightweight infrastructure for fine-tuning, 2024. URL https: / arxiv org/ abs /2008 05517. 5
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 22::3023221, 2019. 8
15

---

Published as a conference paper at ICLR 2025
A ADDITIONAL IMPLEMENTATION DETAILS
A.1 IMPLEMENTATION OF ADOPTING SAM AS MASK REFINER.
We employ SAM with ViT-H architecture our mask refiner. For referring expression segmen- lation tasks, we refine the coarse masks produced by Text4Seg from the semantic descriptors using the following process:
Step 1: Convert the binary mask into logit representation by applying the inverse sigmoid function.
Step 2: Randomly select 10 positive and 10 negative points from the coarse binary mask.
Step 3: Provide the selected points as point prompts, the logit representation as mask prompt, and the RGB image as input to SAM, generating refined mask and updated logits.
Step 4: Repeat Step twice.
This iterative process helps enhance the quality of the segmentation mask. The final mask produced by SAM is then resized to the original image dimensions, resulting in pixel-level segmentation masks. For open-vocan open-vocabulary segmentation, this strategy is applied iteratively across multiple class masks, which are then combined to form the final segmentation maps.
A.2 DETAILS OF TRAINING
Table 7 presents the training hyperparameters used for training Text4Seg on the referring expression segmentation task. We primarily adhere to the same settings as LLaVA-1.55 and these parameters are consistently applied across other tasks as well.
Table 7: Hyper-parameters and training settings for RES task.
Param Name
Value Adam W
Type Learning rate
2e-4
Weight decay (31,32)
0.0 (0.9, 0.95)
Optimizer
Gradient norm clip
1.0
Scheduler
Linearly decay
Warmup ratio
0.03
Rank
64
LoRA
Alpha (a:)
128
Dropout
0.05
Module
Linear layers of connector and LLMs
Trainable #Params.
About 2% of the LLM (7B - 160M)
Numerical precision
FP16
Global batch size
128
Training
Number of samples per epoch
800k
Total epochs
5
GPUs
A800(40G)
Time
About 2 Days
B COMPARISON OF TRAINING DATASETS
Most prior methods follow two-stage training paradigm: Continued Pre-Training (CPT) using large datasets, followed by Supervised Fine-Tuning (SFT) for specific tasks. The datasets used in these approaches are summarized in the following tables:
.Tbb 8: Datasets for Continued Pre-Training (CPT)
16

---

Published as a conference paper at ICLR 2025
Tab. 9: Datasets for Supervised Fine-Tuning (SFT) in Referring Expression Segmenta- tion (RES)
Tab. 10: Datasets for Supervised Fine-Tuning (SFT) in Generalized Referring Expres- sion Segmentation (GRES)
We can note that:
1. For CPT, previous methods rely heavily on large and diverse datasets, whereas our ap- proach, Text4Seg, eliminates this requirement, demonstrating superior efficiency and ef- fectiveness.
2. For SFT, we ensure a fair comparison by following previous works and train on:
The train split of refCOCO series for RES and REC tasks.
The train split of grefCOCO for the GRES task.
Table 8: Training datasets Continued Pre- Training (CPT).
Methods
Datasets
LISA
ADE20K, COCO-Stuff, PACO-LVIS, PartImageNet, PASCAL-Part, refCLEF,
refCOCO, refCOCO+, refCOCOg, 1..VA-vl.-mmi666k
PixeILM
ADE20K, COCO-Stuff, PACO-LVIS, refCLEF, refCOCO, reiCOCO+, ref-
COCOg. LLAVA-150K, multi-target reasoning segmentation (MUSE)
GSVA
ADE20K, COCO-Stuff, PACO-LVIS, Mapillary Vistas, PASCAL-PPar, ref-
CLEE, ref0COO,, refCOCOg, gRefCOCO, InnK., ReasonSeg
AnyRef
ADE20K, COCO-Stulf, PACO-LVIS, reİCLEF, refCOCo, refCOCO+, ref-
COCOg, PhraseCut, Flickr30K Entities, AVSBench
NEXT-Chat
Flickr30K Entities, Visual Genome, RefCOCO, RefCOCO+, RefCOCOg, VQAv2, PointQA, Visual7W, VCR, VA-taVV-nsttuet-Ii 150K, VG grounded cap- tioning, Shikra-RI)
Groundhog
Multi-Modal Multi-Grained Grounding dataset (M3G2): PNG, Flickr30K- Entity, refCLEF, refCOCO, refCOCO+, refCOCOg, gRefCOCO, PhraseCut, D-Cube, ReasonSeg, RIO, SK-VG, VizWiz-G, TexIVQA-X, GQA, VQS, Shikra-BinaryQA, EntityCount, FoodSeg-QA LVIS-QA, RefCOCO-REG, RefCOCO+ REGG RefCOCOg-REGG gRefCOCO REGG SG-Spo,,ap, V7W, PointQA, VCR, ShikraRD, SVIT-RD, Guesswhat, VG-RefMatch, HierText Grrounding-anythinn Dataset (GranD)): 11M images, 810M masks, 84M refer- ring expressions, GranD-f
GLaMM
Text4Seg
None
Table 9: Referring Expression Segmentation Datasets of Supervised Fine-Tuning (SFT). Other methods have already incorporated refCLEF dataset their CPT training datasets.
Methods
Datasets
LISA
refCOCO, reCO0O,,..CCCOO....cc..CCC.
PixeILM
None
GSVA
AnyRef
reCCOCO,, reCCOO0,.eCCOCOO,,eeelCCC..
NEXT-Chat
refCOCCO,eeCCOCOOteeeCOOC None
Groundhog
GLaMM
Text4Seg
refCOCO, reCoCO,,, refoCo0 refCLEF
C ADDITIONAL VISUAL INSTRUCTION DATA DETAILS
Query-answer template. We provide the question-answer templates in the Figs. 11 to 13. For partial segmentation tasks, the templates designed to segment only subset of objects in the
17

---

Published as a conference paper at ICLR 2025
Table 10:
Generalized Referring Expression Segmentation Datasets of Supervised Fine- Tine-Tuning
(SFT).
Methods
Datasets
LISA
greACOCO
GSVA
grefCOCO
Text4Seg
greCCoCO
image, such single object in the RES task, multiple objects in the GRES task, partial labels in semantic segmentation tasks. For conditioned segmentation tasks, the user provides a list of condition labels, and the model segments the entire image based those specified labels. For open- vocabulary segmentation tasks, the model leverages its open-vocabulary capabilities to segment the image and label all detected categories.
Visual instruction data on RES datasets. We adopt the question-answar templates from Fig. 11 construct the training data. Specifically, we iterate through all <image, referring expression, mask>> pairs in the dataset, transforming the vanilla mask into semantic descrip- tors, using the referring expression as descriptor. The referring expression placed in [class_name] placeholder within each question-answer template. The RES training set is con- structed by combining the train splits of refCLEF, refCOCO, refCOCO+, and refCOCOg, with the process repeated twice. This results in final RES training set comprising 800k samples. The same method is applied to construct the GRES training set, which contains 419k samples.
Visual instruction data on open-vocabulary segmentation datasets. For the open-vocabulary segmentation task, we utilize all three types of question-answer templates. Specifically, we construct visual instruction data using the COCOStuff dataset. The ratio of open-vocabulary segmentation templates, partial segmentation templates, and conditioned segmentation templates is set to 1 3 6. To further enhance diversity, we apply random cropping to both the image and mask. By iterating 10 times over the COCOStuff train set, we ultimately generate a training dataset consisting of 1.16M samples.
D ADDITIONAL QUANTITATIVE RESULTS
D. MORE RESULTS ON MASK REFINER
We present additional ablation study results on the mask refiner in Tab. 11, evaluated on the val split of the refCOCO(+/g) datasets. The findings indicate that both SAM with VIT-L and ViT-H architectures achieve similarly strong performance across all datasets, demonstrating the robustness of the mask refinement process regardless the test datasets.
Table 11: Ablation study on mask refiner on refCOCO (+/g) datasets.
refCOCO val
refCOCO+ val
refCOCOg val
Method
Refiner
cloU
Ac5@0.5
Time (s)
cloU Acc@0.5 Time (s)
cloU
Aco@0.5 Time (s)
Text4Seg
None
73.5
89.3
5.34
67.6
83.6
5.26
69.8
84.0
6.18
Text4Seg
SAM-B
75.5
89.9
5.54
69.8
84.7
5.46
71.3
84.6
6.30
Text4Seg
SAM-L
79.1
90.6
5.73
72.8
85.1
5,63
74.2
85.2
6.58
Text4Seg
SAM-H
79.3
90.0
5.92
72.6
84.3
5.84
74.6
85.6
6.75
D.2 MORE RESULTS ON DIFFERENT RESOLUTION OF SEMANTIC DESCRIPTORS
Figure 14 provides the complete results across all RES datasets, including refCOCO+. The results indicate that using a 16 x 16 length of semantic descriptors, combined with the SAM refiner, is an effective approach that delivers strong performance. While is possible eliminate the SAM refiner by further increasing the density of semantic descriptors, this would demand significantly higher computational resources, and will leave this optimization for future work.
18

---

Published as a conference paper at ICLR 2025
Question:
"Please segment only the [class name] in the image.",
"Can you segment the [class name] in the image?",
"Where is the [class name] in this picture? Please respond with segmentation mask.",
"Where is [[elass name]' in this image? Please output segmentation mask.",
"Could you provide the segmentation mask for '[class name]' in this image?",
"Please segment image and highlight [[class name]"."
Answer:
"Sure, here is the segmentation mask for [[laass name]':".
"Here is segmentation map focusing on the [class name]:",
"Here is the segmentation mask highlighting the [class name]:",
"The segmentation map for '[class name]' is:",
"The segmentation mask [class name]' is shown below:",
"Sure, Here's the segmentation of the [class name]:".
"Sure, the segmented output for [[class name]' is:",
"Certainly, the segmentation map for "[class name]' is:",
"Certainly, here is the segmentation mask for [claas name]]:",
"The segmentation mask [class name]' is shown below:"
Figure 11: Question-Ansver--eenllt for partial segmentation tasks, such as referring segmen-
tation and open vocabulary segmentation tasks. [class_name] will be replace with the referring expression in RES datasets or the selected class list in semantic segmentation datasets. The semantic descriptors are appended at the end of each answer.
D.3 MORE RESULTS REGARDING THE MASK REFINER
We provide additional guantitative results Tabs. 12 to 14. While Text4Seg without a mask refiner slightly lags behind LISA and GSVA in terms of average cloU on referring expression segmentation (RES) tasks, traditional mask refinement techniques, such Conditional Random Fields (CRF), can be employed to enhance segmentation accuracy. For instance, Text4Seg IntemV11...n with a CRF refiner improves the baseline performance from average cloU of 67.5 to 70.1 RES tasks. Additionally, when using 32 x 32 semantic descriptors, Text4Seg outperforms its counterpart with 16 16 descriptors. Specifically, Text4Seg IntemVL2-88 with 32 32 semantic descriptors achieves an average cloU of 71.4, surpassing LISA's 69.9 and matching GSVA's 71.4 on RES tasks. On the GRES tasks, as shown in the Tab. 13, both CRF and SAM refiners significantly enhance per- formance, outperforming LISA and GSVA. Notably, Text4Seg Intemm11.2.8 with 32 x 32 semantic descriptors, even without a mask refiner, achieves performance superior to existing methods. nally, the REC tasks, Text4Seg without SAM refiner continues outperform current methods, further demonstrating the effectiveness of Text4Seg's visual grounding capabilities.
E ADDITIONAL QUALITATIVE RESULTS
In this section, we provide more visual examples for different tasks to show the strong capabilities of the proposed Text4Seg.
Referring expression segmentation. Figure 15 provides additional examples of Text4Seg applied to the referring expression segmentation (RES) task. is evident that Text4Seg can segment objects based various criteria. including different classes (e.g., "clear glass"), colors (e.g., "blue"), and positions (e.g., "food in the back right"). This versatility demonstrates its superiority in accurately identifying and segmenting objects complex scenarios.
19

---

Published as a conference paper at ICLR 2025
Question:
"Please segment the image based on the category: [class name].",
"Segment the image according to the specified category: [class name].",
"Segment the image while focusing on the category: [class name].",
"Please provide segmentation map the category: [class name].",
"Segment the image with emphasis on the class: [class name].".
"Please segment the image, focusing on the candidate category: paasss__mm]"",
"Could you segment the image, considering the indicated class: [class name]?"
Answer:
"Sure, here the segmentation based the category '[class name]':",
"The image has been segmented according to the category [[class name]]:"
"Certainly, here is the segmentation map for the category '(class name]!:",
"The image is segmented with emphasis on the class '[class name]':"
"Here is the segmented image focusing the candidate category '[class name]!:",
"The image has been segmented with the category [[class name]' mind:",
"Sure, the segmentation mask is:",
"Sure, the segmented image is:",
"Certainly, the segmented map is:",
"Certainly, here is the segmentation mask:",
"Certainly, here is the segmented output:",
"Sure, here is the segmentation map:",
"The segmentation mask is shown below:"
Figure 12: Duestion-Ansser-Teempaae for conditioned segmentation tasks like open vocabulary segmentation task. cclass_nant will be replace with the condition class list in semantic segmen- tation datasets. The semantic descriptors are appended at the end of each answer.
Referring expression comprehension. We also present additional results the Referring Ex- pression Comprehension (REC) task in Fig. 16. is evident that the coarse masks generated by Text4Seg can be effectively utilized for object localization tasks using the simple mask2box method. This application highlights the accuracy of Text4Seg in referring object localization, demonstrating its capability to precisely identify and locate objects within complex images.
Open vocabulary semantic segmentation. Figure 17 presents additional examples of Text4Seg performing open-vocabulary segmentation. Notably, Text4Seg demonstrates ability to segment not only common large objects but also small objects effectively, such the person and boat on the river. This versatility highlights Text4Seg's proficiency accurately identifying and segmenting wide range of object sizes. Figure 18 illustrates the multi-object segmentation capabilities of Text4Seg. is evident that Text4Seg successfully segments all identified objects within the image, showcasing its strong ability handle multiple objects in complex scenarios. This performance highlights its robustness and effectiveness in accurately distinguishing various elements within single scene.
Visual understanding. Figure 19 presents example where Text4Seg is used for image caption- ing, single-object segmentation, and multi-object segmentation. Additionally, Fig. 20 compares the image reasoning capabilities of Text4Seg with the original LLa While maintaining similar reasoning abilities, our proposed Text4Seg extends functionality by enabling segmentation tasks.
20

---

Published as a conference paper at ICLR 2025
Question:
"Segment the entire image and classify cach category separately."
"Please perform segmentation this image and highlight all identifiable elements."
"Perform segmentation on this image and label all detected categories."
"Please identify and segment all categories present in the image."
"Segment the image and label all categories detected."
"Could you segment the image and label each identifiable category?"
"Segment the image to identify and label all visible categories."
"Segment and classify all elements in the image."
"Identify and segment all categories visible in the image."
"Can you segment and label the image?"
"Might you segment this image?"
"Can you perform segmentation on this image?"
"Could you please segment this image?"
Answer:
"Sure, here is the segmented image with each category classified separately:"
"Surc, here's 's the segmented image showing all visible categories:"
"The image is segmented and annotated with each category:"
"The image segmentation is complete, with all categories marked:"
"Sure, the segmentation mask
"Sure, the segmented image is:"
"Certainly, the segmented map is:"
"Certainly, here is the segmentation mask:"
"Certainly, here is the segmented output:"
"Sure, here is the segmentation map:"
"The segmentation mask is shown below:"
Figure 13: Question-Ansve-TTenllal for open vocabulary segmentation tasks. Following LaSagnA (Wei et al., 2024), the class label lists of the test benchmarks are given in the question for fair quantitative comparison. The semantic descriptors are appended at the end of each answer.
refCOCO val
refCOCO teatA
refCOCO tastB
refCOCo val
82.5
72
80.0
74
76
70
77.5
68
TereDes 5AN)
Tast55e SAMI
TedtS 5UP
Tuettty la
72.5
refCOCOg
refCOCOg
60
70
72
70
SAN Tex-DDe tack 5AM
locrddg
Tost55e WT 5AM
Tedt5e Ovo 5400
68.
Tuoldee laee SA0
242
322
Figure 14: Text4Seg with different resolutions of semantic descriptors on all RES datasets.
21

---

Published as a conference paper at ICLR 2025
Table 12:
Additional Referring Expression Segmentation results (cloU) on refCOCO (+/g)
datasets. Model is based on the semantic descriptors with a resolution of 32x32.
refCOCO
refCOCO+
reiCOCOg
Methods
Refiner
Avg.
val
testA
testB
val
testA
testB
val
test
Generalist Segmentation Models (S8B)
LISA (Lai et al., 2024)
74.9
79.1
72.3
65.1
70.8
58.1
67.9
70.6
69.9
GSVA (Xia et al., 2024)
77.2
78.9
73.5
65.9
69.6
59.8
72.7
73.3
71.4
Text4Seg Deepscck
None
66.2
68.7
63.6
60.7
64.5
54.9
64.2
64.2
63.4
Text4Seg DeepseekVI..I..B
SAM-H
75.0
78.6
70.1
68.4
73.4
60.0
71.5
71.7
71.1
Text4Seg DeepscckVL-7B
None
69.7
71.2
67.9
64.5
68.0
60.2
66.6
66.7
66.9
Text4Seg ReepseekVL.-77
CRF
72.6
74.8
70.0
67.2
71.5
62.2
69.1
69.4
69.6
Text4Seg DeepseekVL.778
SAM-H
78.8
81.5
74.9
72.5
77.4
65.9
74.3
74.4
75.0
Text4Seg Qwen-VL-7B
None
68.3
70.0
67.3
63.1
67.2
59.9
66.5
66.4
66.1
Text4Seg Qwen-VIL-78
CRF
71.3
73.7
69.6
65.9
70.4
61.9
69.3
69.3
68.9
Text4Seg Qwen-V1..7
SAM-H
78.0
80.9
74.6
71.6
77.3
66.0
74.8
74.7
74.7
1ext4Seeg11.v...5.777
None
70.5
72.3
69.3
64.4
68.7
60.6
65.1
66.5
67.2
Text4Seg 1..
CRF
73.2
75.7
71.4
67.0
71.9
62.4
67.3
68.9
69.7
LLaVA-1.5-7B LextA5eggLaVA+1.5-718
SAM-H
79.3
81.9
76.2
72.1
77.6
66.1
72.1
73.9
74.9
Text4Seg InterVLL2-8BB
None
70.3
71.9
68.7
65.0
68.9
60.8
66.7
67.6
67.5
Text4Seg InternVL.2-BB
CRF
73.0
75.2
70.7
67.6
72.1
62.6
68.9
70.3
70.1
Text4Seg InternVTL2-8B
SAM-H
79.2
81.7
75.6
72.8
77.9
66.5
74.0
75.3
75.4
Text4Seg LLternVL2-8B'
None
74.7
77.4
71.6
68.5
73.6
62.9
70.7
71.6
71.4
Text4Seg IntemVVL.2.5 8B
SAM-H
78.6
81.7
74.3
71.8
77.4
65.1
73.9
74.7
74.7
Generalist Segmentation Models (13B)
LISA (Lai et al., 2024)
76.0
78.8
72.9
65.0
70.2
58.1
69.5
70.5
70.1
GSVA (Xia et al., 2024)
78.2
80.4
74.2
67.4
71.5
60.9
74.2
75.6
72.8
Text4Seg LLaVA-1.5-13BB
None
71.3
72.9
70.3
65.9
70.0
61.8
66.8
67.6
68.3
Text4Seg
1..AA- 11avA 1.5--3B
CRF
74.1
76.4
72.4
68.5
72.8
63.6
69.1
70.1
70.9
Text4Seg LLaVA 1.5-13B
SAM-H
80.2
82.7
77.3
73.7
78.6
67.6
74.0
75.1
76.2
22

---

Published as a conference paper at ICLR 2025
Table 13: Additional Generalized Referring Expression Segmentation results on gre(COCo dataset. Model is based on the semantic descriptors with resolution of 32x32.
Validation Set
Test Set A
Test Set B
Methods
Refiner
Avg.
gloU
cloU
gloU
cloU
gloU
cloU
Generalist Segmentation Models (S8B)
LISA (Lai et al., 2024)
61.6
61.8
66.3
68.5
58.8
60.6
62.9
GSVA (Xia al., 2024)
66.5
63.3
71.1
69.9
62.2
60.5
65.6
Text4Seg DeepscekVL-1.38
None
64.3
57.2
62.2
61.2
57.1
54,9
59.5
Text4Seg DespockVLL-1.38
SAM-H
69.9
63.2
69.7
67.5
62.3
59.8
65.4
Text4Seg DeqpscckLL-78
None
69.0
62.7
66.3
65.9
62.1
61.1
64.5
Text4Seg DeepseekVVI.-23
CRF
70.4
65.8
68.9
69.9
63.2
63.6
67.0
Text4Seg DeepscckVL-778
SAM-H
74.7
69.0
74.3
73.0
67.4
66.3
70.8
Text4Seg Qwen-VV--78
None
68.5
61.1
64.6
63.6
61.1
59.6
63.1
Text4Seg Qwen VL.-BB
CRF
69.7
64.1
67.4
67.8
62.4
62.3
65.6
Text4Seg Qwcn-VL--88 V
SAM-H
74.4
68.1
73.1
71.5
66.7
65.3
69.9
Tex
None
67.9
61.6
66.2
65.9
60.9
59.8
63.7
LLVV--5-17
CRF
69.1
64.7
69.9
70.8
62.1
62.3
66.5
Text4Seg
SAM-H
73.6
67.9
74.1
72.8
66.1
64.8
69.9
Text4Seg ImtemLL2-8BB
None
68.8
63.1
66.9
67.1
62.1
61.6
64.9
Text4Seg IntemVVL.--8
CRF
70.0
66.1
69.4
70.9
63.1
64.1
67.3
Text4Seg ImterVLL2-8BB
SAM-H
74.4
69.1
75.1
73.8
67.3
66.6
71.1
Text4Seg IL2-5VL
None
71.8
65.6
71.2
70.0
64.2
62.5
67.6
Text4Seg IntemV11.2.88
SAM-H
74.9
68.8
75.4
73.6
67.0
65.1
70.8
Generahit Segmentation Models (13B)
LISA (Lai et al., 2024)
63.5
63.0
68.2
69.7
61.8
62.2
64.7
GSVA (Xia al., 2024)
68.0
64.1
71.8
70.5
63.8
61.3
66.6
Tex
None
69.2
63.9
67.4
67.6
62.7
62.0
65.5
Text4Seg LL.VA-1.--11
CRF
70.3
66.9
69.8
71.4
63.8
64.4
67.8
Text4Seg LLAVA-LLLLL
SAM-H
74.8
69.8
75.1
74.3
68.0
67.1
71.5
Table 14: Additional Referring Expression Comprehension results (Acc@0.5) on RefCOCO (+/g) datasets. Model is based the semantic descriptors with resolution of 32x32.
refCOCO
refCOCO+
refCOCOg test
Methods
Refiner
Avg.
testA
testBB
testA
testB
Generalist
Segmentation Models ( (<8B)
LISA (Lai et al., 2024)
85.4
88.8
82.6
74.2
79.5
68.4
79.3
80.4
79.8
GSVA (Xia al., 2024)
86.3
89,2
83.8
72.8
78.8
68.0
81.6
81.8
80.3
Text4Seg DeepuockVL-1.3B
None
83.6
87.3
79.1
78.0
83.6
70.3
78.5
78.8
79.9
Text4Seg DeepsecVV--1 VL-1133
SAM-H
86.4
90.3
81.7
80.5
86.3
72.3
82.4
82.7
82.8
Tex14Seg DeepsockV1--78
None
87.2
90.8
83.4
82.1
88.1
76.8
81.1
81.0
83.8
Text4Seg Deepuock
SAM-H
89.6
93.3
85.4
84.2
90.2
78.5
84.4
84.7
86.3
Text4Seg Qwen-V1--78
None
87.2
90.1
83.6
82.1
87.4
76.6
81.5
81.3
83.7
Text4Seg Qwen-V1-78
SAM-H
89.7
93.0
85.8
84.6
90.1
78.6
85.0
85.1
86.5
Text4Seg 1.V41.1g.1.
None
89.2
92.0
86.4
83.4
88.6
78.0
81.7
82.4
85.2
Text4Seg 1.777-1.-78
SAM-H
90.8
93.7
87.6
84.7
90.2
79.0
85.0
87.0
Text4Seg IntemmV1..888
None
88.3
91.4
85.8
83.5
88.2
77.9
82.4
82.5
85.0
Text4Seg LntemVLL--BB
SAM-H
90.3
93.4
87.5
85.2
89.9
79.5
85.4
85.4
87.1
Text4Seg IntemVLL2-BB"
None
88.9
92.4
84.1
83.1
88.6
77.3
83.6
83.8
85.2
Text4Seg 222.8
SAM-H
89.6
92.6
84.9
83.7
88.8
77.6
84.6
84.8
85.8
Generalist Segmentation Models
(13B)
Shikra (Chen et al., 2023a)
Vicuna-13B
87.8
91.1
81.8
82.9
87.8
74.4
82.6
83.2
84.0
LISA (Lai et al., 2024)
85.9
89,1
83.2
74.9
81.1
68.9
80.1
81.5
80.6
GSVA (Xia et al., 2024)
87.7
90.5
84.6
76.5
81.7
70.4
83.9
84,9
82.5
Text4SeguL.vA-..
None
89.6
92.3
87.0
84.4
89.0
79.
82.9
82.9
85.9
Texu4SeguL.v.1..
SAM-H
91.2
94.3
88.0
85.7
90.8
80.1
85.6
85.5
87.7
23

---

Published as a conference paper at ICLR 2025
clear glass
bike farthest right
right
bottom right
für right surfboard
chair right
yellow bike
washing dishes
pile green food closest the end of the bread
black suitcase right that is closest
upper right tv
Example results of Text4Seg referring expression segmentation task. The referring phrases are below the images.
Figure 15:
Police
blue polo white hair
brown stationwagen
baby elephant
hotdog right top
laptop the left
guy the far left
Figure 16: Example results of Text4Seg referring expression comprehension task. Blue boxes are ground truth labels, and green ones are the Text4Seg predictions.
24

---

Published as a conference paper at ICLR 2025
Query: Can you segnent the aeroplane, bicycle, bird, boat, bottle, bus, car, cat, chair, соw, dining table, dog, horse, motorbike, person, potted plant, sheep, sofa, train, TV in the image?
Response: sure, the segnented mask 1s:
Figure 17: Example results of open-vocay segmentation using Text4Seg the PAS-20 bench- mark.
25

---

Published as a conference paper at ICLR 2025
Query: you segnent
inage?
Response: Sure, the segmented mask 1s:
The
OYOT
ohall
pereon
wincow:
Figure 18: Example results of vocabuacy segmentation using Text4Seg on the PC-59 bench- mark.
26

---

Published as a conference paper at ICLR 2025
Descript this image in detail.
oo
The image depicts group of wild horses galloping through grassy field. There are ten horses total, with some running closer to the foreground and others further back in the scene. The horses are spread out across the field, with some closer to the left side, others in the middle, and a few more towards the right side of the image. The horses appear be enjoying their time running freely in the open field.
Might you segment the white horse in this image?
0
Sure, the segmented output for 'white horse' is:
g
Please segment all horses in the image.
The segmentation mask for 'horses' is shown below:
Figure 19: An example result of Text4Seg to perform image captioning, single-object segmentation, and multi-object segmentation.
27

---

Published as a conference paper at ICLR 2025
What is unusual about this image?
The unusual aspect of this image is that man is standing the back of moving yellow car, holding a clothes dryer. This is not typical scene, as Text4Seg people usually do not stand the back of moving vehicle, especially not on top of a car with clothes dryer. The man's actions are both dangerous and unconventional, he could risk falling off the car or causing damage the vehicle or himself.
LLaVA1.5:
The unusual aspect of this image is that is standing clothesline mounted on the back of moving yellow car. This is not a common sight, people typically do use cars as platform for hanging clothes or performing other tasks. The man is also wearing yellow shirt, which adds to the unconventional nature of the scene. The presence a taxi the background further emphasizes the unusual nature of the image, it is not typical setting for such an activity
Could you please segment this man?
Sure, here is the segmentation mask for 'man':
Text4Seg
LLaVA1.5:
0
Figure 20: The capability comparison between Text4Seg and -L..5-
28


Reviews:
[{"review_id": "3XzbRXGzIi", "reviewer": "Reviewer_9zHz", "summary": "The paper introduces TextSeg, a novel approach that integrates image segmentation into Multimodal Large Language Models (MLLMs) by framing segmentation as a text generation task.  The key innovation is the use of semantic descriptors, where each image patch is mapped to a textual label, allowing seamless integration into the auto-regressive training pipeline of MLLMs. By representing images with 16×16 semantic descriptors, the approach achieves competitive segmentation performance. To enhance efficiency, the authors propose Row-wise Run-Length Encoding (R-RLE), which compresses redundant text sequences by 74% and speeds up inference by three times without sacrificing accuracy. Extensive experiments demonstrate that TextSeg attains state-of-the-art results on multiple datasets across various vision tasks.", "questions": "+ Training Datasets: The authors should clearly mention about the training datasets, additional training datasets for fine-tuning (if accapable) of other methods for a fair comparison. Because we know that with more training datasets (GLaMM, Groundhog), and with fine-tuning on task-specific datasets, we can achieve a better performance. \n+ Use of SAM Mask Refiner: The proposed TextSeg method incorporates a post-processing step using the SAM mask refiner. As indicated in Table 6, without SAM, the method achieves a cIoU of 73.5, which is lower than most methods in the Generalist Segmentation Models (~7B) category in Table 1. This raises concerns about the fairness of the comparison, as other methods may not use similar post-processing techniques. The authors should clarify how other methods could benefit from similar enhancements.\n+ Inference Speed: It would be beneficial to include the inference speed of TextSeg compared to other methods. Providing quantitative metrics on inference time would strengthen the evaluation.\n+ Missing Recent Specialized Segmentation Models: The paper lacks comparisons with recent specialized segmentation models such as PolyFormer [1], UNINEXT [2], and HIPIE [3]. Including these models in the evaluation would provide a more comprehensive assessment of the proposed method's performance relative to the current state-of-the-art.\n\n[1] Liu et al., PolyFormer: Referring image segmentation as sequential polygon generation, CVPR, 2023.\n\n[2] Yan et al., Universal instance perception as object discovery and retrieval, CVPR, 2023.\n\n[3] Wang et al., Hierarchical open-vocabulary universal image segmentation, NeurIPS, 2024.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The authors have conducted extensive experiments and ablation studies to demonstrate the effectiveness of their proposed method.", "weaknesses": "The paper presents a method that integrates image segmentation into MLLMs by introducing semantic descriptors and utilizing a SAM mask refiner. While the approach simplifies the segmentation process by treating it as a text generation task, the technical contributions appear to be more incremental and engineering-oriented. The method essentially adapts existing MLLMs with semantic descriptors to perform segmentation tasks, serving as a baseline framework that can be applied to different MLLM models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XECQp6T2Uc", "reviewer": "Reviewer_SUs4", "summary": "This paper proposed TextSeg, a novel text-as-mask paradigm that casts image segmentation as a text generation problem, eliminating the need for additional decoders. It employs semantic descriptors as a new textual representation of segmentation masks where each image patch is mapped to its corresponding text label. It further compresses predicted token length by using Row-wise Run-Length Encoding to handle contiguous region of same semantic region.", "questions": "- please discuss how to extend to instance level segmentation\n- the name of TextSeg was already used: https://paperswithcode.com/dataset/textseg", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- model pixel label as semantic text token by generalized VLLM model\n- compress token length with row-wise run-length encoding\n- comprehensive results on referring express segmentation and comprehension", "weaknesses": "- idea is simple and straightforward\n- relies on SAM to get the final pixel level prediction from patch level semantic text token prediction\n- should compare with other VLLM approach also with SAM refinement", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "08cMTjugkK", "reviewer": "Reviewer_stRB", "summary": "This paper proposes a new scheme for adapting MLLMs to segmentation by reformulating the mask prediction as a sequence of text labels called *semantic descriptors*. Specifically, the MLLM takes as input an image and query and outputs a mask prediction, represented as a sequence of patch-wise semantic descriptors, consisting of <SEG>, 'others', and the queried target label: <SEG> is a special token that indicates the start and end of the mask, 'others' suggests a background patch, and the target label suggests a target patch. Such formulation allows the MLLM to predict the segmentation task as a sequence of text labels, instead of coordinates or special segmentation tokens, maintaining its original auto-regressive training objective and thereby facilitating easier optimization. Furthermore, the authors propose *Row-wise Run-Length Encoding (R-RLE)* that compresses adjacent repeated tokens in the mask prediction to reduce the number of tokens by 74% and triple the inference speed. The framework, names TextSeg, acheives strong performance on locating tasks like RES, gRES, and REC, showing the location ability of TextSeg, and acheives comparable performance to its MLLM backbone on VQA, suggesting that the MLLM's conversational ability is maintained.", "questions": "- In Figure 6, higher resolution patch prompts to SAM seems to result in worse segmentation results, and this is counter-intuitive as more finegrained the prompt, more finegrained the segmentation should be. Could the authors provide an explanation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality \n- Authors propose a new formulation of mask predictions by MLLMs. Different from previous work that represent masks as a special <SEG> token or coordinates, this work formulates the mask as a sequence of text labels (called *semantic descriptors*) of 'others' and the queried target label. Such formulation allows the MLLM to be trained for segmentation with the LLM's original autoregressive training objective, allowing easier optimization and the maintanence of the architecture. Although the idea to represent a mask as a sequence of texts itself is not unimaginable, the realization of it is original and novel. \n- Authors also propose Row-wise RLE to effectively compress the sequence of semantic descriptors and thereby eliminate any computation burden caused by their new formulation. Although, RLE is an existing technique, the authors effectively adapt it to their configuration, acquiring novelty. Instead of simply proposing a new formulation that would be computationlly heavy by itself, the authors further propose a technique that eradicates this issue, making the formulation applicable and thereby more complete. \n\nQuality\n - Authors carry out an extensive experiment to check whether their new formulation is generalizable to various MLLMs by using four MLLM backbones: LLaVA-1.5 (Li et al., 2024a), Qwen-VL (Bai et al., 2023), DeepseekVL (Lu et al., 2024), and InternVL2 (Chen et al., 2023b).\n- Authors also experiment on various tasks (RES, gRES, REC, and Open Vocabulary Segmentation), showing that TextSeg acquires strong performance on various grounding tasks. \n- Authors also show that the conversational ability of the LLM does not diminish with the mask prediction finetuning, which is crucial for MLLMs to be useful as its their main capability. \n\nClarity\n - The paper explains its method and contribution well in overall. \n\nSignificance\n - The paper provides a new formulation of mask prediction by LLMs to the research area of adapting MLLMs to grounding tasks, which is of significant interest nowadays, and thus contributes to the community.", "weaknesses": "- Although authors stress as a main advantage of their mask formulation as not requiring a segmentation decoder throughout their paper, TextSeg uses SAM to acquire performance comparable to other generalist segmentation models. More precisely, TextSeg *does* require a segmentation decoder but *does not* require finetuning it. Thus, instead of saying that TextSeg is a decoder-free framework, authors should explain that it is a decoder-training-free framework. \n- Since the authors say that SAM is optional, the results without SAM should be included in all the main table results, not just in the ablation study. \n\n- Addressing the points below would make the paper clearer:\n  - Is the row length in R-RLE 16? Then, is Fig. 3 a smaller version with row length 6? The discrepancy between Fig. 3 and the textual explanation is a bit confusing. \n  - Add backbone column to main results table (ie. Table 1~3), so that direct comparison between TextSeg-some-backbone and another method using that backbone is easier to do. \n  - Add references of datasets/benchmarks in tables.\n  - Explain what 'mix' under 'Training Data' of LISA means in the caption of Table 4. \n  - References from the end of page 6 to halfway of page 8 seem to be missing. \n  - Explain what training scheme was used for the ablation study: was TextSeg trained on the same combined RES datasets?\n  - Typo in \"Notably, TextSeg with ViT-L increases the average performance on RES tasks from 70.3 to 75.4 cIoU compared to TextSeg without a mask refiner, with only a little increase in inference time.\" Should ViT-L be SAM-L? Also, the numbers 70.3 and 75.4 are not in Tab. 6. \n  - Performance of 79.3 cIoU of TextSeg with SAM-H in Table 6 is different from performance of 79.2 cIoU in Table 1.\n  - Mark best performances in bold in Table 5 (as done in other tables).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)