Text4Seg: Reimagining Image Segmentation as Text Generation

Authors: Mengcheng Lan, Chaofeng Chen, Yue Zhou, Jiaxing Xu, Yiping Ke, Xinjiang Wang, Litong Feng, Wayne Zhang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Multimodal large language model, Image segmentation, Referring expression segmentation

Abstract:
Multimodal Large Language Models (MLLMs) have shown exceptional capabilities in vision-language tasks; however, effectively integrating image segmentation into these models remains a significant challenge. In this paper, we introduce Text4Seg, a novel text-as-mask paradigm that casts image segmentation as a text generation problem, eliminating the need for additional decoders and significantly simplifying the segmentation process. Our key innovation is semantic descriptors, a new textual representation of segmentation masks where each image patch is mapped to its corresponding text label. This unified representation allows seamless integration into the auto-regressive training pipeline of MLLMs for easier optimization. We demonstrate that representing an image with $16\times16$ semantic descriptors yields competitive segmentation performance. To enhance efficiency, we introduce the Row-wise Run-Length Encoding (R-RLE), which compresses redundant text sequences, reducing the length of semantic descriptors by 74\% and accelerating inference by $3\times$, without compromising performance. Extensive experiments across various vision tasks, such as referring expression segmentation and comprehension, show that Text4Seg achieves state-of-the-art performance on multiple datasets by fine-tuning different MLLM backbones. Our approach provides an efficient, scalable solution for vision-centric tasks within the MLLM framework.

Decision: accept (poster)
Reviewer scores: 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1730438835539

thread_type:
official_review

<thread_object>
creator: reviewer_strb

<content>
summary:
This paper proposes a new scheme for adapting MLLMs to segmentation by reformulating the mask prediction as a sequence of text labels called *semantic descriptors*. Specifically, the MLLM takes as input an image and query and outputs a mask prediction, represented as a sequence of patch-wise semantic descriptors, consisting of <SEG>, 'others', and the queried target label: <SEG> is a special token that indicates the start and end of the mask, 'others' suggests a background patch, and the target label suggests a target patch. Such formulation allows the MLLM to predict the segmentation task as a sequence of text labels, instead of coordinates or special segmentation tokens, maintaining its original auto-regressive training objective and thereby facilitating easier optimization. Furthermore, the authors propose *Row-wise Run-Length Encoding (R-RLE)* that compresses adjacent repeated tokens in the mask prediction to reduce the number of tokens by 74% and triple the inference speed. The framework, names TextSeg, acheives strong performance on locating tasks like RES, gRES, and REC, showing the location ability of TextSeg, and acheives comparable performance to its MLLM backbone on VQA, suggesting that the MLLM's conversational ability is maintained.

strengths:
Originality 
- Authors propose a new formulation of mask predictions by MLLMs. Different from previous work that represent masks as a special <SEG> token or coordinates, this work formulates the mask as a sequence of text labels (called *semantic descriptors*) of 'others' and the queried target label. Such formulation allows the MLLM to be trained for segmentation with the LLM's original autoregressive training objective, allowing easier optimization and the maintanence of the architecture. Although the idea to represent a mask as a sequence of texts itself is not unimaginable, the realization of it is original and novel. 
- Authors also propose Row-wise RLE to effectively compress the sequence of semantic descriptors and thereby eliminate any computation burden caused by their new formulation. Although, RLE is an existing technique, the authors effectively adapt it to their configuration, acquiring novelty. Instead of simply proposing a new formulation that would be computationlly heavy by itself, the authors further propose a technique that eradicates this issue, making the formulation applicable and thereby more complete. 

Quality
 - Authors carry out an extensive experiment to check whether their new formulation is generalizable to various MLLMs by using four MLLM backbones: LLaVA-1.5 (Li et al., 2024a), Qwen-VL (Bai et al., 2023), DeepseekVL (Lu et al., 2024), and InternVL2 (Chen et al., 2023b).
- Authors also experiment on various tasks (RES, gRES, REC, and Open Vocabulary Segmentation), showing that TextSeg acquires strong performance on various grounding tasks. 
- Authors also show that the conversational ability of the LLM does not diminish with the mask prediction finetuning, which is crucial for MLLMs to be useful as its their main capability. 

Clarity
 - The paper explains its method and contribution well in overall. 

Significance
 - The paper provides a new formulation of mask prediction by LLMs to the research area of adapting MLLMs to grounding tasks, which is of significant interest nowadays, and thus contributes to the community.

weaknesses:
- Although authors stress as a main advantage of their mask formulation as not requiring a segmentation decoder throughout their paper, TextSeg uses SAM to acquire performance comparable to other generalist segmentation models. More precisely, TextSeg *does* require a segmentation decoder but *does not* require finetuning it. Thus, instead of saying that TextSeg is a decoder-free framework, authors should explain that it is a decoder-training-free framework. 
- Since the authors say that SAM is optional, the results without SAM should be included in all the main table results, not just in the ablation study. 

- Addressing the points below would make the paper clearer:
  - Is the row length in R-RLE 16? Then, is Fig. 3 a smaller version with row length 6? The discrepancy between Fig. 3 and the textual explanation is a bit confusing. 
  - Add backbone column to main results table (ie. Table 1~3), so that direct comparison between TextSeg-some-backbone and another method using that backbone is easier to do. 
  - Add references of datasets/benchmarks in tables.
  - Explain what 'mix' under 'Training Data' of LISA means in the caption of Table 4. 
  - References from the end of page 6 to halfway of page 8 seem to be missing. 
  - Explain what training scheme was used for the ablation study: was TextSeg trained on the same combined RES datasets?
  - Typo in "Notably, TextSeg with ViT-L increases the average performance on RES tasks from 70.3 to 75.4 cIoU compared to TextSeg without a mask refiner, with only a little increase in inference time." Should ViT-L be SAM-L? Also, the numbers 70.3 and 75.4 are not in Tab. 6. 
  - Performance of 79.3 cIoU of TextSeg with SAM-H in Table 6 is different from performance of 79.2 cIoU in Table 1.
  - Mark best performances in bold in Table 5 (as done in other tables).

questions:
- In Figure 6, higher resolution patch prompts to SAM seems to result in worse segmentation results, and this is counter-intuitive as more finegrained the prompt, more finegrained the segmentation should be. Could the authors provide an explanation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730606929254

thread_type:
official_review

<thread_object>
creator: reviewer_sus4

<content>
summary:
This paper proposed TextSeg, a novel text-as-mask paradigm that casts image segmentation as a text generation problem, eliminating the need for additional decoders. It employs semantic descriptors as a new textual representation of segmentation masks where each image patch is mapped to its corresponding text label. It further compresses predicted token length by using Row-wise Run-Length Encoding to handle contiguous region of same semantic region.

strengths:
- model pixel label as semantic text token by generalized VLLM model
- compress token length with row-wise run-length encoding
- comprehensive results on referring express segmentation and comprehension

weaknesses:
- idea is simple and straightforward
- relies on SAM to get the final pixel level prediction from patch level semantic text token prediction
- should compare with other VLLM approach also with SAM refinement

questions:
- please discuss how to extend to instance level segmentation
- the name of TextSeg was already used: https://paperswithcode.com/dataset/textseg
</content>
</thread_object>

</thread>
<thread>
creation date:
1730626353936

thread_type:
official_review

<thread_object>
creator: reviewer_9zhz

<content>
summary:
The paper introduces TextSeg, a novel approach that integrates image segmentation into Multimodal Large Language Models (MLLMs) by framing segmentation as a text generation task.  The key innovation is the use of semantic descriptors, where each image patch is mapped to a textual label, allowing seamless integration into the auto-regressive training pipeline of MLLMs. By representing images with 16×16 semantic descriptors, the approach achieves competitive segmentation performance. To enhance efficiency, the authors propose Row-wise Run-Length Encoding (R-RLE), which compresses redundant text sequences by 74% and speeds up inference by three times without sacrificing accuracy. Extensive experiments demonstrate that TextSeg attains state-of-the-art results on multiple datasets across various vision tasks.

strengths:
The authors have conducted extensive experiments and ablation studies to demonstrate the effectiveness of their proposed method.

weaknesses:
The paper presents a method that integrates image segmentation into MLLMs by introducing semantic descriptors and utilizing a SAM mask refiner. While the approach simplifies the segmentation process by treating it as a text generation task, the technical contributions appear to be more incremental and engineering-oriented. The method essentially adapts existing MLLMs with semantic descriptors to perform segmentation tasks, serving as a baseline framework that can be applied to different MLLM models.

questions:
+ Training Datasets: The authors should clearly mention about the training datasets, additional training datasets for fine-tuning (if accapable) of other methods for a fair comparison. Because we know that with more training datasets (GLaMM, Groundhog), and with fine-tuning on task-specific datasets, we can achieve a better performance. 
+ Use of SAM Mask Refiner: The proposed TextSeg method incorporates a post-processing step using the SAM mask refiner. As indicated in Table 6, without SAM, the method achieves a cIoU of 73.5, which is lower than most methods in the Generalist Segmentation Models (~7B) category in Table 1. This raises concerns about the fairness of the comparison, as other methods may not use similar post-processing techniques. The authors should clarify how other methods could benefit from similar enhancements.
+ Inference Speed: It would be beneficial to include the inference speed of TextSeg compared to other methods. Providing quantitative metrics on inference time would strengthen the evaluation.
+ Missing Recent Specialized Segmentation Models: The paper lacks comparisons with recent specialized segmentation models such as PolyFormer [1], UNINEXT [2], and HIPIE [3]. Including these models in the evaluation would provide a more comprehensive assessment of the proposed method's performance relative to the current state-of-the-art.

[1] Liu et al., PolyFormer: Referring image segmentation as sequential polygon generation, CVPR, 2023.

[2] Yan et al., Universal instance perception as object discovery and retrieval, CVPR, 2023.

[3] Wang et al., Hierarchical open-vocabulary universal image segmentation, NeurIPS, 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732181225082

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response (2/2)

comment:
Table 3: Generalized Referring Expression Segmentation results on the gRefCOCO dataset.


|  | Refiner | Val (gIoU)  | Val (cIoU)  | Test A (gIoU) | Test A (cIoU) | Test B (gIoU) | Test B (cIoU)  | Avg. |
| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| GSVA | w/o | 66.5 | 63.3 | 71.1 | 69.9 | 62.2 | 60.5 | 65.6 |
| GSVA | SAM | 64.5 | 57.9 | 68.5 | 64.9 | 60.7 | 55.7 | 62.0 |
| TextSeg ($16^2$) | w/o | 68.8 | 63.1 | 66.9 | 67.1 | 62.1 | 61.6 | 64.9 |
| TextSeg ($16^2$) | SAM | 74.4 | 69.1 | 75.1 | 73.8 | 67.3 | 66.6 | 71.1 |
| TextSeg ($16^2$) | CRF | 70.0 | 66.1 | 69.4 | 70.9 | 63.1 | 64.1 | 67.3 |
| TextSeg ($32^2$) | w/o | 71.8 | 65.6 | 71.2 | 70.0 | 64.2 | 62.5 | 67.6 |
 
### Appendix CRF code

```python
import pydensecrf.densecrf as dcrf
from pydensecrf.utils import unary_from_labels

def crf_refine(image, mask, n_mask=2, gt_prob=0.9):
    """
    Refine a coarse mask using DenseCRF.
    
    :param image: Input image (HxWx3)
    :param mask: Coarse mask (HxW) 
    :return: Refined mask
    """
    # Create the DenseCRF model
    h, w = image.shape[:2]
    mask = mask.flatten()
    d = dcrf.DenseCRF2D(w, h, n_mask)

    # Get unary potentials (negative log probabilities)
    unary = unary_from_labels(mask, n_mask, gt_prob=gt_prob, zero_unsure=False)
    d.setUnaryEnergy(unary)

    # This adds the color-independent term, features are the locations only.
    d.addPairwiseGaussian(sxy=(3, 3), compat=3, kernel=dcrf.DIAG_KERNEL,
                          normalization=dcrf.NORMALIZE_SYMMETRIC)

    # This adds the color-dependent term, i.e. features are (x,y,r,g,b).
    d.addPairwiseBilateral(sxy=(50, 50), srgb=(13, 13, 13), rgbim=img,
                           compat=10,
                           kernel=dcrf.DIAG_KERNEL,
                           normalization=dcrf.NORMALIZE_SYMMETRIC)

    # Inference
    refined_mask = d.inference(5)  # Number of iterations
    refined_mask = np.argmax(refined_mask, axis=0)

    return refined_mask
```
</content>
</thread_object>

</thread>
<thread>
creation date:
1732181468988

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response (1/2)

comment:
We sincerely thank all reviewers’ and ACs’ for their time, effort, and thoughtful feedback on our paper. We deeply appreciate the reviewers' recognition of the effectiveness and originality of our contributions. **We have provided the latest revision in the system, with the changes highlighted in blue for clarity.** Below, we address the common concern raised by all reviewers:

**Use of SAM as a Mask Refiner**  (@9zHz, SUs4, stRB)

We understand the concerns regarding fairness in using SAM. However, we want to emphasize that our **TextSeg** does not rely on SAM to achieve better performance than existing works. Instead, we employ SAM as an **offline postprocessing** method primarily because it is more effective than other techniques such as CRF. Below, we provide a more detailed explanation:

1. **Fairness of comparing with existing methods**  
In Table 1, we summarise the segmentation decoders used in existing _embeddings-as-mask_ methods and observe that the comparison methods already incorporate SAM (or similar pixel decoders) in their architectures. From this standpoint, **it is entirely fair and justified  to directly compare our TextSeg with a SAM refiner against these existing methods.**

2. **Performance of existing methods with SAM postprocessing**  
We also try to apply the same post-processing technique to the comparison methods. We chose GSVA as a representative due to its available checkponts that could completely reproduce the results on RES and GRES tasks. As shown in Tables 2 and 3, incorporating an additional SAM refiner into GSVA did not yield improved results. Despite experimenting with different combinations of prompts for SAM refiner in GSVA, we were unable to achieve better performance. We hypothesize that the reason may be that GSVA already optimizes the SAM decoder during training to achieve optimal performance, and adding an additional SAM refiner may, in turn, negatively affect its overall performance. Therefore, **there is no clear evidence that adding a SAM refiner would enhance the performance of existing _embeddings-as-mask_ methods**, as these methods already optimize their SAM decoders during training.

3. **Outstanding performance of TextSeg without SAM**  
We provides two solutions to improve the performance of TextSeg (InternVL2-8B) without the SAM refiner.
   - We could use the traditional mask refinement techniques such as Conditional Random Fields (CRF) to produce more fine-grained segmentation masks. As shown in Tables 2 and 3, applying CRF improves TextSeg without SAM from an average cIoU of 67.5 to 70.1 on the RES task, compared to GSVA's 71.4 which uses SAM as decoder in its architecture. Similarly, for the GRES task, **the average score improves from 64.9 to 67.3, surpassing GSVA's 65.6**. 
    - Another solution is to use $32 \times 32$ semantic descriptors without any postprocessing. This makes TextSeg directly generate more fine-grained masks from MLLM. **It boosts average performance to 71.4 cIoU on the RES tasks, outperforming most existing methods, including NEXT-Chat (68.9), LISA (69.9), PixelLM (69.2), LaSagnA (71.1), and equaling GSVA (71.4).** On the GRES tasks, TextSeg with $32 \times 32$ semantic descriptors achieves **an average score of 67.6, higher than LISA's 62.9 and GSVA's 65.6**.  
**These results demonstrate the simplicity and effectiveness of our TextSeg approach, even without relying on a SAM refiner.**


Table 1: Summary of decoders of existing methods.
|   | LISA | GSVA | PixelLM | NEXT-Chat | AnyRef | LaSagnA | Groundhog | GLaMM |
| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| Decoder | SAM  | SAM  | Customized decoder  | SAM  | SAM  | SAM  | Mask2Former+  | SAM  |

Table 2: Referring Expression Segmentation results (cIoU) on RefCOCO (+/g) datasets.
|  | Refiner | val | testA | testB | val(+) | testA(+) | testB(+) | val(g) | test(g) | Avg. |
| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | ------- | ------- |
| GSVA | w/o | 77.2 | 78.9 | 73.5 | 65.9 | 69.6 | 59.8 | 72.7 | 73.3 | 71.4 |
| GSVA | SAM | 70.8 | 73.4 | 66.7 | 60.7 | 64.9 | 55.2 | 67.3 | 67.9 | 65.9 |
| TextSeg ($16^2$) | w/o | 70.3 | 71.9 | 68.7 | 65.0 | 68.9 | 60.8 | 66.7 | 67.6 | 67.5 |
| TextSeg ($16^2$) | SAM | 79.2 | 81.7 | 75.6 | 72.8 | 77.9 | 66.5 | 74.0 | 75.3 | 75.4 |
| TextSeg ($16^2$) | CRF | 73.0 | 75.2 | 70.7 | 67.6 | 72.1 | 62.6| 68.9 |70.3 | 70.1 |
| TextSeg ($32^2$) | w/o | 74.7 | 77.4 | 71.6 | 68.5 | 73.6 | 62.9 | 70.7 | 71.6 | 71.4 |
</content>
</thread_object>

</thread>


Metareview:
metareview:
The reviewers appreciated the novel and sound approach to segmentation using MLLM (stRB), the reasonable design of the proposed model (SUs4, stRB), extensive experiments (9zHz, SUs4, stRB), and clarity (stRB) of te paper. They however unanimously raised concerns about the use of SAM mask refiner that could cause fairness issues in comparisons with prior arts and contradict one of the main advantages the authors claimed in the paper. They also pointed out limited technical contribution (9zHz, SUs4), lack of comparisons with relevant methods (e.g., recent specialist segmentation models and other MLLMs) (9zHz, SUs4), and some presentation issues (stRB).

The authors' rebuttal, revision, and subsequent responses in the discussion period addressed most of these concerns. Consequently, two of the three reviewers were supportive of the paper after the discussion period, and while the other reviewer's rating remains a borderline rejection, that reviewer noted that his/her major concern with the use of SAM mask refiner has been successfully resolved. 

The AC found that the strengths of the paper, particularly the novel approach and extensive experiments, and the rebuttal outweigh the remaining concerns, and thus recommend acceptance of the paper. The authors are strongly encouraged to carefully revise the paper to reflect the valuable comments by the reviewers, to add new results brought up in the rebuttal and discussions, and to further improve the quality of writing.