ARGenSeg: Image Segmentation with Autoregressive Image Generation Model
Authors: Xiaolong Wang
Lixiang Ru
Ziyuan Huang
Kaixiang Ji
DanDan Zheng
Jingdong Chen
JUN ZHOU
Venue: NeurIPS 2025 poster ()
Keywords: image segmentation
unified multimodal understanding and pixel-level perception
image generation
Primary area: applications
TL;DR: We present ARGenSeg, a unified framework for multimodal understanding and pixel-level perception, achieving state-of-the-art performance of image segmentation.

Abstract: We propose a novel AutoRegressive Generation-based paradigm for image Segmentation (ARGenSeg), achieving multimodal understanding and pixel-level perception within a unified framework. 
Prior works integrating image segmentation into multimodal large language models (MLLMs) typically employ either boundary points representation or dedicated segmentation heads.
These methods rely on discrete representations or semantic prompts fed into task-specific decoders, which limits the ability of the MLLM to capture fine-grained visual details.
To address these challenges, 
we introduce a segmentation framework for MLLM based on image generation, which naturally produces dense masks for target objects.
We leverage MLLM to output visual tokens and detokenize them into images using an universal VQ-VAE, 
making the segmentation fully dependent on the pixel-level understanding of the MLLM.
To reduce inference latency, 
we employ a next-scale-prediction strategy to generate required visual tokens in parallel.
Extensive experiments demonstrate that our method surpasses prior state-of-the-art approaches on multiple segmentation datasets with a remarkable boost in inference speed, while maintaining strong understanding capabilities.

ARGenSeg: Image Segmentation with Autoregressive Image Generation Model
Xiaolong Wang, Lixiang Ru, Ziyuan Huang, Kaixiang Ji,
Dandan Zheng
Jingdong Chen, Jun Zhou
Ant Group
(xiaowang.we) wxl, rulixiang.rlx, pishi.hzy, kaixiang.jkx, yuandan, zdd) @antgroup.com (jinddongchen.cj.. jun.zhoujun) @antgroup.com
General Segmentation
USER the
repiono. Image Generation
Understanding Based Segmentation
USER: Please
USBR:
Plese aggmert <beoken botteroapeale>
USER: Plaaee premica pianer about mmontaiooiiaess
Figure I:
ARGenSeg is a unified framework for visual understanding, segmentation, and generation. It supports semantic, instance, interactive, and zero-shot reasoning segmentation, well anomaly detection, by leveraging strong visual understanding capabilities.
Abstract
We propose novel AutoRegressive Generation-basee paradigm for image Segmentation (ARGenSeg), achieving multimodal understanding and pixel-level perception within a unified framework. Prior works integrating image segmen- tation into multimodal large language models (MILMs) typically employ either boundary points representation or dedicated segmentation heads. These meth- ods rely discrete representations or semantic prompts fed into task-specific decoders, which limits the ability of the MLLM to capture fine-grained visual details. To address these challenges, we introduce segmentation framework for MLLM based on image generation, which naturally produces dense masks for target objects. We leverage MILM output visual tokens and detokenize them into images using an universal VQ- VAE, making the segmentation fully dependent
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

on the pixel-level understanding of the MLLM. To reduce inference latency, we em- ploy a next-scale-prees scale--pedictton strategy to generate required visual tokens in parallel. Extensive experiments demonstrate that our method surpasses prior state-of-the-ar approaches on multiple segmentation datasets with a remarkable boost in inference speed, while maintaining strong understanding capabilities.
Introduction
The emergence of large language models (LLMs)
has significantly accelerated the development of artificial general intelligence (AGI) Breakthroughs like ChatGPT [40) enable
8
17 52
the transformer-bast
[54] autoregressive framework to unify diverse tasks of natural language processing [ As for multimodal task, LLaVA 34 employs visual adaptor to map visual features
into the embedding space of LLMs, establishing universal paradigm for multimodal large language
3 22 explore the unified framework for multimodal understanding and generation. However, integrating fundamental visual perception tasks into unified AGI framework remains an open challenge. While sparse-output tasks such as visual grounding can be directly addressed via text expression [14], tasks requiring dense outpus like image segmentation are inherently difficult to represent through natural language.
models 33
5
15
56 Recent studies [48]
49
67
2
Previous methods that incorporate image segmentation into MLLMS typically fall into two categories. The first discretizes dense masks into boundary point sequences [T] 58), which inevitably leads to incomplete segmentation masks and unnatural object boundaries. The second achieves segmentation through downstream dedicated decoders (e.g.. SAM (27). Mask2Former [6)), which are conditioned
on either textual prompts [12 or hidden states 29 45 75 generated by MLLMs. This not only
results in complex model architectures, but also leads to insufficient understanding of pixel-level information for LLM due to its reliance on specialized task head.
To address the above challenges, we propose ARGenSeg, which leverages the image generation- based paradigm integrate image segmentation into unified MLLM framework. To retain the strong understanding capability of MILMs, we use continuous image features as the input. For the generation output, we train the model directly predict quantized image tokens, aligning with the next-token autoregressive prediction mechanism of language models. We use pre-trained VQ-VAE as image tokenizer quantize and detokenize images, with its visual tokens added to the codebook of MLLM. By leveraging the understanding ability of MLLM, ARGenSeg is capable of additional complex reasoning segmentation 129] anomaly detection 7 6 and other image segmentation tasks [5 as shown in Fig. ! The image tokenizer is kept frozen throughout training, thereby avoiding the dependence of LLM on subsequent decoders when learning pixel-level information.
In real-world application, image segmentation often requires fast response times. For this purpose, adopt next-scale prediction strategy for image generation. On one hand, the multi-scale mask generation process aligns with the intuitive process of object segmentation, which typically involves coarse localization followed by fine-grained boundary refinement. On the other hand, generating visual tokens in parallel provides a significant efficiency advantage, achieving over 4x speedup compared to sequential generation methods 19 59
Some methods also propose to use image generation for image segmentation. UniGS 43 uses diffusion model [2T 46] to achieve image segmentation. However, its U-Net structure causes lack of understanding ability. HiMTok [57] proposes an innovative mask tokenizer that enables decoding discrete outputs from the MILM into binary masks via image generation. However, the task-specific tokenizer limits its generality and extensibility. Moreover, both of these methods suffer from significant disadvantages in inference speed.
Extensive experiments demonstrate that the proposed ARGenSeg outperforms existing MLLM-based segmentation methods, while also achieving significantly faster inference. Notably, our method achieves superior performance using substantially less segmentation data compared to prior state- of-the-art approach [57]. In addition, the of a general- purppse visual tokenizer provides the flexibility to extend the framework to additional tasks. As a demonstration, by fine-tuning on a small amount of image generation data, we successfully unlock the image generation capability of our framework, as illustrated in Fig. T
The main contributions of this paper include:
2

---

We propose a novel image segmentation framework based on a unified multimodal under- standing and generation paradigm. To our knowledge, we are the first to show that unified ML.LMs can achieve SOTA segmentation results without any extra segmentation heads.
We leverage universal image tokenizer, allowing segmentation to fully rely on the pixel- level visual understanding of the MI..M. We further show that direct image token prediction by the MLLM is important for achieving high segmentation accuracy.
We propose to use next-scale prediction to speed up inference. And we observe that the coarse-to-fine multi-scale mask generation process also boosts segmentation robustness.
2 Related Work
Integrating image segmentation into MLLMs not only equips them with fine-grained visual percep- tion, but also enables more complex reasoning- based segmentation tasks by leveraging understanding capabilities. However, representing segmentation masks within the MLLM framework remains significant challenge. PolyFormer [35] and VistaLLM 42 represent masks as polygons using
point sequences, which are easy to express but struggle with complex shapes. LISA (29] aggregates segmentation information using special tokens and predicts masks through SAM 27 decoder.
Subsequent works such as GL.MMM [44), PixelLM [45], GSVA [65]. and PSALM 775 build upon this paradigm, and still rely on special tokens and dedicated segmentation decoders. These methods essentially aim to extract semantic embeddings of target objects and then obtain dense segmenta- tion masks by computing similarity with image features. Such representations tend emphasize high-level semantics rather than true pixel-level understanding. HiMTok (57 explores an alternative that removes the reliance on special tokens and SAM-like decoders. However, it still depends dedicated mask tokenizer trained on binary masks. Moreover, the expressiveness of the tokenizer is limited and cannot be extended to support other tasks such as image generation. This suggests that segmentation representation in MLLMs remains an open challenge, which we think can be effectively addressed through autoregressive image generation.
Unified multimodal understanding and generation models have recently attracted increasing attention for their ability to seamlessly perform both understanding and generation tasks within
single framework. Several works [48] (20 63 [51] leverage diffusion models for image generation by
regressing visual embeddings from MILM outputs and using them as conditional inputs. TransFusion (77 and Show-O (67) unify next-token prediction and diffusion-based generation within a single transformer framework. Chameleon 49 and Emu3 (59) adopt shared discrete visual embedding space for both understanding and generation, decoding images through VQ-based tokenizers [19]7T]. Janus [61] decouples the encoder for multimodal understanding and generation, using discrete visual tokens for generation while retaining continuous visual features for better understanding accuracy. VARGPT 78 proposes next-token prediction for understanding and next-scale prediction for image generation, but relies on an additional transformer-base visual decoder.
Image tokenization enables discrete outputs from autoregressive models to be reconstructed into images. VQ-VAE (53] encodes images into a downsampled latent space and quantizes the features into discrete token IDs, simplifying the learning process for generative models. VQGAN [19] improves reconstruction quality and training efficiency through adversarial training. TiTok (72) significantly
reduces the number of tokens required for image representation. improving generation speed, and further shows that increasing the number of latent tokens consistently enhances reconstruction quality. VAR [50] reformulates visual autoregressive generation as a next-scale prediction task, achieving high efficiency while maintaining relatively large number of visual tokens.
3 Method
In this paper, we propose novel image segmentation framework based on autoregressive image generation model, using Vector- Dector-Quantzeed (VQ) autoencoder [53] 19 tokenize images into discrete tokens and reconstruct them from generated outputs. To address the unique challenges of segmentation, we introduce two key designs. (1) The MILM is trained to directly output image tokens, which is crucial for achieving high pixel-level accuracy. (2) We utilize a multi-scale generation process that performs coarse-to-fine refinement. This not only enhances segmentation robustness but also improves inference efficiency. This section first presents the background of the
3

---

image tokenizer (Sec. 3.1 then details the architecture (Sec. 3.2 training procedure (Sec. 3.3 and
inference process (Sec. 3.4)
of our proposed model.
3.1 Preliminary
Dector-Quantize Autoencoder space and reconstruct them from discrete tokens. Given an input image I #xx33 x3 the encoder maps it to latent feature space:
The standard VQ model learns to encode images into latent
E fer.eeRRRR fERt**x,
(1)
where is the spatial downsampling factor and D denotes the feature dimesion. The latent features are then quantized by a vector quantizer Q into discrete token indices E
[V] x
q 2(), ((
arg miggm mim)) c²I12
(2)
=E[V]
where e² is the v-th embedding vector in the visual codebook C E RRRD and [V] denotes the set of codebook indices (1,2,-. .,.,...VV,,
The reconstruction of the image can be interpreted as detokenizing discrete visual tokens into an image, In this procedure, the quantized indices q are used to index the corresponding embedding from the visual codebook C. producing the estimated latent feature map f. The estimated feature map is then passed through the decoder 'D to generate the reconstructed image Í:
f = lookup(C,, (C, 9), i D(f).
When using VQ-VAE for autoregressive image generation, the inference process typically requires (n²2) steps. To address this inefficiency, (AR introduces next-scale prediction paradigm for visual token generation. Specifically, the feature map f is quantized into multi-scale token maps (r1, 72, ..... TK). where each map corresponds
Multi-Scale VQ Autoencoder
different resolution. At each inference step, the model generates all h& x tokens required for the current scale rı in parallel, repeating this process until L reaches the target resolution of # W. Moreover, the coarse- to to-fine predictions can enhance the generation quality. Based on this paradigm, an image of resolution 256 256 can represented using 680 visual tokens, while requiring just K autoregressive inference steps, significantly improving generation efficiency. Given the fast response requirements of image segmentation tasks, we adopts this paradigm to enable efficient autoregressive image generation.
ARGenSeg
Training Inference
Image
Trrbaddigg
Discrete visua tokens
Tokes
Vision
Up Interplation
Viraa Embottinn
Vicaa Vokal Toker
Projector
Ger Projector
Tokentzer
Vis Projector
Gen Projector
Large Language Model
Large Language Model
nead
Segmentation
Mask
Cross: Entropy
Figure The architecture ARGenSeg and its training and inference procedures. Left: ARGenSeg integrates image segmentation into the MILM via autoregressive image generation paradigm. A unified classification prediction head is used to generate both text and visual tokens. Right: Visual tokens are generated in parallel using the next-scale prediction strategy. During training, VAE encoder used construct supervision for cross-entropy loss. During inference, the VAE decoder reconstructs the image from the predicted visual tokens. [S][[E] denotes been_start>/gge_eeet

---

3.2 Architecture
Multimodal Understanding ARGenSeg uses a unified autoregression framework for image un- derstanding and generation as shown in Fig. 2 Our framework employs the built-in tokenizer of the LLM to convert text input into discrete token IDs and corresponding embeddings. For image input, a vision encoder is used to extract features, which are then mapped to the LLM''s embedding space via a vision projector. After the concatenated embeddings are fed into the LLM, the model performs next-token prediction to sequentially generate token embeddings. These embeddings are then passed through classification head to sample discrete token IDs, which are subsequently detokenized into meaningful text. For multimodal understanding tasks, decoupling the framework from image generation preserves the native understanding capabilities of the LLM.
To integrate image generation into the framework, we introduce special tokens <gen_start> and $gen_end> to mark the beginning and end of the generation process. Additionally, the visual token IDs from the visual tokenizer are added to the LLM's vocabulary in the form of Saital_toke_______ ID> When image generation is required, the framework autonomously determines whether to initiate generation based on the input instruction. Upon encountering the <gen_ "gen_start> token, multi-scale image generation begins, where visual tokens for each scale are predicted in parallel. At k-tth scale, the visual feature corresponding to the visual token map from the previous scale is retrieved by looking up the visual codebook C and then upsampled to match the resolution of the current scale. A lightweight linear layer, referred to as the generation projector, maps these upsampled visual features into the embedding space of L.M,, serving as input for the next scale. This design allows one-step parallel inference to obtain all visual tokens at the current scale. Importanly, the unified prediction head is used to generate visual tokens, which are then directly converted to the corresponding index IDs in the codebook C. Once all visual tokens across scales are generated, they are detokenized by the visual tokenizer to reconstruct the final image.
Image Generation
3.3 Training Procedure
Training Strategy In our framework, the vision encoder, large language model, vision projector and classification prediction head are initialized using InternVL. 2.5 while the multi-scale visual tokenizer is initialized from VAR (50). During training, the vision encoder and visual tokenizer are kept frozen reduce the model's reliance on dedicated decoders for pixel-level understanding. By leveraging pre-trained multimodal understanding, the framework converges rapidly when training on image segmentation data. Thus, we employ single-stage supervised finetuning (SFT) strategy, jointly optimizing both image segmentation and multimodal understanding data. For image generation, we further finetune the pre-trained ARGenSeg model using image generation data to unlock its text-to-image generation capabilities.
Training Objective Since our framework unifies both text and image generation outputs within the LLM codebook, the entire training process is directly supervised using cross-ents cross-entropy loss, as shown in During supervision construction, the <gen_start> token is added as a marker before image generation begins. The model is expected to learn both when to initiate image generation and how to generate all the required visual tokens. The ground-truth visual tokens are obtained using the encoder and quantizer of the VAE.AE. When constructing input embeddings, the visual tokens for the first scale are obtained by using the $gen_start> token as the query. For each subsequent scale, the input embeddings are derived by upsampling the visual token map Tk-1 of the previous scale to match the size of the current scale. Finally, the <gen_ends token is added to ensure the proper progression of subsequent predictions.
Fig. 2
3.4 Inference
During inference, our model follows a next-token prediction strategy, generating outputs sequentially until the agen_starta token is produced. This token then serves as a query to initiate the generation of visual tokens for the first scale. For the subsequent K-1 scales, query embeddings of size h4 x are obtained by upsampling and projecting the visual token map ft-1 predicted at the previous scale, enabling parallel generation of all visual tokens at the current scale. Since the upsampling process determines the number of queries, our framework naturally ensures alignment between the number of generated tokens and the input size required by the VQ-VAE decoder. Once the visual tokens for
5

---

Table 1: Performance comparison with state-of-the-ar methods on three referring image segmentation benchmarks using cloU. (ft) indicates models further finetuned on RefCOCO//g after mixed training.
RefCOCO
RefCOCO+ testA
RefCOCOg
Paradigm
Method
testA
testB
testB
val
test
Boundary based
PolyFormer-B B
42
74.8
76.6
71.1
67.6
72.9
59.3
67.8
69.1
VistaL 7B
42
74.5
76.0
72.7
69.1
73.7
64.0
69.0
70.9
LiSA-7B(ft) 29
74.9
79.1
72.3
65.1
70.8
58.1
67.9
70.6
PixeILM--B 445
73.0
76.5
68.2
71.7
69.3
70.5
GSVA-7B 65
76.4
77.4
72.8
64.5
67.7
58.6
71.1
72.0
GSVA-7B(ft)
77.2
78.9
73.5
65,9
69.6
59.8
72.7
73.3
LaSagnA-7B 660
76.8
78.7
73.8
66.4
70.6
60.1
70.6
71.9
VisionLLM
62
76.6
74.3
69.8
70.7
71.2
Dedicated
OMG-LLAVA
77
75.6
77.7
71.2
65.6
69.7
58.9
70.7
70.2
Segmentation
OMG-LLAVA(1t)
78.0
80.3
74.1
69.1
73.1
63.0
72.9
72.9
Head-based
GLaMM 44
79.5
83.2
76.9
72.6
78.7
64.6
74.2
74.9
u-LLAVA 68
83.0
80.5
77.1
81.7
70.6
77.1
78.0
PSALM 75
83.6
84.7
81.6
72.9
75.5
70.1
73.8
74.4
GroundHog 7B 74
78.5
79.9
75.7
70.5
75.0
64.9
74.1
74.6
SAM4MLLM-BB 8B 442
79.8
82.7
74.7
74.6
80.0
67.2
75.5
76.4
LMM SB [57
81.1
79.2
77.1
78.8
75.8
76.7
LMM -ft((t)
85.0
85.2
83.5
79.7
82.7
76.0
80.0
80.6
Generation
ARGenSeg
82.2
80.1
81.8
78.4
79.6
based
ARGenSeg (ft)
86.3
87.5
82.7
82.3
85.8
77.0
81.7
83.5
all K scales are obtained, the VAR tokenizer decodes them into the final image. To ensure smooth progression of subsequent inference, the <gen_end> token is manually added.
Experiments
4.1 Experimental Setup
Datasets As described in Sec. 3.3 we perform single-stage supervised finetuning to jointly train on both image segmentation and multimodal understanding data. Details provided in Appendix [A] The training of ARGenSeg relies entirely publicly available external datasets. Specifically, we use 402K image segmentation samples, which are significantly fewer than the 2.91M samples used by HiMTok[[7] and constitute strict subset of their data. For multimodal understanding, we use 1.25M samples derived from the open-source dataset of Intern 1.2 [4].
datasets used are
Implementation Details Our model accepts input images of arbitrary resolutions, while the output images are generated at the resolution 256 x 256. The image tokenizer uses a downsampling ratio l 16, with feature dimension D 32 and visual codebook size V 4096. The model operates with = 10 scales. During training, we use the Adam W (36) optimizer with a maximum learning rate 10 5 and employ cosine learning rate scheduling. The batch size
128.
4.2 Referring Segmentation
Referring Expression Segmentation Recent works have increasingly focused on equipping multi- modal large language models with image segmentation capabilities, aiming to leverage their strong language understanding for more complex segmentation tasks. Referring Expression Segmentation (RES) requires models segment target objects in an image based on natural language descriptions.
We evaluate our approach on standard RES benchmarks RefCOCO/+/g [37 70] Following prior works [2 17), we assess two versions of our model: one trained on the mixed dataset, and another further finetuned on the in-domain training sets of RefCOCO/+/g, As shown in Tab. [ our method consistently outperforms the previous state-of-the-arr. HiMTok [57], across both versions, despite training on fewer segmentation data. It is worth noting that, our approach achieves superior results without relying on dedicated segmentation head, demonstrating the effectiveness of our unified multimodal understanding and generation framework.

---

Table 2: Performance comparison with state-of-the-an methods on generalized referring expression segmentation. * 'nniicttes zero-shot performance.
val
testA
testB
Method
Average
cloU
gloU
cloU
gloU
cloU
gloU
LISA-7B 29
38.7
32.2
52.6
48.5
44.8
39.7
42.8
LISA-7B(ft)
61.8
61.6
68.5
66.3
60.6
58.8
62.9
GSVA-7B (65
61.7
63.3
69.2
70.1
60.3
61.3
64.3
GSVA-7B(ft)
63.3
66.5
69.9
71.1
60.5
62.2
65.6
LaSagnA [60]
38.1
32.4
50.4
47.3
42.1
38.9
41.5
PSALM* [75
42.0
43.3
52,4
54.5
50.6
52.5
49.2
GroundHog-785
74
66.7
66.7
SAM4MLLLM-8B
42
67.8
71.9
72.2
74.2
63.4
65.3
69.1
LMM
[57
66.8
68.7
68.6
67.6
65.8
64.1
66.9
ARGenSeg
72.2
74.7
73.6
73.7
70.0
70.4
72.4
original
(b) scale == 4
scale k 5
scale at k
6
(f)
(g)
(b)
Figure 3: Multi-scale generation process of the segmentation mask. The model first localizes the
target object and then progressively refines its boundaries.
Fig. 3 illustrates the multi-scale mask generation process of ARGenSeg. The model first locates the target object and then progressively refines the segmentation boundaries. This coarse-to-fine reasoning process aligns with human intuition and enhances the robustness of image segmentation.
Generalized Referring Expression Segmentation We further evaluate our model on the more challenging gReíCOCO benchmark (32). where segmentation instructions may refer to multiple objects none at all. As shown in Tab, 2 our method outperforms all prior approaches that rely on dedicated segmentation heads, highlighting the strong understanding and segmentation capabilities of our unified framework.
4.3
Multumodal Understanding
Our model adopts Intern 2.5 13 as the underlying MILM and is finetuned on both understanding and segmentation data. To fairly assess the effect of adding segmentation supervision on the model's understanding capability, we finetune baseline using only understanding data. We evaluate the model's 's understanding performance using two tasks. The first task is visual grounding, where use the RefCOCO/+/g datasets for referring expression comprehension (REC). As shown in Tab, our model successfully retains and even slightly enhances its grounding ability while acquiring segmentation capabilities. The second task evaluates object hallucination in MLLMs using POPE 30 as the benchmark. Results in Tab. 3 also demonstrate a performance improvement of our model compared to the baseline. These results highlight the effectiveness of our proposed framework in unifying understanding and segmentation tasks. further discussion on the understanding performance is provided in Appendix C.I

---

Table 3: Multimodal understanding performance compared with the bascline. finetuning on understanding data.
indicates further
RefCOCO testA
RefCOCO+ testA
RefCOCOg test
Method
POPE
val
testB
val
testB
val
InternVL2.5-8B**
13
89.0
92.6
84.3
83.4
89.1
76.5
83.5
85.0
86.73
ARGenSeg
89.6
92.8
84.4
83.8
88.8
76.5
86.1
85.6
87.57
4.4 Function Extension
Interactive Segmentation Interactive segmentation allows users to provide diverse input prompts during segmentation tasks meet varying application needs. We finetune ARGenSeg on the COCO- Interactive dataset [75] to unlock its interactive segmentation capabilities. During training, various forms of interactive prompts are used, including points, scribbles, and bounding boxes. Bounding boxes are provided as textual input to the MILMM, while points and scribbles are represented as binary masks and fed as additional visual inputs. We observe that, building upon pre-trained segmentation capabilities, the model quickly adapts to interactive segmentation tasks. Qualitative results are shown the top portion of Fig. while the quantitative evaluation can found the Appendix C.22
Image Generation Our model leverages universal image tokenizer, enabling the potential for image generation. We finetune ARGenSeg on 1.28M class-based samples from the ImageNet- Instruct-class dataset [78. using batch size of 512 for 20k: iterations. This successfully enables class-conditional image generation, as illustrated in Fig. [ We then continue training for an additional 30k iterations with batch size of 256 on the ImageNet-Insiin Instruct on instruction-condition in the bottom of Fig. 4 Notably, our model achieves these results without relying on pre-trained generation model, using only a small amount of data and training iterations.
dataset [78), which is based generation. The results of instruction- -based image generation are shown
Point
Scribble
Can you produce painting of largo while house large porch.
Please render designed photo of bascball the grass.
would like crafted figure of white shark swinming in the
Can you design crafted image of starfish the beach.
noed sketched figure of the ocean very rocky and has
Please cralt visualized figure large volcano in the distance.
Figure 4: Top: Visualization of interactive segmentation. Points and scribbles are provided as visual prompts, while bounding boxes input via text. Bottom: Visualization results of instruction-baase image generation. The model is trained on image generation data for only 50k iterations.
4.5 Efficiency Analysis
We compare ARGenSeg with previous autoregressive generation models and MLLM-based segmen- tation methods in terms of inference time required to generate 256 x 256 image mask. All experiments are conducted using official implementations on an NVIDIA A100 GPU. Segmentation performance evaluated using cloU on RefCOCO-val. Detailed results provided in Tab.
Compared sequential token generation approaches such as Emu3 [59], our parallel inference achieves more than x0x speedup. While VARGPT [78] also employs VAR as its visual tokenizer,

---

Table 4: Computational efficiency comparison.
Table 5: Ablation study on the impact of under- standing capability, eettrining-stage and gener- ation projector on segmentation performance.
"Num." " represents the number of required tokens. Time is tested by seconds per image.
Method
Paradigm
Num.
cloU
Time
Experiment
Ref.
Ref.+
Ref.g
Average
Emu3
59
VQ-GAN[19
1024
59.4
Baseline
82.2
77.9
78.4
79.5
VARGPT
PixelLM
78
Query + Seg.Dee
VAR + Des
680
73.0
0.91
2.64
Only-Seg.
80.5
73.8
73.2
75.8
HiMTok
Mask Tokenizer
32
81.1
1.89
Gen Projetctor
80.5
73.7
73.4
75.9
ARGenSeg
VAR Tokenizer
680
82.2
1.28
Pretrain
80.8
74.9
74.1
76.6
Table Ablation study on the visual tokenizer. All results are reported on the val splits, using gloU (per-sample IoU averaged over the dataset) as the segmentation metric.
Tokenizer Type
Prediction
RefCOCO
RefCOCO+
RefCOCOg
Average
Time
Single-scale
next-token next-scale
82.1
71.8
65.8
73.23
5.50
Multi-scale
80.5
76.7
70.4
75.87
1.28 s
method is approximately 2x more efficient, due its simplified architecture. In contrast to VARGPT, our model directly uses the classification head to predict token IDs from the VAR codebook, climinating the need for additional transformer-basee visual decoder. PixelLM 45], identifier- based approach, uses only six tokens and a dedicated segmentation decoder, making slightly faster than ARGenSeg. However, its segmentation performance is significantly lower. While HiMTook [57] employs a dedicated mask tokenizer to achieve notable segmentation performance using only 32 visual tokens for efficiency, method achieves superior performance while offering clear advantage in inference speed.
4.6 Ablation Study
Ablation Understanding Data We compare baseline, fine-tuned both understanding and segmentation data, against counterpart trained solely on segmentation data. As shown in Tab. incorporating understanding data significantly improves performance reasoning-basee based segmentation, particularly on the semantically challenging RefCOCO+/g dataset. This highlights the value of unifying segmentation with multimodal large language model.
Ablation on Model Architecture and Training Strategy We analyze the effects of model archi- tecture and training strategy. First, to ablate the architecture, replace our default single-layer generation projector with a two-layer variant. Results indicate that the simpler design is sufficient. Second, to assess the training strategy, projector is trained, followed by a full fine-tuning stage. As shown in Tab. [ this two-stage approach offers only marginal gains on RefCOCO+/g and little impact RefCOCO, while increasing training complexity. Therefore, for efficiency, our final model adopts a direct, single-stage fine-tuning strategy,
introduce a pre-training phase where only the generation
Ablation Visual Tokenizer We ablate multi-scale visual tokenizer by comparing it against single-scale tokenizer, for which we adopt the pre-trained VQ-GAN (59) from Janus [61] As shown Tab. 6 using multi-scale scheme not only demonstrates clear speed advantage but also improves robustness through its inherent coarse-to-fine refinement process. Further ablations, including an analysis of using semantic embeddings instead of visual tokens, are provided Appendix D
Conclusion
In this paper, we present ARGenSeg, unified framework that integrates image segmentation into multimodal large language models through an image generation paradigm. To address the unique challenges of segmentation, we design the framework so that the MLLM directly outputs image tokens for pixel-level accuracy and utilizes multi-scale image generation for high responsiveness and robustness through coarse-to-fine refinement. Our experiment results are the first to show that unified MILMM models can perform state-of-the-ard segmentation without any extra specifc segmentation heads, providing effective technical pathway for unified AGI.

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 23X1023033.8877,, 2023.
[2] Inclusion AI, Biao Gong, Cheng Zou, Chuanyang Zheng, Chunluan Zhou, Canxiang Yan, Chunxiang Jin, Chunjie Shen, Dandan Zheng, Fudong Wang. al. Ming-omni: A unified multimodal model for perception and generation. arXiv preprint arXiv:2506.00 09344, 2025.
[3] Inclusion AI, Biao Gong, Cheng Zou, Dandan Zheng. Hu Yu, Jingdong Chen, Jianxin Sun, Junbo Zhao, Jun Zhou, Kaixiang Ji, et al. Ming-lite-uni: Advancements in unified architecture for natural multimodal interaction. arXiv preprint S.Ki:2505.00277/ 2025.
[4] Jinze Bai. Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint UrXiv:2309.66699 2023.
[5] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision- languaee model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12000 2023.
[6] Paul Bergmann, Kilian Batzner, Michael Fauser, David Sattlegger, and Carsten Steger. The mvtec anomaly detection dataset: comprehensive real-world dataset for unsupervised anomaly detection. International Journal of Computer Vision, 129(4): 1038-1059, 2021.
[7] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a comprehen- sive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 9592-9600), 2019.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[9] Sébastien Bubeck, Varun Chadrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence:
Early experiments with gpt-4, 2023.
[10] Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th international conference on computational linguistics, pages 1511-1520, 2022.
[11] Ting Chen, Saurabh Saxena, Lala Li, Tsung- Yi Lin, David J Fleet, and Geoffrey E Hinton. A unified sequence interface for vision tasks. Advances in Neural Information Processing Systems, 15:31333-31344 2022.
[12] Yi-Chia Chen, Wei-Hua Li, Cheng Sun, Yu-Chiang Frank Wang, and Chu-Song Chen. Sam4mllm: Enhance multi-modal large language model for referring expression segmentation. In European Conference on Computer Vision, pages 323-340. Springer, 2024.
[13] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, arXiv preprint 2:210:2412.05221, 2024.
[14] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):220101, 2024.
[15] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internyl: Scaling up vision foundation models and aligning for generic visual-linguistie tasks. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 24185-24198, 2024.
10

---

[16] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 1290-1299, 2022.
[17] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1- 113, 2023.
[18] Christopher Clark and Matt Gardner. Simple and effective malti-par reading comprehen- sion. arXiv preprint 0rXiv:1710.10725. 2017.
[19] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883 2021.
[20] Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang. and Ying Shan. Making Ilama see and draw with seed tokenizer. arXiv preprint arXiv:2310.012/8, 2023.
[21] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
[22] Ziyuan Huang, DanDan Zheng, Cheng Zou, Rui Liu, Xiaolong Wang, Kaixiang Ji, Weilong Chai, Jianxin Sun, Libin Wang, Yongjie Lv, et al. Ming-univision: Joint image understanding and generation with a unified continuous tokenizer. arXiv preprini
2025.
[23] Drew A Hudson and Christopher Manning. Gqa: new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[24] Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648-5656, 2018
[25] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 235-251. Springer, 2016.
[26] Geewook Kim, Teakgyu Hong. Moonbin Yim, Jeong Yeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Conference on Computer Vision, pages 498-59 517. Springer, 2022.
[27] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015-4026, 2023.
[28] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz. Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
[29] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li. Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579 95899 2024.
[30] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object ballucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
[31] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755, Springer, 2014.

---

[32] Chang Liu, Henghui Ding, and Xudong Jiang. GRES: Generalized referring expression seg- mentation. CVPR, 2023.
[33] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[34] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
[35] Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and R Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 18653-18666 2023.
[36] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arX1v:1711.05101,, 2017.
[37] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan I. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 11-20, 2016.
[38] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprins arXiv:2203. 10244, 2022.
[39] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947-952. IEEE, 2019.
[40] OpenAl. Chatgpt.
com/ 2023. Accessed: 2023.
[41] Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, al. Transfer between modalities with metaqueries. arXiv preprint 2504:255040 2025.
[42] Shraman Pramanick, Guangxing Han, Rui Hou, Sayan Nag. Ser-Nam Lim, Nicolas Ballas, Qifan Wang. Rama Chellappa, and Amjad Almahairi. Jack of all tasks master of many: Design- ing ceneral-purposs coarse-to- coarse-to-fine vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14076-14088, 2024.
[43] Lu Qi, Lehan Yang, Weidong Guo, Yu Xu, Bo Du, Varun Jampani, and Ming-Hsuaa Yang. Unigs: Unified representation for image generation and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6305-6315, 2024.
[44] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages I 13009-13018, 2024.
[45] Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 26374-26383, 2024.
[46] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[47] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8317-8326 2019.
[48] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. arXiv preprint 2rXtn:2307.05522, 2023.
12

---

[49] Chameleon Team. Chameleon: Mixed-modal early- early-fusion foundation models. arXiv preprint arXiv:2405.09878, 2024.
[50] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 81848339-8865, 2024.
[51] Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal under- standing and generation via instruction tuning. arXiv preprint arXiv:24/2.14166 2024.
[52] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arX1v:2302.13377, 2023.
[53] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[55] Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recogmition in natural images. arXiv preprint 160iv:1001.000 140, 2016.
[56] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing Saiiunla model's perception of the world at any resolution. arXiv preprint
2024.
[57] Tao Wang, Changxu Cheng, Lingfeng Wang. Senda Chen, and Wuyue Zhao. Himtok: Learning hierarchical mask tokens for image segmentation with large multimodal model. arXiv preprint arXiv: /3026, 2025.
[58] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model also an open-ended decoder for vision-centti tasks. Advances in Neural Information Processing Systems, 36:61501-61513, 2023.
[59] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409. / 18869, 2024.
[60] Cong Wei, Haoxian Tan, Yujie Zhong, Yujiu Yang, and Lin Ma. Lasagna: Language-based segmentation assistant for complex queries. arXiv preprint wxXiv:2404008506, 2024.
[61] Chengyue WU, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410. 13848, 2024.
[62] Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai. Zhaoyang Liu, Zhe Chen, Wenhai Wang. Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. Advances in Neural Information Processing Systems, 017-9925-699773 2024.
[63] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal 11m. In Forty-first International Conference on Machine Learning, 2024.
[64] Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.04443 04429, 2024.
[65] Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3858-3869 2024.
13

---

[66] Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, Han Cai, et al. Sana 1.5: Efficient scaling of trainin-timm and inference-time compute in linear diffusion transformer, 2025.
[67] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single trans- former to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
[68] Jinjin Xu, Liwu Xu, Yuzhe Yang, Xiang Li, Fanyi Wang, Yanchun Xie, Yi-Jie Huang, and Yaqian Li. u-llava: Unifying multi-modal tasks via large language model. In ECAI 2024, pages 618-625, IOS Press, 2024.
[69] Senqiao Yang. Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. An improved baseline for reasoning segmentation with large language model. CoRR, 2023.
[70] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L. Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69-85. Springer, 2016.
[71] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats tiffusion-tokenizerr is key to visual generation. arXiv preprint 231002231. 2023.
[72] Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and HtangCChich Chen. An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 37: 37:128940-12866, 2024.
[73] Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. Advances in Neural Information Processing Systems, 37:71737-71767, 2024.
[74] Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. Ground- hog:
Grounding large language models to holistic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14227-14238, 2024.
[75] Zheng Zhang, Yeyao Ma, Enming Zhang, and Xiang Bai. Psalm: Pixelwise segmentation with large multi-modal model. In European Conference on Computer Vision, pages 74-91. Springer, 2024.
[76] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 633-641, 2017.
[77] Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one
model. arXiv preprint arXiv:2408. /1039, 2024.
[78] Xianwei Zhuang, Yuxin Xie, Yufan Deng, Liming Liang, Jinghan Ru, Yuguo Yin, and Yuexian Zou. Vargpt: Unified understanding and generation in a visual autoregressive multimodal large language model. arXiv preprint arXiv:2501.12327, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contents in the abstract and Instruction clearly reflect the contribution of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We analyzed the computational efficiency and included the Limitations subsec- tion in the appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
We do not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We clearly explain the structure, dataset and training details of the method.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The data and the underlying model used in this article have provided detailed information and are publicly accessible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
public/guides/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee//mdeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details of the training and test have been explained in the Experiment section of the text.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due the resource limitation, do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The platform and running time for the experiment are given in the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics battps://neurippe: http:/pp....ppubllttttttiiciiiie
Answer: [Yes]
Justification:
The research in this article is in line with NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss societal impact of the work the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: We do not plan release any data. For the model, we currently do not have safeguards for releasing it. We will make sure that the guidelines and instructions are in place when release the model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we have.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
No new assets are provided.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buctps://neure wwwparippps..coooeeeeeeeee for what should or should be described.
21

---

Appendix of ARGenSeg
A Implementation Details
Datasets The datasets used for image segmentation, multimodal understanding, and image genera- tion are listed in Tab. 7 To ensure a fair comparison, we exclusively use subsets of the data employed by the previous state-of-the-an method, HiMTok [57]. Specifically, we train on 402K segmentation samples compared to HiMTok's 2.91M, and 1.25M multimodal understanding samples compared to HiMTok's s 4.2M. Image generation data are used only in the optional function-extension stage.
Table 7: Training data used in our experiments.
Task
Datasets
Image Segmentation
ADE20K(20K) [76], COCO--anontic (8K) [3, gReíCOCO (79K) 32 RefCOCO/+//g1277
37 70 LISA++ (nst.Seg(58K)) 669
A12D 25
ChartQA (38,, COCO- Text [55], [8c LLaVA- 50K|
Multimodal
GQA 23
DVQA(24], OCR- (ERR--QA33) TexIVQA SynthDoG-EN 26
Understanding
Intern MeeeVV--SAI--e CaCtiion [14, VisualGenome 228] GeoQA+ [0
Image Generation
TaaggNet-Innstruttee
778. MmageNNotIInsrrut122000 [78]
During inference, we get visual outputs exclusively from the logits corresponding to visual tokens in the MLLM codebook. This constraint ensures compatibility with the visual tokenizer and enables successful reconstruction of the image. For image segmentation tasks, we adopt deterministic argmax sampling strategy to obtain the predicted visual tokens. For image generation tasks, we apply classifier-free guidance (CFG) to compute the output distribution over visual tokens, followed by top-k sampling to enhance the diversity and quality of generated images.
Inference Details
B Additional Qualitative Results
Multi-scale Image Generation We provide visualization of segmenting similar objects in the same image using different instructions, as shown in Fig From the multi-scale mask generation process, it is evident that our model can correctly understand and localize the target based on the given instructions. The ability to correctly follow distinct segmentation commands indicates that ARGenSeg possesses a robust understanding of both spatial positions and semantic relationships.
Comparison with Single-scale Generation We compare our method with HiMTok (57] treating it as a representative single-scale generative segmentation approach. We conducted thorough evaluation on the test set and visualized cases where ARGenSeg succeeds while HiMTok fails. As shown in Fig. 6 these cases reveal two primary advantages of our coarse-to- fine, multi-scale generation scheme: (1) Robust Target Identification in Multi-object Scenarios. The initial coarse localization stage effectively identifies the target object even when multiple similar objects present. (2) Enhanced Mask Quality through Progressive Refinement. Following target identification, the multi-scale refinement process progressively improves mask precision for higher-quality segmentation. For instance, in the case of a partially occluded teddy bear, both HiMTok and our coarse localization stage initially segment only a visible part. However, our model's subsequent fine-grained refinement successfully reconstructs the entire object while correctly excluding the occluder.
C Additional Quantitative Results
C.1 Performance Multimodal Understanding
We further assess the multimodal understanding capabilities of ARGenSeg. As shown in Tab. g the inclusion of segmentation data does not cause the model to lose its reasoning capability. While we
22

---

computer right side of picture
laptop next to cat
Figure 5: Visualization of using different segmentation instructions in the same image.
Ground Truth
HiMTok
ARGenSeg: corange front slight right of gIE
Ground Truth
5
HiMTok
ARGenSeg: <bear the child is hugging>
Figure 6: Comparison between multi-scale and single-scale generative segmentation approach. The examples highlight scenarios where the multi-scale approach excels.
23

---

observe slight performance drops on some benchmarks, we attribute this minor degradation not to the segmentation task itself, but to the significantly smaller and lower-quality understanding corpus used for fine-tuning (1.25M vs. the 16.3M samples used for V1.2.5 [3] To validate this hypothesis, we conducted control experiment: fine-tuning Intern solely on the same understanding data for an increasing number of steps. The performance declined monotonically, mirroring the trend observed with joint segmentation training and thus confirming our attribution.
Table 8: Multimodal understanding results across benchmarks.
Method
POPE
TextVQA
VQAv2
MMMU-val
A121
Intern V1.2.5-ft- lep
86.73
63.54
80.40
43.7
78.7
Intern -ee2.5-ft-4ep
86.01
59.73
79.28
36.8
74.8
ARGenSeg
87.57
56.98
77.87
33.4
69.6
C.2 Results on Interactive Segmentation
To ensure fair comparison with HiMTook, which was not trained on seeractive-semmentatee data, omitted this task from our main experiments. Here, we evaluate model the COCO-Interactve benchmark (75), porting the cloU metric. is worth noting that while PSALM 755 was fine-tuned for 10 epochs according to its official implementation, our model is fine-tuned for only single epoch due to computational constraints. As shown in Tab. 9 ARGenSeg significantly outperforms SAM 277 in interactive segmentation. Moreover, it achieves per- formance comparable to PSALM with substantially less fine-tuning, which underscores the strong generalization capabilities of our model.
Table 9: Quantitative results interac- tive segmentation. The results for SAM and PSALM are sourced directly from the PSALM paper.
Method
Point
Scribble
Box
SAM-B 27
33.6
68.7
SAM-L
27
37.7
71.6
PSALM 75
74.0
80.0
80.9
ARGenSeg
65.6
68.6
79.1
D Additional Ablation Studies
Table 10: Ablation study of MLLM backbones and image generation strategies. The segmentation performance measured in cloU.
Method
Backbone
Generation Strategy
RefCOCO
RefCOCO+
RefCOCOg
HiMTok 57
V
Single-scale VQ
81.1
77.1
75.8
ARGenSeg-LLan -LVVA
VL-155
Multi-scale VQ
72.7
68.3
69.1
ORGenSeg-Intern'i
Intern VnternV--2.5
Multi-scale VQ
82.2
77.9
78.4
ARGenSeg-Dij
Intern VL-2.2
Diffusion Head
59.0
62.7
64.1
D.1 Ablation on MILMM Backbone
Our approach, which integrates a VQVAE codebook into the MLLM's token space, is designed to be model-agnostie. To demonstrate this portability, replaced the default InternVL-2.. backbone with LLaVA-1.5 [3]. -LLMA-2-bbsed MLLM. As shown in Tab. 10 our pipeline successfully imparts segmentation capabilities to LLaVA-..5.
As established Sec. [.6 referring segmentation performance highly correlated with the MLLM's underlying understanding ability. Consequently, given ALLV5--..''s weaker understanding capa- bilities compared to InternVL-222 the resulting segmentation performance expectedly lower. Nevertheless, Tab. 10 shows that with the same powerful Intern VL--2.5 backbone, our method out- performs HiMTok. This confirms that our performance gains are inherent to our approach and not merely byproduct of stronger backbone.
24

---

ARGenSeg -DIT
ARGenSag- IntgrnVL
Comparison between direct visual token generation and DiT-based generation. The DiT-based approach, which uses semantic embeddings from the MLLM, struggles with pixel-level accuracy, leading to artifacts like spatial shifts and imprecise boundaries.
Figure 7:
D.2 Ablation on Image Generation Strategy
To further validate our choice of generation strategy, we explore an alternative approach where MLLM outputs semantic embeddings to separate diffusion head (DiT) for segmentation, inspired by MetaQuery [41). Specifically, we configure the MILM to generate learnable queries, which are then mapped to the feature space of the pre-trained SANA-1.5 1. 1.6B [66] via a connector module.
This alternative strategy, labeled as ARGenSeg- DireenSee--TTa in Tab. 10 led severe performance degrada- tion. As show in Fig. 7 while the model could roughly localize the target region, the generated masks suffered from significant artifacts, such as spatial shifts and inflation, indicating poor pixel-level accuracy. This experiment underscores the importance of the MLLM directly generating discrete image tokens maintain the high pixel-level precision crucial for segmentation tasks.
E
Limitations
This paper proposes novel image segmentation paradigm based on autoregressive image generation, integrating multimodal understanding, generation, and image segmentation into a unified framework. Our model demonstrates strong performance across range of segmentation tasks, and further shows the potential to extend to more complex scenarios, such as interactive segmentation and text-to-image generation. The unified framework also shows promise for expanding to broader tasks, such image editing and depth estimation. However, due to resource constraints, exploring these extensions is beyond the scope of this work, and we consider them as promising directions for future research.
F Broader Impacts
This work contributes to the development of unified multimodal frameworks by integrating dense image segmentation into the unified multimodal understanding and generation models. The proposed framework may inspire future research toward more generalizable, modular, and efficient visual- language models that require fewer specific components. Potential applications include human- robot interaction, assistive vision systems, and real-world visual understanding under low supervision. However, like most large-scale models, ARGenSeg may inherit biases from pre-trained components or datasets. Care should be taken evaluate faimess and robustness when deploying it in real-world scenarios, especially in sensitive domains such as healthcare or surveillance.
25


Reviews:
[{"review_id": "MYkCriqVm5", "reviewer": "Reviewer_6HDm", "summary": "The authors propose a unified multimodal understanding and generation model by training MLLMs to predict image tokens and decode into predicted masks. To speed up inference, the authors integrate next-scale prediction, which aligns itself to the segmentation task. Qualitative evaluation shows improved performance on referring segmentation, while allowing down-stream finetuning to interactive segmentation and image generation via small amount of supervised data.", "questions": "My main concerns as below and please refer to the weakness section for more details.\n- Method: What insight can be drawn from the proposed method? How and why are certain design choices (e.g., directly predicting image latent, adding <gen_start> and <gen_end>, etc) made?\n- Evaluation: How significant does this method improve over baseline? What are some particular scenarios where the proposed method consistently outperform baseline?\n\nAs I do not work directly on this topic, I am happy to change my recommendation as I await to read other reviewer’s comments and the author's response to my questions.", "limitations": "Limitations are briefly discussed in the Appendix, though not particularly regarding the proposed method.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- Integrating dense perception tasks with generative models is an important research direction.\n- The work is well-motivated: tasks outputting dense predictions like image segmentation are inherently difficult to process along natural language instructions, making a unified framework an intuitive investigation.\n- The idea of localization first and details second (native for segmentation) fits the auto-regressive scale prediction paradigm well.\n- Quantitate evaluation shows improved performance over baseline methods. Though I am less familiar with past works and cannot judge the completeness of the evaluation benchmarks. \n- Nice extensions to interactive segmentation and image generation via down-stream finetuning. This further verifies the capabilities of the proposed method.\n\n### Weakness\n- The proposed method, although straight-forward, has limited technical novelty and appears more engineering and similar to existing works (e.g., Janus and VARGPT). Though I am less familiar with the existing literature and not best fit to judge the architectural novelty.\n- There is somewhat limited discussions and intuitions in the method section. For instance, the Section 3.2 and 3.3 contain mostly descriptions on the design choice with limited to no insights.\n\t- Discussions on why/how this particular architecture is chosen may improve the scope of the work.\n- No statistical significance is reported. Given how similar some of the performances are in Tab. 2, having statistical significance is important to measure performance improvement.\n- Limited qualitative results in the manuscript &Appendix. In my opinion, the benefit of MLLM comes from its ability to handle inputs in the wild / edge cases and should be highlighted via more examples.", "was_revised": true, "final_justification": "I would like to first thank the authors for their efforts in the new response, as many of my concerns are addressed. While I still find the technical novelty of the proposed method to be lacking, I find the paper to be valuable with enough scope to the broader research community. Therefore, I am inclined to keep my recommendation as borderline accept.", "rebuttal": "Thank you very much for your positive feedback on our work. We apologize for any confusion caused by our writing, and we hope that the following responses will address your concerns.\n\nTo help clarify the architectural novelty of our approach, please allow us to first address your second concern regarding the insights and design of our method.\n\n### **W2: insights and design choices**\n\n**A2:** Thank you for this valuable suggestion. We will add a clearer, more detailed discussion of all design choices and the insights behind them in the final version.\n\nBelow we summarize the key insights that guided our design decisions when integrating segmentation into a unified MLLM framework, and then explain the corresponding architectural choices:\n1. **Insight1:** The strong **comprehension capabilities** of MLLMs can significantly boost segmentation performance, especially for reasoning-intensive tasks.\n- *Design1:* We therefore cast segmentation as an image-generation problem and **embed it into the unified MLLM framework.** (See ablation study, L255–258.)\n\n2. **Insight2:** Unlike general image generation—which primarily emphasizes semantic alignment—segmentation demands **pixel-level accuracy.**\n- *Design2:* Instead of producing semantic embeddings that are later decoded by a downstream model, we let the MLLM **directly predict image tokens.** (See Introduction and Related Work, L42–44 and L83–84.)\n\n3. **Insight3:** Segmentation is a perception task that often requires **faster response times.**\n- *Design3*: We adopt a **multi-scale VQ-VAE tokenizer** that allows all required visual tokens to be predicted in parallel during a single MLLM inference pass. (See Introduction and Table 6, L45–50 and L83–84.)\n4. **Insight4:** We aim to incorporate dense perception tasks into a **unified AGI framework** that already supports multimodal understanding and generation.\n- *Design4:* By solving segmentation via autoregressive image generation, we demonstrate that **dense perception can be seamlessly integrated without specialized segmentation heads,** achieving higher performance and offering a practical path toward unified AGI models. (See Introduction, L24–25.)\n\nIn addition to the points already raised, the following design decisions and analyses may also be helpful:\n- Continuous visual features for input image.\nTo preserve the MLLM’s strong understanding capability (Insight 1), we keep the input visual features continuous. Discrete tokens inevitably degrade understanding performance, which is critical for reasoning-heavy segmentation tasks (see L36–37).\n- Single-stage fine-tuning for efficiency.\nWe adopt a single-stage fine-tuning strategy. Our ablation study (L264–269) shows that adding a separate pre-training stage yields only marginal gains, so we omit it to improve training efficiency.\n\nPlease let us know if any questions remain.\n\n### **W1: technical novelty and differences from Janus and VARGPT**\n\n**A1:** We apologize for any confusion regarding the technical novelty of our work. First, we would like to emphasize that our main motivation is to explore the possibility of unifying dense perceptual tasks within the unified framework of understanding and generation.\n- We successfully demonstrate that an image-generation formulation can indeed solve segmentation and achieve SOTA results, offering a practical route toward unified AGI models. This finding alone carries intrinsic technical value and novelty.\n\nWith respect to the architecture, our design differs fundamentally from both Janus and VARGPT, and these differences are driven by the unique challenges of segmentation:\n1. **Janus vs ours: Janus relies on next-token prediction; in contrast, we adopt next-scale prediction.**\n-  This design originated from *latency challenge for perception tasks*. This allows all tokens of a given scale to be generated **in parallel** within a single forward pass, yielding significantly faster inference.\n   - Under identical settings, replacing our tokenizer with Janus’s VQ-GAN raises per-sample latency from **1.28 s to 5.50 s.**\n2. **VARGPT vs ours: VARGPT appends an extra VAR transformer, while our approach enables the MLLM to directly output visual tokens.**\n- This is designed to tackle the *pixel-level accuracy and consistency challenge.* Unlike VARGPT, which has the MLLM **provide only semantic conditions to the generation head**, our model **directly predicts image tokens**, ensuring better pixel-level precision.\n  - When we experimented with letting the MLLM output semantic embeddings to a DiT decoder, the resulting masks roughly located the target but exhibited noticeable dilation or misalignment—clear evidence of **pixel-accuracy issues.**\n  - The technical report for VARGPT-v1.1 [1] only discusses image editing experiments related to style transfer. The visual results in Figure 9 reveal clear consistency issues, which we suspect stem from the same limitation.\n\n**To the best of our knowledge, we are the first to use multi-scale VQ-VAE visual indices as the MLLM’s native output within a unified understanding-and-generation framework.**\n\n### **W3: statistical significance  to measure performance improvement.**\n\n**A3:** Thank you for raising this important point. We followed the evaluation protocol commonly adopted in the community and therefore did not include formal statistical-significance tests. Below we provide additional evidence that we hope will alleviate any concerns about the reliability of the reported improvements.\n- **Controlled training data.**\nTo ensure a fair comparison, we deliberately used a strict subset of the training data employed by HiMTok—402 K image–segmentation samples versus their 2.91 M (L193–194). This rules out any advantage from additional data.\n- **Training stability.**\nDue to computational cost we did not repeat the main training runs. However, in our ablation study we re-initialized and re-trained the Gen projector multiple times; the resulting performance matched the original checkpoint within 0.2 cIoU, indicating low run-to-run variance.\n- **Deterministic inference.**\nSegmentation results are deterministic under our decoding strategy. We verified this by running inference (i) multiple times on the same checkpoint, (ii) on both single-GPU and multi-GPU setups, and (iii) with both our own re-implementation and the official evaluation script of HiMTok. All runs produced identical cIoU scores. Re-evaluating HiMTok with its official code also reproduced the numbers reported in their paper.\n- **Cross-dataset consistency.**\nOur method yields consistent gains across all benchmark datasets (Table 2). The aggregated metrics further corroborate that the improvements are not artifacts of a particular data split or sampling bias.\n\nTable1 (Average performance on all datasets)\n| Method           | Average   |\n|------------------|-----------|\n| LMMHiMTok-8B     | 77.675    |\n| ARGenSeg         | **79.6625**   |\n| LMMHiMTok-8B(ft) | 81.5875   |\n| ARGenSeg (ft)    | **83.35**     |\n\n### **W4: Scenarios where our method outperforms the baseline.**\n\n**A4:**\nThank you very much for your suggestion. Your point is both correct and valuable, and we will include more qualitative results in the final version of the paper. \n\nFollowing your suggestion, we exhaustively evaluated both HiMTok and ARGenSeg on the test set and visualized cases where ARGenSeg succeeds while HiMTok fails. These visualizations reveal two clear advantages of our coarse-to-fine, multi-scale generation scheme:\n1. **Coarse localization of segmentation objects helps identify the target in multi-object scenarios:**\nOur approach allows for the rough localization of target objects, making it easier to identify the correct segmentation target when multiple similar objects are present in the image.\n- For example, when there are 6 bowls in an image, HiMTok may segment an entirely unrelated object for a simple instruction such as \"bowl middle three red dots,\" while our method can accurately segment the intended target.\n- Other examples include: With 14 hot dogs, the instruction \"left hand row of hot dogs fifth from the front\" and, with 7 people present, \"kid in black arms out.\" In such scenarios, our method is able to localize the correct target more successfully.\n2. **Progressive refinement with multi-scale generation helps enhance mask quality:**\nThe progressive, multi-scale refinement process enables our model to produce higher quality masks once the segmentation target is identified. \n- For instance, in the case of a teddy bear being held, both HiMTok and our method's coarse localization stage only identify a single part separated by an arm. However, during our method’s subsequent fine-grained refinement, the entire teddy bear is correctly segmented, with the arm successfully excluded from the final mask.\n\n### **Q1: Method: insights and design choice**\n\n**A5:**\nWe hope our responses **A2** have fully clarified the motivations and design choices behind our method. Should any questions remain, please do not hesitate to let us know—we are happy to provide further details.\n\n### **Q2: Evaluation:  What are some particular scenarios where the proposed method consistently outperform baseline?**\n\n**A6:**\nPlease refer to **A4.**\n\n[1] Zhuang X, Xie Y, Deng Y, et al. Vargpt-v1. 1: Improve visual autoregressive large unified model via iterative instruction tuning and reinforcement learning[J]. arXiv preprint arXiv:2504.02949, 2025."}, {"review_id": "pSLDW6hN8g", "reviewer": "Reviewer_5AFh", "summary": "This work presents a new model for language-prompted image segmentation. The proposed method employs a pre-trained MLLM model (e.g., InternVL) as the backbone architecture, equipped with newly trained VQ tokens that encodes the segmentation mask to the MLLM latent space, therefore the model can be trained to predict the VQ tokens corresponding to the target mask. A special design of this VQ tokenization design is employing the multi-scale structure (up to k=10 different scales), where the MLLM can iteratively refer to the previously decoded tokens at larger scales (i.e., smaller k's), and produce next tokens at smaller scales (i.e., larger k's).\n\nIn the experiments, the proposed model is compared with several image segmentation models that also utilizes MLLM, and the ablation study is performed on different training strategies of the proposed method.", "questions": "- Please provide more detailed analyses on the VQ tokenization of the masks. How significant is it to train the proposed multi-scale VQ tokens for the segmentation masks? Why can it better perform than more conventional designs to handle masks, such non-discrete features?\n\n- How does the proposed method generalize when other MLLMs are employed? For example, could we reproduce a similar results in llama MLLM architecture?", "limitations": "Despite the use of MLLM backbone, the reasoning ability of the proposed model is not well discussed. Since the training data is the mixture of the conventional image segmentation dataset and the multimodal understanding (without the notion of masks), the model might not retain reasoning ability in terms of the image segmentation tasks.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- Strength of the paper includes easy-to-follow presentation and straightforward model design\n\n- Missing crucial ablation study of the proposed modules, for example, the impact of multi-scale VQ tokenization, compared to the basic single-scale VQ tokenization, or non-VQ mask encoding/decoding methods.\n\n- The method relies on +20 different large-scale datasets for image segmentation, generation, and multi-modal understanding tasks. However, the specific details on the training objective design to handle each dataset and task is insufficiently discussed. Furthermore, as the training dataset of the baseline methods are not clearly compared, it is hard to fully understand whether the improved performances of the proposed method is mainly due to larger training data, or the proposed key design ideas.", "was_revised": true, "final_justification": "I appreciate the authors' comprehensive response to my review. After reviewing the additional experimental results, I find the paper's findings more convincing. As the authors themselves acknowledged, the model's reasoning ability warrants further attention. I encourage them to include their analyses covering different MLLM backbones in the final manuscript.\n\nIn summary, I am updating my score to Borderline accept. I believe this method could be a valuable addition to the research community.", "rebuttal": "Thank you very much for your valuable feedback. Please allow us to first clarify your two key confusions:\n\n1. **We strictly use subsets of the data employed by the previous SOTA method, HiMTok [2], to ensure a fair comparison.**\n- Specifically, we train on 402 K segmentation samples versus HiMTok’s 2.91 M, and 1.25 M multimodal understanding samples versus HiMTok’s 4.2 M. \n\n2. **Our method can be seamlessly extended to any MLLM framework, and all tasks are trained using a unified cross-entropy loss objective.**\n- All tasks—segmentation, visual understanding, and generation—are trained under the same autoregressive paradigm and optimized solely with cross-entropy loss (Fig. 2 and L168–170).\n\n### **W1 & Q1: Ablation study and more detailed analyses on the VQ tokenization of the masks.**\n\n**A1:**\nThank you very much for this insightful suggestion. During the rebuttal period we conducted an ablation study in which we replaced our multi-scale VQ tokenizer with the single-scale VQ-GAN pretrained in Janus [1] and trained it on the same pure-segmentation data. We will include a detailed discussion of these results in the final version of our paper.\n\nOur choice to use multi-scale VQ tokens for training is motivated by the unique challenges posed by the image segmentation task:\n\n1. **We use multi-scale VQ instead of single-scale to meet the latency requirements of perception tasks.**\n- Multi-scale tokens allow the MLLM to predict all visual tokens of the current scale in parallel, dramatically reducing inference time. Under identical hardware and framework settings, the single-scale variant (token-by-token generation) needs **5.50 s per sample**, whereas our **multi-scale design finishes in 1.28 s** (Table 1).\n2. **We enable the MLLM to directly output VQ image tokens, rather than semantic embeddings, to meet the pixel-level precision requirement of segmentation.**\n- Segmentation demands higher low-level and pixel-wise accuracy than generic image generation. We therefore let the MLLM directly predict image tokens instead of producing semantic embeddings for an external decoder. When we instead fed MLLM-generated condition embeddings to a DiT decoder, **the resulting masks suffered from severe pixel inconsistency.**\n\nFor ablation with single-scale VQ tokenization, we used the VQGAN pretrained by Janus as the single-scale VQ tokenizer, and trained it solely with segmentation data. We report the segmentation metric \"giou,\" which represents the per-sample IoU averaged over the dataset.\n- As show in Table 1, the multi-scale scheme not only yields a clear speed advantage but also improves robustness via its inherent coarse-to-fine refinement, as visualized in Fig. 3.\n\nTable1 (Ablation results on VQ tokenizer)\n| Method          | Ref. | Ref.+ | Ref.g | Average | Time   |\n|-----------------|------|-------|-------|---------|--------|\n| Single-scale VQ | 82.1 | 71.8  | 65.8  | 73.23   | 5.50s  |\n| Multi-scale VQ  | 80.5 | 76.7  | 70.4  | 75.87   | 1.28s  |\n\nCompared with non-VQ pipelines that let the MLLM act only as a text encoder and rely on semantic-similarity decoding (L82), our approach performs true pixel-level understanding and significantly outperforms such methods (Table 2 in paper). Moreover, those pipelines break the unification of understanding and generation that our framework is designed to achieve.\n\n### **W2: training objective and datasets scale compare to baseline**\n\n**A2:**\nWe hope our earlier replies have already clarified this point. To summarize succinctly:\n1. **We use an unified objective.**\n- Across all tasks—segmentation, multimodal understanding, and generation—we supervise the model with **cross-entropy loss** under one unified paradigm.  \n  - Image-generation data are employed **only** in the optional Function-Extension stage.\n\n2. **Our data scale is significantly smaller than the compared method.**\n- Our method achieves higher segmentation performance while using significantly less data compared to the previous SOTA method, HiMTok [2] **(402K vs. 2.91M samples).** \n   - This further highlights the effectiveness of our proposed approach. For a direct comparison, the datasets used by HiMTok [2] are listed in Table 1 of their paper.\n\n\n### **Q2: How does the proposed method generalize when other MLLMs are employed?**\n\n**A3:**\nThank you for this valuable question. We will include the following ablation in the final version.\n\n**1. Our method enables image segmentation in MLLM frameworks by incorporating the VQVAE codebook into the MLLM's token space, allowing seamless extension to any MLLM architecture.** \n- In practice, one important detail is that during image generation, the visual tokens predicted by the MLLM are not directly fed as input for the next prediction step. Instead, the predicted image token IDs are mapped to their corresponding visual features in the VQVAE codebook, upsampled to the next scale, and then passed through a Generation Adapter before being provided as input to the MLLM.\n\n**2. We conducted extended experiments using LLaVA-1.5, which is an MLLM framework based on LLaMA-2.** The experimental results show in Table 2.\n- Our pipeline successfully endows the LLaMA-based LLaVA-1.5 with segmentation capability.\n- As noted in our ablation (L255–256), reasoning-segmentation performance is strongly correlated with the MLLM’s understanding ability. \nWhile LLaVA-1.5’s understanding capability is notably weaker than InternVL2.5 (with InternVL2.5 averaging ~68.1 and LLaVA-1.5 ~46 as shown in InternVL2.5’s Figure 1), segmentation results with LLaVA-1.5 are accordingly lower.\nNevertheless, **when the same backbone is used, our method still outperforms HiMTok,** confirming that the gains are not merely due to a stronger backbone.\n\nTable 2 (Extension experiment with LLaVA-1.5 (LLaMA-2 backbone))\n| Method          | backbone    | Ref. | Ref.+ | Ref.g |\n|-----------------|-------------|------|-------|-------|\n| HiMTok          | InternVL2.5 | 81.1 | 77.1  | 75.8  |\n| ARGenSeg-llava  | LLaVA–1.5   | 72.7 | 68.3  | 69.1  |\n| ARGenSeg-internvl | InternVL2.5 | 82.2 | 77.9  | 78.4  |\n\n### **Limitation:  the model might not retain reasoning ability in terms of the image segmentation tasks.**\n\n**A4:**\nThank you for raising this important concern. We shared the same worry and therefore ran targeted experiments to verify whether adding segmentation data would cause the model to be unable to maintain its reasoning ability. **Our findings show that it does not.**\n\n**1. Theoretical justification**\n- During both multimodal reasoning and segmentation training, the model **never treat the mask as input**; the mask is used only as a supervision signal for the perceptual task.\n- In our framework, multimodal understanding and image generation are decoupled at the output stage only: text tokens are decoded by the text tokenizer, while image tokens are decoded by the VQ-VAE decoder. There is no architectural conflict that would degrade reasoning.\n\n**2. Empirical verification**\n- In response to your suggestion, we have further evaluated the model’s reasoning abilities from multiple perspectives. Our experimental results in Table 3 show that adding segmentation data does not cause the model to lose its reasoning capability.\n- Although the absolute scores drop slightly on some benchmarks, we attribute these small decreases not to the segmentation task itself, but to the **much smaller and lower-quality understanding corpus** we used (1.25 M vs. InternVL2.5’s full 16.3 M).\n- To confirm this hypothesis, we fine-tuned InternVL2.5 on understanding data only for an increasing number of steps; performance declined monotonically, mirroring the trend we observe when segmentation data are added. \n  - As explained in L194–195, InternVL’s open-source release only covers version 1.2, limiting the data we can legally use. \n\nTable3 (Ablation experiment of unifying MLLM model reasoning ability)\n| Method          | POPE   | TextVQA |VQAv2 |MMMU-val | AI2D |\n|-----------------|--------|---------|---------|---------|---------|\n| InternVL2.5-ft–1ep | 86.73 | 63.54  |80.40 | 43.7  | 78.7  |\n| InternVL2.5-ft–4ep | 86.01 | 59.73  | 79.28 | 36.8 | 74.8  |\n| ARGenSeg        | 87.57 | 56.98  | 77.87 | 33.4 | 69.6 |\n\n[1] Wu C, Chen X, Wu Z, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 12966-12977.\n\n[2] Wang T, Cheng C, Wang L, et al. Himtok: Learning hierarchical mask tokens for image segmentation with large multimodal model[J]. arXiv preprint arXiv:2503.13026, 2025."}, {"review_id": "pk19NTV5Sn", "reviewer": "Reviewer_pPbW", "summary": "This paper introduces a novel approach to image segmentation by integrating it into a multimodal large language model (MLLM) framework using an auto-regressive generation strategy. The method aims to generate dense segmentation masks by treating them as visual tokens, which are decoded via a universal VQ-VAE detokenizer. To improve efficiency, the authors propose a next-scale prediction strategy that reduces inference latency. The method is evaluated across several segmentation benchmarks.", "questions": "Please see weakness points.", "limitations": "yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n- The paper addresses an emerging and important problem: how to unify dense prediction tasks like segmentation within the auto-regressive MLLM framework.\n\n## Weaknesses\n\n- Motivation is confusing\n    - The overall motivation of the paper is unclear. While the title emphasizes segmentation with auto-regressive models, much of the method closely mirrors existing work (e.g., VAR), with little clarity on what differentiates this approach specifically for segmentation.\n    - If the main contribution is the use of a universal detokenizer for both image generation and segmentation, the benefits of doing so are not convincingly argued. Segmentation masks are typically simpler than natural images, and prior work (e.g., HiMTok) has demonstrated efficient task-specific decoding. The universal approach seems less efficient, as shown in Table 6 where segmentation requires more tokens. The paper should explicitly clarify why a universal detokenizer is preferable and what advantages it offers.\n\n- Details about segmentation part is unclear\n    - Given that segmentation is the focus, the paper lacks insight into how this task is uniquely handled. The current presentation makes it appear as though the VAR framework is simply fine-tuned with segmentation data, which may not constitute a strong methodological contribution.\n    - The integration of segmentation into the auto-regressive MLLM is not clearly described. Without this clarity, it is difficult to assess the novelty and effectiveness of the proposed method. This weakens the interpretability and credibility of the experimental results.\n\n- Some experimental conclusions need further analysis. For example:\n    - Some experimental findings, such as those in Table 5, are not thoroughly analyzed. For example, improvements may stem from dataset-specific artifacts (e.g., overly simplistic language annotations), rather than from the claimed benefits of multi-task training.\n    - The paper does not adequately investigate whether the joint training of segmentation and image generation leads to mutual benefit. Further ablation or control experiments are needed to identify the source of performance gains.\n\n## Conclusion\n\nWhile the paper presents an interesting direction in leveraging auto-regressive MLLMs for segmentation, the contributions are currently underdeveloped. The motivation needs to be clarified, technical integration of segmentation should be described in more detail, and experimental claims require deeper analysis.", "was_revised": true, "final_justification": "As noted in comments, the contribution of this work is overstated and falls short of making a meaningful step toward AGI. The proposed method appears as a combination of existing techniques without clear motivation or conceptual insight. Moreover, the experimental evaluation is insufficient to support the claimed benefits of integrating image generation and segmentation. Given this disconnect between the claims and the evidence, I maintain my original rating.", "rebuttal": "Thank you very much for your positive feedback on our results. We realize that some aspects of our motivation and framework were misunderstood, so allow us to restate them clearly.\n\n**Motivation**\n- Our goal is to **unify dense perception tasks such as segmentation inside a single MLLM framework**, so that multimodal understanding, image generation, and visual perception coexist in one general-purpose framework. To this end, we **re-formulate segmentation as an autoregressive image-generation problem**. \n - Our research demonstrates, for the first time, that dense perception can be integrated into a unified understanding-and-generation model and obtains more advanced performance than methods requiring additional  task-specific segmentation heads, offering a practical route toward unified AGI (L24–25).\n\n**Method**\n- We **extend the MLLM’s vocabulary with the VQ-VAE codebook indices**, enabling the MLLM to directly predict visual tokens for image generation (L144–147).  \n- We adopt a **multi-scale VQ-VAE** to accelerate inference, **not** to be confused with VAR.\n   - VAR is a pure generative model with no multimodal understanding capability, whereas our framework **endows an existing MLLM with both image-generation and segmentation abilities**.\n\n\n\n### **W1.1: Motivation of the paper**\n\n**A1.1:**\nWe hope our earlier clarifications have resolved any confusion regarding the motivation of our work.\n\n**1. We aim to provide a concrete technical demonstration that dense perception can be unified with multimodal understanding and generation inside a single, end-to-end model.** \n- In our work, segmentation (a perception task) is accomplished through generation. This demonstrates a path where understanding and generation are not separate capabilities but are deeply intertwined. We hope this will inspire further research into more holistic, unified architectures.\n\n**2. Our approach is fundamentally different from VAR which is a pure generative model.**\n- VAR lacks image-understanding capabilities and cannot perform complex visual-perception tasks.  \n  - In our framework, the MLLM is the core engine for both understanding and perception. We use image generation via a VQ-VAE simply as the expressive medium for the MLLM to output its perceptual results.\n\n**3. Rather than fine-tuning an existing unified understanding-and-generation model, we made unique design choices specifically to address segmentation-specific challenges.**\n- **Pixel-level accuracy.** Segmentation demands fine-grained precision beyond semantic alignment. Hence we let the MLLM **directly predict image tokens** instead of acting merely as a text encoder that feeds semantic embeddings to external decoders.  \n- **Latency.** Perception tasks require fast response. Multi-scale VQ tokens allow **all tokens of the current scale to be generated in parallel within a single forward pass**, cutting per-sample latency from 5.50 s (single-scale) to 1.28 s.  \n- **Understanding boosts segmentation.** To preserve the MLLM’s reasoning power—crucial for referring segmentation—we keep **continuous visual embeddings** as input features.\n### **W1.2:  Advantages of using a universal tokenizer**\n\n**A1.2:**\nThank you for highlighting the role of the universal tokenizer.\nIts use is directly tied to our motivation: we aim for a single, unified model, so we need a single, unified tokenization scheme that simultaneously enables image segmentation and unlocks image-generation capability.\n\n**1. Using a universal tokenizer enables understanding, segmentation, and generation tasks to be performed simultaneously within a unified MLLM framework.**\n- By avoiding task-specific segmentation heads and instead relying on a general-purpose generative paradigm, our framework can jointly perform multimodal understanding, visual perception, and image generation—offering a practical path toward general-purpose multimodal intelligence (L54–55).\n\n**2. Our multi-scale tokenizer achieves higher segmentation performance with less inference time, demonstrating superior efficiency.**\n- Table 6 shows that, despite producing more tokens, **our multi-scale universal tokenizer requires less inference time and delivers higher segmentation accuracy, confirming its efficiency.**\n- The key message of Table 6 is that multi-scale tokenization yields a clear inference-speed advantage, which is why we adopt it as our universal tokenizer while still gaining the ability to generate images.\n\n### **W2.1: Insight into how segmentation task is uniquely handled**\n\n\n**A2.1**\nWe hope our earlier explanations have resolved this concern; please let us know if any questions remain.\n\n**1. To meet the pixel-level accuracy and latency demands of segmentation, we deliberately designed the framework around direct visual-token prediction by the MLLM and multi-scale image generation.**  \n- To the best of our knowledge, we are the first to use multi-scale VQ-VAE indices as the **native output of an MLLM** within a unified understanding-and-generation pipeline, which constitutes a **methodological contribution.**\n\n**2. Our architecture couples an MLLM with a multi-scale VQ-VAE decoder, endowing the same model with multimodal understanding, segmentation, and generation (Fig. 2).**\n- VAR, in contrast, is a pure image-generation transformer that \n  - (i) lacks multimodal understanding and \n  - (ii) can only ingest a single semantic embedding, \n  - (iii) making it **impossible to fine-tune for segmentation**.\n- We start from a general multimodal LLM, equip it with image-generation capability, and then fine-tune the entire model for segmentation using a **single cross-entropy objective**.\n\n### **W2.2  How segmentation is integrated into the auto-regressive MLLM**\n\n**A2.2**\nAs illustrated in Figure 2, **our model directly employs the MLLM itself for image generation**, and we re-cast segmentation as an image-generation task. In this way, multimodal understanding, segmentation, and image generation are all handled by the same autoregressive MLLM without any external segmentation heads.\n\nTo be explicit:\n- Our research question is how to enable an MLLM—originally designed for understanding—to perform both image generation and segmentation, rather than how to use a standalone generative model for segmentation.\n- If any part of this integration remains unclear, we warmly welcome further discussion and are happy to clarify any details.\n\n### **W3.1 Further analysis experimental conclusions**\n\n**A3.1:**\nWe would like to first emphasize that:\n\n**1. The \"multi-task\" in Table 5 is obtained by jointly fine-tuning the MLLM on multimodal understanding data and segmentation data only; no image-generation data are involved.** \n- Image-generation data are used solely in the Function-Extension stage to demonstrate that the same unified framework can also generate images.\n\n**2. In the paper L255-257, we analyzed the performance improvement from the model's understanding ability from both absolute and relative perspectives.**\n- Absolute gains: Simply adding understanding data yields higher cIoU on all three referring-segmentation benchmarks, confirming that stronger language understanding directly benefits segmentation.\n- Relative gains: RefCOCO+ and RefCOCOg, which contain more semantically challenging referring expressions, enjoy the largest relative improvements.\n\n**3. From another perspective, even considering only the complexity of the language annotations, it is logical that training on a richer and more diverse set of linguistic instructions (from the understanding tasks) would enhance the model's ability to follow the complex commands required for accurate segmentation.**\n\n\n### **W3.2 The joint training of segmentation and image generation**\n\nThank you for this insightful comment.\n\n**1. In the present work we focus on the mutual reinforcement between multimodal understanding and segmentation**\n- Table 4 demonstrates that segmentation training, in turn, benefits visual grounding and reduces hallucination.\n- Table 5 shows that stronger understanding improves segmentation, especially on reasoning-heavy benchmarks.\n\n**2. A systematic study of the interaction between image generation and segmentation is beyond the current scope and is left as future work.**"}, {"review_id": "ApLM3JEutB", "reviewer": "Reviewer_TNWY", "summary": "This paper presents ARGenSeg, a unified framework for image segmentation and generation based on a multimodal large language model. The approach discretizes images and segmentation masks into visual tokens using VQ-VAE, and generates these tokens via VAR. The model produces token sequences for both segmentation and image generation. Experimental results demonstrate reasonable improvements over existing methods.", "questions": "Please refer to the weaknesses for the rebuttal.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. The paper is well written, and the figures and tables are informative.\n\n2. The proposed framework provides some inspiration for segmentation tasks by unifying segmentation and generation within a multimodal model.\n\n3. The method achieves reasonable improvements in segmentation performance and inference speed compared to previous methods.\n\nWeaknesses\n\n1. The paper lacks deep insights. The concept of a unified framework for visual generation and understanding is not novel, and the use of VAR for acceleration is adapted from existing work. The segmentation aspect is not thoroughly explored, as the method appears to treat the segmentation mask as a special case of image generation, but this is not discussed in detail. As a result, the paper lacks an in-depth analysis specific to the unique challenges of segmentation tasks.\n\n2. The Function Extension section lacks thorough quantitative analysis. For example, for interactive segmentation, it would be helpful to compare the proposed method with SAM to better assess ARGenSeg’s generalizability.\n\n3. There is a lack of experimental analysis on high-resolution images. The experiments primarily focus on low-resolution images and do not address the performance of the method on high-quality segmentation over high-resolution images.", "was_revised": true, "final_justification": "The authors' rebuttal addressed some of my concerns by providing additional experiments and clarifications. Therefore, I am keeping my current score of borderline accept.", "rebuttal": "Thank you for your positive assessment of our method. We would like to address your concerns as follows.\n\n### **W1: deep insights and architecture novelty**\n\n**A1:**\nWe sincerely apologize for any confusion regarding the depth of our architectural insights. In the final version we will expand the discussion to make these points explicit.\n\nWe did not simply treat segmentation as yet another generation task and fine-tune an existing unified model. Instead, **we redesigned the unified MLLM around the distinctive challenges of dense perception.** \nIn the course of our research, we have carefully considered the unique challenges of image segmentation, which informed several key design choices in our model:\n\n**1. The strong understanding capabilities of MLLMs are particularly beneficial for image segmentation.**\n- Therefore, we **utilize continuous visual features as input** to preserve the MLLM's strong understanding ability (see L36-37 of the paper).\n\n**2. image segmentation requires higher accuracy and precision at the low-level, pixel-wise scale, whereas standard image generation focuses more on semantic alignment.**\n- Rather than adding an external VAR-style transformer as a generation head (as in VARGPT), or letting the MLLM produce only semantic conditions for downstream models, we **force the MLLM itself to generates the image tokens.**(L42-44, L83-84).\n   - We conducted experiments where the MLLM only outputs semantic conditions to DiT for image generation. The results show that, while the generated segmentation roughly locates target regions, the outputs often suffer from noticeable inflation or shifts—indicative of significant pixel-level accuracy and consistency issues.\n   - The technical report of VARGPT-v1.1[1] discusses image editing but only experiments with style transfer. The visual results (Fig. 9) display marked consistency issues, which we believe stem from the same limitation.\n\n**3. As a perceptual task, image segmentation has higher requirements for response speed.** \n- Therefore, we **adopt a multi-scale image-generation scheme that predict all visual tokens of the current scale in parallel.** (discussed in the Introduction and Table 6: L45-50, L83-84).\n   - A single-scale VQGAN baseline slows inference **from 1.28 s to 5.50 s**, confirming the necessity of this design.\n\n**4. Our goal is to integrate not only multimodal understanding and generation but also visual perception tasks into a unified AGI framework.** \n- For this reason, we do not use specialized segmentation heads such as SAM. Instead, we address image segmentation through an autoregressive image generation approach. \n- Our research demonstrates the feasibility of incorporating dense perceptual tasks into a unified understanding and generation framework, and achieves higher performance compared to approaches requiring additional specialized segmentation heads. \n  - This provides an effective technical pathway and practical foundation for the development of unified AGI models (as discussed in the Introduction: L24-25).\n\nTo the best of our knowledge, **we are the first to integrate multi-scale VQ-VAE visual indices directly into an MLLM’s output space for a unified understanding-and-generation framework.** This design choice was conceived independently and precedes the public release of VARGPT; it was driven by the challenges above rather than adopted from prior work.\n- **We hope this clarifies the novelty of our approach and inspires further research on unified models.**\n\n### **W2: Quantitative result for Interactive segmentation**\n\n**A2:**\nThank you for this insightful suggestion. We will add the requested quantitative results to the final version.\n\n**1.  The quantitative results for interactive segmentation show that our method significantly outperforms SAM.**\n\nBecause the training set of our main baseline (HiMTok [2]) does not contain interactive-segmentation data, we excluded such data from the main experiments to ensure a fair comparison; we only provided a qualitative proof-of-concept in the “Capability Extension” section. Following your advice, we now evaluate on the COCO-Interactive benchmark and report cIoU below. Results for SAM and PSALM are taken from Table 10 of the PSALM [3] paper. Please note that, according to the official code, PSALM was fine-tuned for 10 epochs, whereas we continued fine-tuning for only 1 epoch due to limited compute.\n- The experimental results in Table 1 show that our model significantly outperforms SAM in interactive segmentation. Furthermore, our method achieves performance comparable to PSALM with much less training, which demonstrates the strong generalization ability of ARGenSeg.\n\nTable1 (Quantitative evaluation of interactive segmentation)\n| Method   | Point | Scribble | Box   |\n|----------|-------|----------|-------|\n| SAM-B    | 33.6  | –        | 68.7  |\n| SAM-L    | 37.7  | –        | 71.6  |\n| PSALM    | 74.0  | 80.0     | 80.9  |\n| ARGenSeg | 65.6  | 68.6     | 79.1  |\n\nRegarding image generation, our model learns the generative component from scratch (in contrast to VARGPT, which uses a pre-trained generator). Owing to computational constraints, we stopped training once we verified that the model could indeed unlock image-generation capability.\nWe leave a more thorough exploration of the image generation ability of our model for future work.\n\n### **W3: experimental analysis on high-resolution images.**\n\n**A3:**\nThank you for this valuable suggestion. In the final version we will add discussion on high-resolution segmentation.\n\n**1. ARGenSeg natively supports variable-resolution inputs up to 4 K**\n\nIn our experiments, the model accepts image inputs of arbitrary resolution, with the only restriction being that the output segmentation mask is set to a resolution of $256\\times256$.\n\n- Our MLLM backbone natively supports variable-resolution inputs up to 4 K, so ARGenSeg can directly process high-resolution images without extra preprocessing.\n- Following your advice, we evaluated ARGenSeg on images at $ 2344 \\times 3190 $ resolution. Qualitative results show that the model produces accurate segmentation masks on these high-resolution inputs, confirming its scalability.\n\n[1] Zhuang X, Xie Y, Deng Y, et al. Vargpt-v1. 1: Improve visual autoregressive large unified model via iterative instruction tuning and reinforcement learning[J]. arXiv preprint arXiv:2504.02949, 2025.\n\n[2] Wang T, Cheng C, Wang L, et al. Himtok: Learning hierarchical mask tokens for image segmentation with large multimodal model[J]. arXiv preprint arXiv:2503.13026, 2025.\n\n[3] Zhang Z, Ma Y, Zhang E, et al. Psalm: Pixelwise segmentation with large multi-modal model[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 74-91."}]

Author rebuttal:

Meta-review: This paper proposes ARGenSeg which integrates image segmentation into a unified MLLM framework via autoregressive generation.  The final ratings are 3 borderline accepts and 1 borderline reject.  The reviewers are overall positive about the importance of the problem and technical contribution.  Concerns regarding novelty, experimental reliability, and the need for additional experiments (qualitative results, ablation with single-scale generation, other backbones, interactive segmentation, and high-resolution segmentation) were largely addressed by the rebuttal and discussion. Concerns remained regarding overstated contributions and limited insights.  This is a borderline paper.   The AC carefully reviewed the paper, reviews, rebuttal and discussion, and deem that the positives of the overall framework and promising results outweigh the negatives, and hence recommend acceptance.  Please incorporate the additions during the rebuttal/discussion phase into the final version.
Decision: Accept (poster)