Title: ARGenSeg: Image Segmentation with Autoregressive Image Generation Model
Venue: NeurIPS 2025
Accepted: yes
Keywords: image segmentation

unified multimodal understanding and pixel-level perception

image generation
Abstract: We propose a novel AutoRegressive Generation-based paradigm for image Segmentation (ARGenSeg), achieving multimodal understanding and pixel-level perception within a unified framework. 
Prior works integrating image segmentation into multimodal large language models (MLLMs) typically employ either boundary points representation or dedicated segmentation heads.
These methods rely on discrete representations or semantic prompts fed into task-specific decoders, which limits the ability of the MLLM to capture fine-grained visual details.
To address these challenges, 
we introduce a segmentation framework for MLLM based on image generation, which naturally produces dense masks for target objects.
We leverage MLLM to output visual tokens and detokenize them into images using an universal VQ-VAE, 
making the segmentation fully dependent on the pixel-level understanding of the MLLM.
To reduce inference latency, 
we employ a next-scale-prediction strategy to generate required visual tokens in parallel.
Extensive experiments demonstrate that our method surpasses prior state-of-the-art approaches on multiple segmentation datasets with a remarkable boost in inference speed, while maintaining strong understanding capabilities.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents ARGenSeg, a unified framework for image segmentation and generation based on a multimodal large language model. The approach discretizes images and segmentation masks into visual tokens using VQ-VAE, and generates these tokens via VAR. The model produces token sequences for both segmentation and image generation. Experimental results demonstrate reasonable improvements over existing methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The paper is well written, and the figures and tables are informative.\n\n2. The proposed framework provides some inspiration for segmentation tasks by unifying segmentation and generation within a multimodal model.\n\n3. The method achieves reasonable improvements in segmentation performance and inference speed compared to previous methods.\n\nWeaknesses\n\n1. The paper lacks deep insights. The concept of a unified framework for visual generation and understanding is not novel, and the use of VAR for acceleration is adapted from existing work. The segmentation aspect is not thoroughly explored, as the method appears to treat the segmentation mask as a special case of image generation, but this is not discussed in detail. As a result, the paper lacks an in-depth analysis specific to the unique challenges of segmentation tasks.\n\n2. The Function Extension section lacks thorough quantitative analysis. For example, for interactive segmentation, it would be helpful to compare the proposed method with SAM to better assess ARGenSeg’s generalizability.\n\n3. There is a lack of experimental analysis on high-resolution images. The experiments primarily focus on low-resolution images and do not address the performance of the method on high-quality segmentation over high-resolution images.\n\nQUESTIONS:\nPlease refer to the weaknesses for the rebuttal.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to image segmentation by integrating it into a multimodal large language model (MLLM) framework using an auto-regressive generation strategy. The method aims to generate dense segmentation masks by treating them as visual tokens, which are decoded via a universal VQ-VAE detokenizer. To improve efficiency, the authors propose a next-scale prediction strategy that reduces inference latency. The method is evaluated across several segmentation benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper addresses an emerging and important problem: how to unify dense prediction tasks like segmentation within the auto-regressive MLLM framework.\n\n## Weaknesses\n\n- Motivation is confusing\n    - The overall motivation of the paper is unclear. While the title emphasizes segmentation with auto-regressive models, much of the method closely mirrors existing work (e.g., VAR), with little clarity on what differentiates this approach specifically for segmentation.\n    - If the main contribution is the use of a universal detokenizer for both image generation and segmentation, the benefits of doing so are not convincingly argued. Segmentation masks are typically simpler than natural images, and prior work (e.g., HiMTok) has demonstrated efficient task-specific decoding. The universal approach seems less efficient, as shown in Table 6 where segmentation requires more tokens. The paper should explicitly clarify why a universal detokenizer is preferable and what advantages it offers.\n\n- Details about segmentation part is unclear\n    - Given that segmentation is the focus, the paper lacks insight into how this task is uniquely handled. The current presentation makes it appear as though the VAR framework is simply fine-tuned with segmentation data, which may not constitute a strong methodological contribution.\n    - The integration of segmentation into the auto-regressive MLLM is not clearly described. Without this clarity, it is difficult to assess the novelty and effectiveness of the proposed method. This weakens the interpretability and credibility of the experimental results.\n\n- Some experimental conclusions need further analysis. For example:\n    - Some experimental findings, such as those in Table 5, are not thoroughly analyzed. For example, improvements may stem from dataset-specific artifacts (e.g., overly simplistic language annotations), rather than from the claimed benefits of multi-task training.\n    - The paper does not adequately investigate whether the joint training of segmentation and image generation leads to mutual benefit. Further ablation or control experiments are needed to identify the source of performance gains.\n\n## Conclusion\n\nWhile the paper presents an interesting direction in leveraging auto-regressive MLLMs for segmentation, the contributions are currently underdeveloped. The motivation needs to be clarified, technical integration of segmentation should be described in more detail, and experimental claims require deeper analysis.\n\nQUESTIONS:\nPlease see weakness points.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a new model for language-prompted image segmentation. The proposed method employs a pre-trained MLLM model (e.g., InternVL) as the backbone architecture, equipped with newly trained VQ tokens that encodes the segmentation mask to the MLLM latent space, therefore the model can be trained to predict the VQ tokens corresponding to the target mask. A special design of this VQ tokenization design is employing the multi-scale structure (up to k=10 different scales), where the MLLM can iteratively refer to the previously decoded tokens at larger scales (i.e., smaller k's), and produce next tokens at smaller scales (i.e., larger k's).\n\nIn the experiments, the proposed model is compared with several image segmentation models that also utilizes MLLM, and the ablation study is performed on different training strategies of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n- Strength of the paper includes easy-to-follow presentation and straightforward model design\n\n- Missing crucial ablation study of the proposed modules, for example, the impact of multi-scale VQ tokenization, compared to the basic single-scale VQ tokenization, or non-VQ mask encoding/decoding methods.\n\n- The method relies on +20 different large-scale datasets for image segmentation, generation, and multi-modal understanding tasks. However, the specific details on the training objective design to handle each dataset and task is insufficiently discussed. Furthermore, as the training dataset of the baseline methods are not clearly compared, it is hard to fully understand whether the improved performances of the proposed method is mainly due to larger training data, or the proposed key design ideas.\n\nQUESTIONS:\n- Please provide more detailed analyses on the VQ tokenization of the masks. How significant is it to train the proposed multi-scale VQ tokens for the segmentation masks? Why can it better perform than more conventional designs to handle masks, such non-discrete features?\n\n- How does the proposed method generalize when other MLLMs are employed? For example, could we reproduce a similar results in llama MLLM architecture?\n\nLIMITATIONS:\nDespite the use of MLLM backbone, the reasoning ability of the proposed model is not well discussed. Since the training data is the mixture of the conventional image segmentation dataset and the multimodal understanding (without the notion of masks), the model might not retain reasoning ability in terms of the image segmentation tasks.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a unified multimodal understanding and generation model by training MLLMs to predict image tokens and decode into predicted masks. To speed up inference, the authors integrate next-scale prediction, which aligns itself to the segmentation task. Qualitative evaluation shows improved performance on referring segmentation, while allowing down-stream finetuning to interactive segmentation and image generation via small amount of supervised data.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- Integrating dense perception tasks with generative models is an important research direction.\n- The work is well-motivated: tasks outputting dense predictions like image segmentation are inherently difficult to process along natural language instructions, making a unified framework an intuitive investigation.\n- The idea of localization first and details second (native for segmentation) fits the auto-regressive scale prediction paradigm well.\n- Quantitate evaluation shows improved performance over baseline methods. Though I am less familiar with past works and cannot judge the completeness of the evaluation benchmarks. \n- Nice extensions to interactive segmentation and image generation via down-stream finetuning. This further verifies the capabilities of the proposed method.\n\n### Weakness\n- The proposed method, although straight-forward, has limited technical novelty and appears more engineering and similar to existing works (e.g., Janus and VARGPT). Though I am less familiar with the existing literature and not best fit to judge the architectural novelty.\n- There is somewhat limited discussions and intuitions in the method section. For instance, the Section 3.2 and 3.3 contain mostly descriptions on the design choice with limited to no insights.\n\t- Discussions on why/how this particular architecture is chosen may improve the scope of the work.\n- No statistical significance is reported. Given how similar some of the performances are in Tab. 2, having statistical significance is important to measure performance improvement.\n- Limited qualitative results in the manuscript &Appendix. In my opinion, the benefit of MLLM comes from its ability to handle inputs in the wild / edge cases and should be highlighted via more examples.\n\nQUESTIONS:\nMy main concerns as below and please refer to the weakness section for more details.\n- Method: What insight can be drawn from the proposed method? How and why are certain design choices (e.g., directly predicting image latent, adding <gen_start> and <gen_end>, etc) made?\n- Evaluation: How significant does this method improve over baseline? What are some particular scenarios where the proposed method consistently outperform baseline?\n\nAs I do not work directly on this topic, I am happy to change my recommendation as I await to read other reviewer’s comments and the author's response to my questions.\n\nLIMITATIONS:\nLimitations are briefly discussed in the Appendix, though not particularly regarding the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}