ARGenSeg: Image Segmentation with Autoregressive Image Generation Model
Authors: name: Xiaolong Wang, name: Lixiang Ru, name: Ziyuan Huang, name: Kaixiang Ji, name: Dandan Zheng, name: Jingdong Chen, name: Jun Zhou
Organization: inclusionAI

We propose a novel AutoRegressive Generation-based paradigm for image
Segmentation (ARGenSeg), achieving multimodal understanding and pixel-level
perception within a unified framework. Prior works integrating image
segmentation into multimodal large language models (MLLMs) typically employ
either boundary points representation or dedicated segmentation heads. These
methods rely on discrete representations or semantic prompts fed into
task-specific decoders, which limits the ability of the MLLM to capture
fine-grained visual details. To address these challenges, we introduce a
segmentation framework for MLLM based on image generation, which naturally
produces dense masks for target objects. We leverage MLLM to output visual
tokens and detokenize them into images using an universal VQ-VAE, making the
segmentation fully dependent on the pixel-level understanding of the MLLM. To
reduce inference latency, we employ a next-scale-prediction strategy to
generate required visual tokens in parallel. Extensive experiments demonstrate
that our method surpasses prior state-of-the-art approaches on multiple
segmentation datasets with a remarkable boost in inference speed, while
maintaining strong understanding capabilities.