Title: Seg-VAR:Image Segmentation with Visual Autoregressive Modeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: image segmentation

autoregressive modeling

reinforcement learning
Abstract: While visual autoregressive modeling (VAR) strategies have shed light on image generation with the autoregressive models, their potential for segmentation, a task that requires precise low-level spatial perception, remains unexplored. Inspired by the multi-scale modeling of classic Mask2Former-based models, we propose Seg-VAR, a novel framework that rethinks segmentation as a conditional autoregressive mask generation problem. This is achieved by replacing the discriminative learning with the latent learning process. Specifically, our method incorporates three core components: (1) an image encoder generating latent priors from input images, (2) a spatial-aware seglat (a latent expression of segmentation mask) encoder that maps segmentation masks into discrete latent tokens using a location-sensitive color mapping to distinguish instances, and (3) a decoder reconstructing masks from these latents. A multi-stage training strategy is introduced: first learning seglat representations via image-seglat joint training, then refining latent transformations, and finally aligning image-encoder-derived latents with seglat distributions. Experiments show Seg-VAR outperforms previous discriminative and generative methods on various segmentation tasks and validation benchmarks. By framing segmentation as a sequential hierarchical prediction task, Seg-VAR opens new avenues for integrating autoregressive reasoning into spatial-aware vision systems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Seg-VAR, a novel autoregressive framework for image segmentation tasks. It rethinks segmentation as a conditional mask generation problem and introduces a spatial-aware latent representation of segmentation masks called seglats. These seglats, along with a multi-stage training strategy and hierarchical autoregressive decoding, allow the model to capture spatial and instance-level structures more effectively. The method demonstrates strong empirical performance across semantic, instance, and panoptic segmentation tasks.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*:\n\n1. The proposed autoregressive approach to segmentation is conceptually compelling. The idea of disentangling semantic features from mask representations via seglats is both intuitive and well-motivated.\n1. The spatial-aware color mapping mechanism for encoding instance-level information into seglats offers a creative and effective solution to a core challenge in generative segmentation—distinguishing overlapping or adjacent instances.\n1. The experimental results are strong: Seg-VAR achieves state-of-the-art performance across semantic, instance, and panoptic segmentation tasks on COCO, ADE20K, and Cityscapes, demonstrating the generalizability and robustness of the approach.\n\n*Weaknesses*:\n\n1. Insufficient component-level ablation: While Table 5 highlights the effect of major training stages, the paper lacks finer-grained ablation studies isolating the contributions of individual modules such as the seglat encoder/decoder or latent decoder/decoder. The comparison to alternative generative backbones (e.g., SD-XL, DALL·E 2, VQGAN) primarily supports the use of VAR itself, but does not clearly attribute the performance gains to the proposed seglat design. Although the supplementary includes FID comparisons, it would strengthen the work to show segmentation performance using vanilla VAR (i.e., without the seglat module).\n1. Pipeline complexity and resource demands: The framework involves multiple interacting components—image encoder, seglat encoder/decoder, latent encoder/decoder—which likely require significant computational resources for training.\n1. Terminology and Figure Clarity: The term Latent Encoder is potentially misleading. According to the paper, its main function is to assign spatially-informed colors to segmentation masks based on instance centroids—more akin to a handcrafted spatial embedding mechanism than a latent feature encoder. If the primary role is color assignment via spatial rules, the use of transformers and the need for learning are not well justified. Additionally, Figure 1 lacks clarity: while Line 132 states that the segmentation masks are binary, the figure shows input images instead of segmentation masks or their encoded seglat representations. A clearer illustration of how seglats are generated and used would greatly aid understanding.\n\nQUESTIONS:\n1. How sensitive is the performance to the choice of grid size or color palette used in the spatial-aware color mapping? The paper uses a grid-based mapping to assign instance-specific colors, but it's unclear how robust this is to changes in grid granularity or color discretization.\n1. Why are both a seglat decoder and a latent decoder needed? The model has two decoding stages—one for seglats, another for final masks. Could a single decoding stage suffice if trained end-to-end?\n1. What is the role of the [CLS] and [TYP] tokens in the seglat encoder/decoder, and how much do they contribute to performance? These tokens are introduced without much analysis or ablation.\n\nLIMITATIONS:\nThe primary limitation of this method lies in its architectural complexity and computational cost. The framework includes multiple encoders and decoders—image encoder, seglat encoder/decoder, and latent encoder/decoder—but the necessity of each component is not clearly justified. A more thorough discussion or ablation of these modules would help clarify whether the full complexity is essential for the observed performance gains.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an innovative architecture for image segmentation under an autoregressive paradigm. Specifically, the authors design three components: 1) an image latent encoder, 2) a feature representation called \"seglat\" for encoding segmentation results, and 3) an autoregressive generator/decoder. Under this framework, the image segmentation task is modeled as a next-token prediction problem. The authors conduct extensive experiments to validate the effectiveness of the proposed algorithm.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. **Clear narrative structure**: The paper presents a coherent storyline with well-aligned claims and supporting evidence.\n\n2. **Superior performance**: The model achieves excellent accuracy with significant performance improvements compared to baseline methods. The ablation studies are comprehensive and well-designed.\n\n3. **Novel paradigm**: Although the approach draws inspiration from VAR and ControlVAR, the proposed autoregressive modeling paradigm represents an innovative contribution to the field of image segmentation.\n\n**Weaknesses:**\n\n1. **Insufficient mathematical formulation**: The seglat representation appears to be the core contribution of this work, and the authors mention that this component references the maskige design from GSS. However, the paper lacks formal mathematical expressions for the seglat modeling process, which hampers the comprehensibility of the proposed method.\n\n2. **Open vocabulary capability unclear**: Given this seglat representation and the complete modeling paradigm, I want to know how well the model performs on open vocabulary segmentation tasks.\n\n3. **Parameter efficiency concerns**: While the model demonstrates significant performance improvements, a lingering question remains: how would the performance compare if the parameter count were kept consistent with the comparison methods?\n\nQUESTIONS:\nI rate this article very highly, and would be happy to improve it further if the author could answer my small questions with clarity.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates visual autoregressive (VAR) modeling for image segmentation. The authors propose a novel encoder-decoder framework designed to repurpose the architecture and infrastructure of generative models to handle image segmentation tasks. Their approach leverages autoregressive modeling to sequentially generate segmentation masks, demonstrating improved performance compared to state-of-the-art discriminative segmentation models such as MaskFormer and DETR. The core contribution is the adaptation of generative modeling principles to segmentation. Learning the generative process underlying segmentation masks may provide more nuanced representations than traditional discriminative methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The central idea of reusing generative model architectures for segmentation is both novel and interesting. This direction can potentially bridge the gap between generative and discriminative paradigms in vision, enabling deeper insight into the structure of segmentation masks.\n- The proposed method achieves better performance than current state-of-the-art segmentation models across several benchmarks, indicating its practical value and significance.\n\nWeaknesses\n- The paper is often unclear and challenging to follow, particularly in the method section. The presentation of the framework, especially regarding the generative model and tokenization process, lacks sufficient detail to ensure reproducibility.\n\t- The connection between the proposed segmentation latent encoder (seglat) and established VAR models (which typically rely on multi-scale residual tokenization) is not clearly articulated. It remains ambiguous whether the proposed latent representations inherit the multi-scale and residual design principles from classic VARs.\n- The manuscript suffers from awkward phrasing, grammatical errors, and poor structural organization, which impedes comprehension and technical assessment.\n- The role of the generative model component is under-explained. It is unclear if the framework uses a pure autoregressive transformer, a next-scale prediction model, or another architecture. The relationship between the autoencoder, tokenization scheme, and segmentation mask reconstruction needs clarification.\n\nQUESTIONS:\n1.\tClarification on Generative Model:\nCould the authors elaborate on the generative modeling aspect of Seg-VAR? Specifically, does the model use a pure autoregressive transformer, a next-scale prediction VAR, or some hybrid? More detail about the architecture (e.g., sampling procedure, training objective) would be helpful.\n2.\tTokenization and Latent Representation:\nHow is the latent encoder for segmentation related to the traditional VAR tokenization scheme? Does Seg-VAR employ a multi-scale, residual tokenization process, and if so, how are these latent tokens structured and used?\n3.\tConnection to Image Autoencoders:\nIs the segmentation autoencoder in this framework structurally similar to classic VQ-VAE or other image autoencoders? How is the mapping between segmentation masks and discrete latent tokens performed, and does it incorporate spatial or instance-aware encoding?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework that casts image segmentation as a conditional autoregressive latent mask generation problem instead of a standard discriminative or diffusion pipeline. It introduces a spatial-aware encoder  that encodes masks into discrete tokens \"seglats\" with position-aware color mapping, an autoregressive transformer decoder -- that sequentially generates mask tokens -- and a multi-stage training regime that aligns image priors with the mask latent space. Experiments on COCO, Cityscapes, and ADE20K show consistent improvements over Mask2Former and prior generative baselines.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- novel formulation: well-motivated argument that sequential autoregression can better model mask dependencies than purely parallel discriminative predictions.\n- unified framework: similar to prior recent work, the same backbone and decoder work for semantic, instance, and panoptic tasks, needing no task-specific heads.\n- empirical improvement: Consistent point gains in all PQ/AP/mIoU across standard benchmarks.\n- careful ablations: The supplementary shows the impact of seglat encoding, latent alignment, and the autoregressive decoder clearly.\n\n## Weaknesses\n- heavy compute cost: The design is substantially larger than Mask2Former and MaskFormer in both parameters and FLOPs (e.g., Swin-L variant: 522M params vs. 216M, 1320G vs. 868G). Inference speed (FPS) is lower due to sequential decoding and additional modules. While the latent space reduces output dimensionality, the net cost is higher.\n- multi-stage complexity: The full pipeline depends on careful multi-stage training, including pretraining of seglat encoder/decoder, latent modules, and final alignment. This is more demanding than standard discriminative setups.\n- though the authors mark ‘statistical significance’ section in the checklist as ‘yes’, but the paper does not report significance tests, confidence intervals, or error bars to support this claim.\n\nQUESTIONS:\n- how sensitive is the approach to the grid size and colormap encoding in the seglat encoding? Could learned positional embeddings work equally well or is this strictly better? also, the colormap, though taken from prior work, could be expanded in the supplementary for completion \n- the compute intensity is much higher and inference is slower due to autogressive nature, therefore, can the sequential inference be parallelized to offset this?\n- similarly, due to higher inference cost, are there other practical advantages to the autoregressive formulation -- e.g. you mentioned extending to videos\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}