Seg-VAR:Image Segmentation with Visual Autoregressive Modeling
Authors: rongkun Zheng
Lu Qi
Xi Chen
Yi Wang
Kun Wang
Hengshuang Zhao
Venue: NeurIPS 2025 poster ()
Keywords: image segmentation
autoregressive modeling
reinforcement learning
Primary area: applications
TL;DR: we propose Seg-VAR, a novel framework that rethinks segmentation as a conditional autoregressive mask generation problem

Abstract: While visual autoregressive modeling (VAR) strategies have shed light on image generation with the autoregressive models, their potential for segmentation, a task that requires precise low-level spatial perception, remains unexplored. Inspired by the multi-scale modeling of classic Mask2Former-based models, we propose Seg-VAR, a novel framework that rethinks segmentation as a conditional autoregressive mask generation problem. This is achieved by replacing the discriminative learning with the latent learning process. Specifically, our method incorporates three core components: (1) an image encoder generating latent priors from input images, (2) a spatial-aware seglat (a latent expression of segmentation mask) encoder that maps segmentation masks into discrete latent tokens using a location-sensitive color mapping to distinguish instances, and (3) a decoder reconstructing masks from these latents. A multi-stage training strategy is introduced: first learning seglat representations via image-seglat joint training, then refining latent transformations, and finally aligning image-encoder-derived latents with seglat distributions. Experiments show Seg-VAR outperforms previous discriminative and generative methods on various segmentation tasks and validation benchmarks. By framing segmentation as a sequential hierarchical prediction task, Seg-VAR opens new avenues for integrating autoregressive reasoning into spatial-aware vision systems.

Seg-VAR: Image Segmentation with Visual Autoregressive Modeling
Rongkun Zheng'
Lu Qi Xi Chen' Yi Wang Kun Wang" Hengshuang Zhao'!
The University of Hong Kong Ilnsta360 Shanghai Artificial Intelligence Laboratory
SShangaai Innovation Institute "SenseTime Research {zrk228connect, hszhao@cs} .hku.kk hk
Abstract
While visual autoregressive modeling (VAR) strategies have shed light on image generation with the autoregressive models, their potential for segmentation, a task that requires precise low-level spatial perception, remains unexplored. Inspired by the multi-scale modeling of classic 2ask2Former-basel models, we propose Seg- VAR, novel framework that rethinks segmentation as conditional autoregressive mask generation problem. This is achieved by replacing the discriminative learning with the latent learning process. Specifically, our method incorporates three core components: (I) an image encoder generating latent priors from input images, (2) spatial-aware seglat (a latent expression of segmentation mask) encoder that maps segmentation masks into discrete latent tokens using location-sensitive color mapping to distinguish instances, and (3) decoder reconstructing masks from these latents. A multi-stage training strategy is introduced: first learning seglat representations via image-seglat joint training, then refining latent transforma- tions, and finally aligning image-encoder-deee eddeeedrreeddeee latents with seglat distributions. Experiments show Seg- VAR outperforms previous discriminative and generative methods various segmentation tasks and validation benchmarks. By framing segmentation as a sequential hierarchical prediction task, VAg opens new avenues for integrating autoregressive reasoning into spatial-aware vision systems.
Introduction
Image segmentation the task of partitioning pixels into semantically meaningful regions- requiress models to capture hierarchical spatial relationships, from coarse object categories to fine-grained instance boundaries. While advancements in convolutional and wranfoommr-bassee architectures have pushed performance on semantic, instance, and panoptic segmentation, these approaches often treat segmentation parallel prediction task, struggling to model the iterative, context-dependent spatial and semantic relationships in complex scenarios. Recent work in visual autoregressive (VAR [58]) modeling, which sequences images into tokens for generative tasks, offers promising alternative: its sequential, context-accumulating nature could naturally capture the progressive refinement inherent segmentation. However, existing VAR frameworks prioritize image synthesis, neglecting their potential to unify segmentation tasks through structured spatial autoregression.
A key obstacle lies in representation: most autoregressive frameworks encode images into latent spaces that lack explicit spatial or instance-level structure. For example, while Generative Semantic Segmentation (GSS [5]) learns latent distributions to guide segmentation, its encoder fails to disam- biguate overlapping instances or preserve fine-grained positional cues. Conversely, autoregressive image generators typically treat pixels patches unordered tokens, sacrificing the geometric
CCorresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Discriminative
Condi
Diffusion
Encode
Decode
Image
Pred.
Image
Pred.
Image
VAR
Pred.
Model
tion
Model
(a) Discriminative segmentation
(b) Previous generative segmentation
(c) Seg- VAR (Ours)
Figure I: Our Seg- VAR is visual autoregressive model that is designed for generic image segmen- tation. Different from (a) traditional discriminative segmentation models and (b) diffusion based generative models that mainly take input image as condition, our Seg- VAR rethinks segmentation as a sequence modeling task by encoding the input image latent distribution and generating the masks hierarchically.
control needed when generating images that demand strong spatial relationship, such as behind or next to. Bridging this gap requires a VAR framework that (1) decomposes images into hierarchi- cal, position-aware tokens to represent objects at multiple scales, and (2) leverages autoregressive dependencies to propagate spatial coherence across these tokens.
In this work, we introduce Seg- VAR, visual autoregressive model that is designed for generic image segmentation (semantic, instance, and panoptic). Seg-VAR modeling and employs a hybrid design that combines hierarchical autoregressive decoding with next- scale prediction principles. As shown in Fig. 1, our approach rethinks segmentation conditional autoregressive mask generation task, where discrete tokens encode both semantic classes and instance- aware positional information. Our approach hinges multiple innovations: 1) Spatial-aware seglat encoding: We introduce seglats-latent representations segmentation maaks--generate via a novel encoder that maps masks to discrete tokens using location-sensitvve color mapping. This mechanism assigns unique RGB values to instances based on their spatial centroids, enabling transformers to distinguish overlapping objects through positional awareness. 2) Hierarchical autoregressive decoder reconstructs masks by sequentially predicting seglat tokens conditioned image features, ensuring spatial coherence through autoregressive attention. This mimics human-like iterative refinement, where early tokens establish global context and later tokens resolve local ambiguities. 3) Multi-stage latent alignment: A three-stage training strategy first learns seglat-image correlations, refines latent transformations, and finally aligns image-derived priors with seglat distributions via KL divergence minimization.
VAR is built on visual autoregressive (VAR)
decoding: A transformer-biat
By training Seg VAR to maximize the likelihood of ground-truth token sequences-wwite jointly optimizing pixel-level mask fidelity-the model learns to harmonize semantic accuracy with geo- metric consistency. Experiments demonstrate state-of-the-at performance on COCO, Cityscapes, and ADE20K. with significant gains in occluded scenes and small-object segmentation. Notably, Beg-VAR's 's autoregressive tokenization generalizes across segmentation tasks: the same architecture achieves top-tier results in semantic, instance, and panoptic settings, showcasing VAR's 's versatility as a unified paradigm for spatial understanding. Our contributions are as follows:
We analyze the limitations of existing VAR-based and discriminative methods and propose a framework named Seg-VAR with autoregressive modelling that reconsiders segmentation conditional mask generation problem.
We develop two critical strategies: Spatial-aware seglat encoding and image-seglat joint training. These designs enable our Seg-VAR to be adaptable for three segmentation settings.
We conduct extensive experimental evaluations on challenging image segmentation bench- marks, including COCO, Cityscapes, and ADE20K, and the achieved state-of-the-an results demonstrate the effectiveness and generality of the proposed approach and shed new light the autoregressive modeling segmentation strategy.
2
Related Work
Image segmentation models. Since the inception of FCN [46], semantic segmentation have flour- ished by various deep neural networks with the ability to classify each pixel. The follow-up meth- ods then change focus to improve the limited receptive field of these models. PSPNet [78] and DeepLabV2 [7] aggregate multi-scale context between convolution layers. Sequentially, Nonlo-

---

Latent Encoder
Latent Decoder
Seglat
Seglat
Seglat
Encoder & Decoder
Segmentation Mask
Recon. Loss
Recon. Mask
Image
Recon. Image
Illustration of the latent and seglat learning (9s. pe) of proposed Seg- VAR. We first jointly model the seglat and image during training in the seglat encoder and decoder (red module). Then, with the well-trained encoder and decoder, we try to optimize the latent encoder and decoder (green module). Worth mentioning, we use different color in the binary segmentation mask to highlight different instances of the image.
Figure 2:
cal [66], CCNet [30], and DGMN [76] integrate the attention mechanism in the convolution structure. Later on, Transformer-based methods (e.g. SETR [79] and Segformer (72]) are proposed following the introduction of Vision Transformers. More recently, MaskFormer [15] and Mask2Former [14] realize semantic segmentation with bipartite matching.
Specialized instance segmentation architectures are typically based upon "mask classification." They predict a of binary masks each associated with a single class label. The pioneering work, Mask R-CNN [27], generates masks from detected bounding boxes. Follow-up methods either focus on detecting more precise bounding boxes [3, 6], finding new ways to generate dynamic number of masks, e.g., using dynamic kernels [60, 67, 1] clustering algorithms [35, 12]. Although the performance has been advanced in each task, these specialized innovations lack the flexibility to generalize from one to the other, leading to duplicated research effort. For instance, although multiple approaches have been proposed for building feature pyramid representations [43], as we show in our experiments, BiFPN [55] performs better for instance segmentation while FaPN [29] performs better for semantic segmentation.
Panoptic segmentation has been proposed to unify both semantic and instance segmentation tasks [34]. Architectures for panoptic segmentation either combine the best of specialized semantic and instance segmentation architectures into a single framework [73, 33, 12, 41] or design novel objectives that equally treat semantic regions and instance objects [4, 65]. Despite those new architectures, researchers continue to develop specialized architectures for different image segmentation tasks [53, 24]. We find panoptic architectures usually only report performance on single panoptic segmentation other tasks. For example, panoptic
task [65], which does not guarantee good performance segmentation does not measure architectures' abilitics to rank predictions as instance segmentations. Instead, here, we evaluate our VAR on all studied tasks to guarantce generalizability. Commonly, all the methods adopt the discriminative pixel-wise classification learning paradigm. This in contrast to our generative image segmentation.
Autoregressive models. Autoregressive models, leveraging the powerful scaling capabilities of LLMs [51, 2, 17, 61, 62], use discrete image tokenizers [63, 52, 21] in conjunction with transformers to generate images based on next-token prediction. VQ-based methods [63, 52, 21, 38, 54] employ vector quantization to convert image patches into index-wise tokens and use decoder-only trans- former to predict the next token index. However, these methods are limited by the lack of scaled-up transformers and the quantization eπrоr inherent in VQ-VAE [63], preventing them from achieving performance on par with diffusion models. Parti [74], Emu3 [68], chameleon [57], loong [69] and VideoPoet [37] scaled up autoregressive models in text-to-image video synthesis. Inspired by the global structure of visual information, Visual AutoRegressive modeling (VAR) redefines the autoregressive modeling images next-scale prediction framework, significantly improving gen- eration quality and sampling speed. HART [56] adopted hybrid tokenizers based on VAR. Fluid [23] proposed random-order models and employed continuous tokenizer rather than discrete tokenizer.
Generative models for visual perception. Image-to-image translation made one of the earliest attempts in generative segmentation, with far less success in performance [31]. Some good results

---

Image
Image Encoder
Latent Distribution
Latent Loss
Condition
Seglat Encoder
GT Latent Distribution
Soglat
Figure 3: Illustration of the latent learning (pe) of proposed Seg- VAR. order learn the latent representations, Seg- VAR optimizes the image encoder while freezing the seglat encoder. We also introduce the latent loss minimize the divergence between latent distributions for inference.
were achieved in limited scenarios such as face parts segmentation and Chest X-ray segmentation [39]. Replacing the discriminative classifier with generative Gaussian Mixture model, GMMSeg [42] is claimed as generative segmentation, but the most still of discriminative modeling. The promising performance of Pix2Seq [10] on several vision tasks leads to the prevalence of sequence-to-seequeee lask-agnostic vision frameworks. For example, Unified-I/O [48] supports a variety of vision tasks within a single model by segentializing each task to sentences. Pix2Seq-D [9] deploys hierarchical
VAE (i.e. diffusion model) to generate panoptic segmentation masks. This method is inefficient due the need for iterative denoising. UViM [36] realizes its generative panoptic segmentation by introducing latent variable conditioned on input images. It is also computationally heavy due to the need for model training from scratch. To address these issues, GSS introduces notion maskige for expressing segmentation masks in the form of RGB images, enabling the adoption of off-the-shelf data representation models (e.g. VGVAE) already pretrained on vast diverse imagery. However, the transformation of maskige is a simple MLP which restricts GSS from identifying the specific instances. Thus, we propose Seg-VAR with location- designs and hierarchical autoregressive modeling that solves the dilemma.
3 Method
3.1 Preliminaries
Conventionally, image segmentation can be formulated as discriminative learning problem depend- ing on the form of tasks:
max log Px (c x),
(Semantic Segmentation)
max log Pa (c, y x))
(Instance Segmentation)
9
max log Po Po (Cault,, Cthings x)). (Panoptic Segmentation)
where x E #R is an input image, E {0, M is a segmentation mask in K semantic categories, N is the instance number identifier, and pr, Po,P, Ps are the discriminative pixel classifiers, Focusing on learning the classification boundary of input pixels, this approach enjoys high data and training efficiency [49].
In this work, we based our Seg- on GSS [5] by introducing a discrete L-dimension latent distribution (=|c) (with 3 the above log-likelihood as:
logp(c/x) > Egg(()))) log p(z, clz)
(=(c)
which is known as the Evidence Lower Bound (EIBO) [32], and we can easily expand it to instance and panoptic segmentation settings due the chain rule and the independence of these variables (details are given in the supplementary material). And the ELBO can be written in the form of:
Ego(ele) [logre(c]3) DEL (go(2lc), (zlxx))

---

Image
Pred. Mask
Image Encoder
Seglat Decoder
Latent Decoder
Figure Illustration the inference stage. The latent distribution generated by the image encoder is fed to the seglat decoder to generate the predicted seglat, and then finally generates the final prediction.
where we have three components in our formulation: (1) Pai An image encoder (denoted Ig) that generates the prior distribution of latent tokens 2 conditioned on the input image z. (2) go: A representative encoding function that encodes the semantic segmentation mask e into discrete latent tokens 2, which includes seglat encoder (denoted as &g, implemented by VAR [64]) and latent encoder (Tş) that is built up with attention modules). (3) po: A function that decodes the semantic segmentation mask from the discrete latent tokens which includes seglat decoder (denoted De. implemented by VAR decoder [64]) and a latent decoder ((Ta)).
3.2 Overall Architecture
As shown in Fig. 2, Seg--eer mainly contains several modules: image encoder (Iz)), seglat encoder and decoder (E., Pg), and latent encoder as well as decoder (Tş, Te).
Image Encoder. Zu is comprised of an image backbone (ResNet [28], Swin Transformer [45], etc.) and Multi-scale fusion module. Multi-scale fusion is implemented with transformer layers and projection layer. The output of the Z.p is the latent token E H//
Latent Encoder and Decoder. Latent encoder To on the other hand, is responsible for transforming the segmentation masks M R xN into corresponding seglats S TR xxxx3 We use transformer layers to generate the desired seglats. Thus, seglats can be viewed as kind of RGB image. What's more, in order to spatially-sensitive, we implement colormap encoder y that converts the binary segmentation mask M E {0, 1) 10 x xN into an additional colormap M. E IR HW x as:
M. = (MM),
where N denotes the number of instances. M, initialized to a value and then assigned to the corresponding color for each instance area by the spatial-aware color mapping. Inspired by grids, where each grid has an uniquely-assignet color. Each instance area is associated with these fixed colors if their gravity centers fall in the grids. To better distinguish the color difference, we select candidate values (Ó, 51, 102, 153, 204, 255) for each RGB channel (if the categories are less than 124, then 1,,,,,22,,9922222 | is more preferred). Thus, the overall color number 215 63 (color (0,0,0) indicates the background). And the grid number a² should be less than 215. This location-awain aware color mapping can be effective because transformer design has position encoding that can help predict the corresponding colors. On the other hand, hand-crafted random assigning color will struggle to distinguish instances because of the large color space. Then, we concat into the seglat encoder. Unlike standard VAR, seglat tokens incorporate spatial information via color mapping. This mapping assigns unique RGB values to instances based on their centroids (gridded into a*a regions), enabling the transformer to distinguish overlapping objects through positional cues.
UniGS [50]. an image is partitioned into
with Ma. and the output is the final seglat feed
Seglat Encoder and Decoder. Here, we adapt the design of ControlVAR (our design shares similarities with VQ- VAE but specialized for segmentation, where we add extra image controls into modeling), and jointly model the image and seglat in each stage of the transformer structure. Let 21,2.... K) denote scale the hierarchical transformer structure (where S is the total number of scales). At scale k; image tokens and seglat tokens are tokenized by shared tokenizer &: Image features are X₃ E [V] and Seglats are S₁ E [V] / where V indicates the vocabulary size. A flatten operation is adopted to convert the sequence of 2D features into ID. Full attention is
5

---

enabled for both control and image tokens belonging to the same scale, which allows the model to maintain spatial locality and to exploit the global context between control and image:
X S = Attention( Xx: Sk: Sk).
(4)
Full attention is adopted for both seglat and image tokens within the same scale, so that the Seg-VAR maintain the spatial locality and excavate the global connection within seglat and the input image. What's more, we employ the [CLS] and [TYP] as two pre-defined special tokens as start tokens. These two tokens are required to distinguish between different segmentation tasks. [CLS] provide semantic context for the generated image (categories), and [TYP) token is used to select the type of segmentation tasks. Both tokens vital to Seg- VAR and can not be ablated. A standard cross entropy loss is utilized to supervise the seglat encoder and decoder in the reconstruction process.
3.3 Multi-stage Training
Our training can be divided into several stages: 1) the learning of the seglat encoder and decoder (E,, Dz). 2) the learning of the latent encoder and decoder (To, Ta). 3) the image encoder (In)). Namely, the first and second stage are illustrated in Fig. 2 (which is the posterior modules of latent tokens), while the third stage is shown in Fig. (which is the prior module of latent tokens). We now delve into details of the multiple stages of posterior and prior training process.
Stage 1: Seglat Learning. In the first stage is mainly about the joint training of image with seglat, which is discussed in ControlVAR, except that we novelly change the control signal to a unique type of RGB image, seglat.
Stage 2: Latent Learning, In the second stage, the training is mainly to optimize:
min R. (xle) (c|=) cll- 8.0
However, we introduce the latent encoder and decoder to transform the c, and now we have:
min E (eITT (2(2) Te(c))l + min E (eIT( (I (D)) cll. 3,8
where (c(c) S, which is the seglat (we consider as the latent representation of segmentation mask, and name this for convenience). The first term refers to image reconstruction, while the second term can be simplified as:
min Ei (EIT (T() - cll.
(6)
where the training parameters far less than training VAR from seratch.
Stage 3: Image Encoder Learning. As shown in Fig. 3, the optimization target of this stage should be minimize the distance between the distribution predicted by In and the latent codes generated by seglat encoder (VAR). We use the cross entropy loss to measure the distance of this distance:
min DE (20()))), v
3.4 Inference
As illustrated in Fig. 4, during inference, we first take the latent tokens = that are predicted by the image encoder Ig. and feed them into the latent decoder D& to generate the predicted latent concatenation $(e), Next, we apply the inverse transform decoding to the predicted latents obtain the final segmentation mask é.
Experiments
In the first part, we present the evaluation datasets and metries. Then we present the training settings. Finally, we demonstrate that Seg- Seg-VAR is an effective generative architecture for universal image segmentation through comparisons with different specialized methods on standard benchmarks. We evaluate our model on all three tasks, which all obtain the state-of-the-an results.
Datasets. We study Seg-VAR using four widely used image segmentation datasets that support semantic, instance and panoptic segmentation: COCO [44] (80 "things" and 53 "stuff" categories),

---

Method
Backhone
PQ
PQ
PQ"
APp
minDpas
#params..
FL.PPs
fps
DETR [4]
R,50
43.4
48.2
36.3
31.1
MaskFonner [16]
R50
46.5
51.0
39.8
33.0
181G
Mask2Former [14]
R50
51.9
57.7
43.0
41.7
61.7
44M
226G
8.6
Sog-VAR
R50
54.1
60.1
45.8
44.3
64.2
315M
605G
5.2
DETR [4]
R101
45.1
50.5
37.0
33.0
MaskFormer [16]
R101
47.6
52.5
40.3
34.1
59.3
64M
248G
14.0
Mask2Former
R101
52.6
58.5
43.7
42.6
62.4
63M
293G
7.2
Seg-VAR
R101
54.7
60.4
46.2
44.5
64.6
335M
624G
4.6
Max DeepLab [65]
Max
51.1
57.0
42.2
451M
3692G
MaskFormer [16]
Swin-L
52.7
58.5
44.0
40.1
64.8
2/2M
792G
5.2
Swin-L
Mask2Former [14]
Swin-1.7
57.8
64.2
48.1
48.6
67.4
216M
868G
4.0
[5]
Swin-L
44.9
50.2
32.6
36.9
54.2
386M
1142G
3.4
Seg-VAR
Swin-
59.7
65.6
50.5
49.6
68.7
522M
1320G
3.2
Table 1: Panoptic segmentation on COCO panoptic va12017 with 133 categories. Seg- VAR consistently outperforms Fask2Former [14] by a large margin all metries. Our best model outperforms prior state-of-the-aa models by 1.9 PQ and GSS [77] by 14.8 PQ. Backbones pre-trained on ImageNet-22K, marked with t.
Method
Backhone
Ap$
ApM
ApL.
Aphondaay
tparams.
FLOPP
fps
MaskFormer [16]
R50
34.0
16.4
37.8
23.0
181G
19.2
Mask CNN (27)
R50
37.2
18.6
39.5
53.3
23.1
44M
201G
15.2
Mask R-CNN [27]
R50
42.5
23.8
45.0
60.0
28.0
46M
358G
10,3
Mask2Former
30.6
2260
Seg. VAR
R50
45.8
25.2
49.8
68.1
33.4
315M
605G
5.9
Mask R-CNN [27]
R101
38.6
19.5
41.3
55.3
24.5
63M
266G
10.8
Mask R-CNN [27]
R101
43.7
24.6
46.4
61.8
29.1
65M
423G
8.6
Mask2Forner [1I]
47.7
Seg- VAR
R101
46.5
25.2
49.6
70.1
34.6
335M
624G
5.2
QueryInst [24]
Swin L
33.55
Swin-HTC++ [6]
Swin-1.f
49.5
31.0
52.4
67.2
34.1
284M
1470G
Mask2Former
Swin-1
50.1
29.9
53.9
72.1
36.2
216M
868G
Seg-VAR
Swin
52.7
31.2
55.2
75.4
39.4
522M
1320G
Table 2: Instance segmentation on COCO va12017 with categories. Seg- VAR outperforms strong Mask2Former [14] baselines for both AP and A Ahhunndary [13] metrics. Our Seg. VAR surpasses Fask2Former by a huge 2.6 AP on the largest backbone Swin-L., and demonstrates superior perfor- mance across all metrics and backbones. Backbones pre-trained ImageNet-22K marked with t,
ADE20K [80] (100 "things" and 50 "stuff" categories), and Cityscapes [18] (8 "things" and 11 "stuff" categorics). Panoptic and semantic segmentation tasks are evaluated on the union of "things" and "stuff" categories, while instance segmentation only evaluated the "things" categories.
Evaluation metrics. For panoptic segmentation, we use the standard PQ (panoptic quality) met- ric [34]. We further report APTH
pan: which is the AP evaluated on the "thing" categories using instance segmentation annotations, and mIoU which the mloU for semantic segmentation by merging instance masks from the same category, of the same model trained only with panoptic segmentation annotations. For instance segmentation, we use the standard AP (average precision) metric [44]. For semantic segmentation, use mloU (mean Intersection-over- Union) [22].

---

Method
Pretrain
Backbone
Iteration
mloU
Method
Pretrain
Backhone
Iteration
mioU
Discriminativve modeling:
Discriminative modeling:
FCN [46]
PSPNet [78]
1K
ResNet-101
ResNet-101 101
80k
80K
79.77
77.02
FCN
ResNet-101
160k
41.40
NonLocal [66]
DeepLab-x3+ [8]
IK
ResNet-101
ResNet-101
80k
80K
79,40
80.65
CCNet [30]
DANet [25]
IK
ResNet-101
ResNet- 101
160k
160k
44.17
43.71
CCNet [30]
Maskformer [15]
1K
ResNet-101
ResNet-101 101
90k
80K
78.50
79.45
Deeplab-v3+ [8]
UperNet [7]]
IK
ResNet-101
ResNet- 101
160k
160k
45.47
43.82
Musk2former [14]
1K
ResNet-101
90K
80.10
Musktoomer [15]
Mask2former [1-]
1K
IK
ResNet-101
ResNet- 101
160k
160k
47,80
45,50
UperNet [21]
SegFormer [72]
SETR
Mask2former [14]
22K
22K
22K
IK
Swin-Large
Swin-Large
ViT-Large
MiT-B5
160k
80k
90k
80k
83.30
82.8
82.25
SegFormer [72]
OCRNet [75]
[79]
IK
HRNet.W48
MiT BB
arg-
160k
160k
160k
43.25
50.08
48.28
Generative modeling:
Generative modeling:
UVIM! [36]
22k
Swin Large
160%
43.71
UVIM [36]
22K
Swin-Large
160k
70.77
GSS-FF [S]
Swin-Large
160%
46.29
GSS-FF[S]
22K
Swin-Large
80k
78.90
GSS-FT-W [5]
22K
Swin-Large
160k
48.54
GSS-FT-W
Swin-Large
80k
80.05
Seg-VAR
22K
Swin-Large
160k
54.90
Scg-VAR
22K
Swin-Large
80k
85.82
Table 4: Semantic Segmentation comparison with previous art methods on the ADE20K val split. "IK" means pretrained on ImageNet IK [19] while "22K" means pretrained on Ima- geNet 22K [19]. Our model surpasses previous state-of-the-arr by 4.82 mloU, demonstrating the effectiveness of Seg- VAR.
Table 3: Semantie Segmentation on the Cityscapes val split: UViMi [36] is reproduced on PyTorch. "IK" means pretrained on ImageNet 1K [19] while "22K" means pretrained on Ima- geNet 22K [19]. Our model surpasses previous state-of-the-art by 2.52 mloU, demonstrating the effectiveness of Seg-VAR.
4.1 Training settings
Panoptic and instance segmentation. We operate all experiments with 8 V100 GPUs. We use Detectron2 [70] and follow the updated Mask R-CNN [27] baseline settings for the COCO dataset. More specifically, we use Adam [47] optimizer and the step learning rate schedule. We use an initial learning rate of 0.0001 and a weight decay 0.05 for all backbones. learning rate multiplier of 0.1 is applied to the backbone and decay the learning rate at 0.9 and 0.95 fractions of the total number of training steps by a factor of 10. Training iterations are also reported in all experimental figures. For data augmentation, we use the large-scale jittering (LSJ) augmentation [26, 20] with a random scale sampled from the range 0.1 to 2.0 followed by a fixed size crop to 1024 x 1024. We use the standard Mask R-CNN inference setting where resize image with shorter side to 800 and longer side up-to 1333. We also report FLOPs and fps. FLOPs are averaged over 100 validation images (COCO images have varying sizes). Frames-per-seccne (fps) is measured V100 GPU with batch size of by taking the average runtime on the entire validation set including post-processing time.
Semantic segmentation. We follow the same settings as [11] to train our models, except: 1) a learning rate multiplier of 0.1 is applied to both CNN and Transformer backbones instead of only applying it to CNN backbones in [16], 2) both ResNet and Swin backbones use an initial learning rate of 0.0001 and weight decay of 0.05, instead of using different learning rates in [16].
VAR modeling. We follow VAR [59] and ControlVAR [40]. During training, we leverage the pre-trained VAR tokenizer to tokenize seglat and control. The training details follow the strategy in ConrrolVAR, which refers to an approach of sampling both pixel- and token-level controls for image generation with teacher-forcing guidance. For inference, we utilize top-k top-p sampling with k=900 and p=0.96 for encoding and decoding the seglat. As for the training objectives, the training objective is based on the Evidence Lower Bound (ELBO), optimizing three components: (1) mask reconstruction loss via seglat decoder, (2) KL divergence to align image-derived latents with seglat distributions, and (3) cross-entropy loss for token prediction.
4.2 Main results
Panoptic segmentation. In Table. 1, we compare Seg- VAR with state-of-the-art models for panoptic segmentation on the COCO panoptic [34] dataset validation split. Seg- consistently outperforms Fask2Former by 1.9. With Swin-L backbone, Seg-V.s VAR sets new ofate-of-the-ar of 59.7
8

---

Seglat Leamning Img. Enc. Learning
ADE20K COCO
Generation Model
ADE20K
COCO
Stage
Stage
mloU
VQGAN
74.6
42.8
78.9
DALL-E2
80.2
47.9
2
83.4
52.0
SD-XI.
81.8
48.9
81.6
49.3
VAR
85.8
52.7
85.8
52.7
Table 6: Ablation on different generation models. We experimented on different image generation models, the results indicate the supe- riority of VAR in segmentation tasks.
Table 5: Ablation on the key design of Seg- VA R.
These results demonstrate the effectiveness of our designs and training strategy.
PQ, outperforming existing stateofftthearrt [14] by 1.9 PQ and generative method GSS by 14.8 PQ. This indicates the effectiveness of our jointly modeling strategy with specially-designel generative encoders and decoders, which successfully encode localization information as well as instance information. GSS, on the other hand, fails to identify different instances effectively.
Beyond the PQ metric, our Seg- Seg also achieves higher performance on two other metrics compared Mask2Former: APTh pan- which the AP evaluated on the 80 "thing" categories using instance segmentation annotation, and mloUpmm pun. which is the mloU evaluated on the 133 categories for semantic segmentation converted from panoptic segmentation annotation. This shows Seg- VAR's universality: Even trained only with panoptic segmentation annotations, it can be used for instance, and semantic segmentation.
Instance segmentation. We compare Seg- VAR with state-of-the-art models on the COCO [44] dataset in Table. 2. With the Swin-L backbone, Seg-VAR outperforms the -fate-of-the-art Mask2Former by 2.6 AP and 3.2 A.hisudurry On other backbones, including R50 and R101, our Seg-VAR still shows superiority over previous approaches across all metrics (2.3 AP and 2.1 AP, respectively). These results further validate the efficacy of our jointly hierarchical modeling strategy with location-ave awarre generative segmentation latent encoders and decoders, which successfully encode localization information as well as instance information.
Semantic segmentation. We compare VAR with SOTA models for semantic segmentation on the Cityscapes [18] dataset in Table. 3. With the Swin-L backbone, Seg-VAR outperforms previous SOTA methods, including Mask2Former [14] with 2.52 increase in fewer training iterations, and huge boost of 5.77 mloU compared to the previous generative segmentation model GSS.
We also compare Seg-VAR with state-of-the-art models for semantic segmentation the ADE20K [80] dataset Table. Seg-VAR outperforms previous SOTA methods, including Mask2Former [14] with increase of 7.1 mloU and SegFormer with 4.82 improvement. What's more, our Seg-VAR VAR outperforms GSS with a 6.4 increase, which is large margin. This should credit to the modeling and novel design in latent encoders and decoders.
The consistent superiority of our framework across both datasets (Table. 3, 4) empirically validates its capacity to reconcile structural priors with discriminative feature learning. These results highlight the critical role of our architectural innovations, particularly synergistic design of latent encoders for disentangled representation learning and decoders for geometry-awan refinement, in advancing semantic segmentation performance.
4.3 Ablation studies
In this part, we ablate various key designs our Seg- VAR from different aspects, ranging from the ablation of key designs, choice of generation models, parameter efficiency, and hyper parameters of grid size and palette.
Key design of Seg- VAR. In Table. 5, we demonstrate the effectiveness of our key components design and corresponding multi-stage training strategy. stage represents seglat encoder/decoder,, stage refers to latent encoder and decoder, and stage 3 represents image encoder learning. Since latent learning core idea of Seg- VAR, we keep the latent encoder and decoder in ablation studies (in Table. 7). As shown in the table, with the seglat learning strategy implemented, the performance

---

Method
Dataset
mIoU
Method
Dataset mloU
Grid Number mloU Palette Size mloU
4
84.4
124
85,4
Vanilla VAR ADE20K 77.4
Mask2Former
R50
63.8
85.2
215
85.8
Seg-VAR ADE20K 85.8
Seg- VAR
R50
64.2
12
85.8 342
85.3
Table 7: Ablation on the key design of Seg-VAR, These re- sults demonstrate that simply us- ing vanilla VAR without seglat modules, the result is 8,4 lower than our Seg-VAR.
Table 8: Ablation on the pa-
Table 9: Ablation on grid num-
rameters. These results demon- strate that under comparable pa- rameters, our Seg-VAt VAR still out- performs Mask2Former in the R50 backbone.
ber and palette size. We exper- imented on different settings of grid number and palette size, the results indicate the robustness of these variations.
improves greatly, showing 4.5 mloU and 5.8 AP enhancement. The image encoder learning strategy also shows great effectiveness with 2.7 and 3.1 mloU improvement, respectively. By implementing these designs, our model is capable of harmonize semantic accuracy with geometric consistency.
Different generation model designs. In Table. 6, we show that VAR is better than previous VQ-VAE and diffusion- basoo generation models. With VAR as the encoder, our model surpasses SD-XL by 4.0 mloU and 3.8 AP. This indicate that VAR is capable of being implemented in general image segmentation tasks for its superior structure of autoregressive modeling.
Seglats designs. To explicitly validate the role of seglats, we conducted additional experiments using a "vanilla VAR" baseline (remove latent encoder and decoder, and using plain VAR). As shown the Table. 7, simply using vanilla VAR without seglat modules and tested on ADE2OK, the result is 8.4 lower than our Seg- VAR. This indicate the importance of latent learning strategy in our Seg-VAR. VAR.
Model parameters. In Table. 8, we examined the model parameter efficiency. To better compare our model with traditional discriminative segmentation models, adjust the parameters of Faskerormer by extending its transformer layer number so that the parameters can be comparable. As shown in the table, our Seg- VAR still outperforms Mask2Former by 0.4 mloU in COCO panoptic dataset.
Sensitivity of grid size and colormap. We evaluate the robustness of grid number and Palette on ADE20K. As shown in the Table. 9, the performance decreases the number decreases, because the granularity can help model better distinguish instances. As for the palette size of colormap, we discover that 6 is the optimal number because we have to balance between large color space and loss of generality. (Ideally, the size should be larger than the category). We find that grid numbers more sensitive than palette size, but they still contribute the performance gain. These results confirm that VAR is robust to reasonable variations in grid/color size.
Conclusions
In this work, analyze the limitations of existing VAR-based and discriminative methods and propose a framework named Seg-VAR VAR with autoregressive modelling that reconsiders segmentation conditional mask generation problem. We develop two critical strategies: Spatial-aware seglat encoding and image-seglat joint training. These designs enable our Seg- VAR to be adaptable for three segmentation settings. By decomposing segmentation into a coarse-to-fing token prediction process, Seg-VAR VAR bridges the gap between autoregressive modeling's 's sequential dependency learning and segmentation's 's demand for precise spatial reasoning. Our experiments demonstrate that autoregressive methods, long dominant in generation tasks, can rival and even surpass parallel architectures in segmentation accuracy.
Limitations and Broader Impact. Even though our model demonstrates great potential in generating high-quality segmentation masks, its application to video domains is yet to be discovered. Also, due to the memory of image generation models, the memory cost is larger than transformer-based segmentation models. As for broader impact, we believe our work lay foundation for future works in unifying generation and perception tasks.
Acknowledgement. This work is supported by the National Natural Science Foundation of China (No. 62422606, 62201484, 624B2124) and the computation resources provided by Shanghai Artificial Intelligence Laboratory.
10

---

References
[1] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lec. YOLACT++: Better real-time instance segmentation, 2019.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[3] Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: Delving into high quality object
detection. CVPR, 2018.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 3,
[5] Jiaqi Chen, Jiachen Lu, Xiatian Zhu, and Li Zhang. Generative semantic segmentation. In CVPR, pages 7111-7120, 2023. 1, 4, 7,8
[6] Kai Chen, Jiangmiao Pang. Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, al. Hybrid task cascade for instance segmentation. In CVPR, 2019. 7,
[7] Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Semantic image segmentation with deep convolutional nets and fully connected erfs, arXiv preprint 14X: 7062, 2014.
[8] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
[9] Ting Chen, Lala Li, Saurabh Saxena, Geoffrey Hinton, and David J Fleet. A generalist framework for panoptic segmentation of images and videos. arXiv preprint, 2022, 4
[10] Ting Chen, Saurabh Saxena, Lala Li, David Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. arXiv preprini arXiv:2109. 10852, 2021. 4
[11] Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexan- der G Schwing. Mask2former for video instance segmentation. 1rXiv:2112.10764, 2021,
8
[12] Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chinn Chich Chen. Ranoptic-DeepLab A simple, strong, and fast baseline for bottom-up panoptic segmentation. In CVPR, 2020. 3
[13] Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C Berg, and Alexander Kirillov. Boundary iou: Improving object-centric image segmentation evaluation. In CVPR, 2021.
[14] Bowen Cheng, Ishan Misra, Alexander Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022. 3, 7, 8,
[15] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. 2021. 3,
[16] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurIPS, 2021.
[17] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra. Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1 113, 2023.
[18] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding, In CVPR, 2016. 7,9

---

[19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009,
[20] Xianzhi Du, Barret Zoph, Wei-Chih Hung, and Tsung- Yi Lin. Simple training strategies and model scaling for object detection, arXiv preprint arXiv:2107.00057, 2021. 8
[21] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873-12883, 2021.
[22] Mark Everingham, SM Ali Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes challenge: A retrospective. IJCV, 2015.
[23] Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive ext-tto-mmae generative models with continuous tokens. arXiv preprint arXiv:2410.13866 2024. 3
[24] Yuxin Fang, Shusheng Yang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Instances queries. In ICCV, 2021. 3,7
[25] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, 2019. 8
[26] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D Cubuk, Quoe V Le, and Barret Zoph. Simple copy-pas opy-paste is a strong data augmentation method for instance segmentation. In CVPR, 2021. 8
[27] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017. 3,
7.8
[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[29] Shihua Huang, Zhichao Lu, Ran Cheng, and Cheng He. Fapn: Feature- alitune-aligeed pyramid network for dense image prediction. In ICCV, pages 864-873 2021.
[30] Zilong Huang. Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In ICCV, 2019. 3, 8
[31] Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
[32] Diederík P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint, 2013.
[33] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, 2019.
[34] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In CVPR, 2019. 3, 7,7.8
[35] Alexander Kirillov, Evgeny Levinkov, Bjoern Andres, Bogdan Savchynskyy, and Carsten Rother. InstanceCut: from edges to instances with multicut. In CVPR, 2017. 3
[36] Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. Uvim: A unified modeling approach for vision with learned guiding codes.
arXiv preprint, 2022. 4, 8
[37] Dan Kondratyuk. Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hormung, Vighnesh Birodkar, Jimmy Yan, Ming-Cham Chiu, et al. Vidcopoet: A large language model for zero-shot video generation. arXiv preprint arxiv:2312. 14122,...11,, 2023. 3
[38] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR, pages 11523-11532, 2022. 3
12

---

[39] Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralaa, and Sanja Fidler. Semantic segmentä- tion with generative models: Semi-supervised learning and strong out-of-domain generalization. In CVPR, 2021. 4
[40] Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Zhe Lin, Rita Singh, and Bhiksha Raj. Controlvar: Exploring controllable visual autoregressive modeling. arXiv preprint arxiv:2406.097500 2024.
8
[41] Yanwei Li, Hengshuang Zhao, Xiaojuan Qi, Yukang Chen, Lu Qi, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia. Fully convolutional networks for panoptic segmentation with point-based supervision. arXiv preprint 1rxiv:2108.0777
2021. 3
[42] Chen Liang, Wenguan Wang, Jiaxu Miao, and Yi Yang. Gmmseg: Gaussian mixture based generative semantic segmentation models. 2022. 4
[43] Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017. 3
[44] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 6, 7,9
[45] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. ICCV, 2021.
[46] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic
segmentation. In CVPR, 2015. 2, 8
[47] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 8
[48] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. In ICLR, 2022. 4
[49] Kevin P Murphy, Machine learning: probabilistic perspective. MIT press, 2012. 4
[50] Lu Qi, Lehan Yang. Weidong Guo, Yu Xu, Bo Du, Varun Jampani, and Ming-Hsuun Yang. Unigs: Unified representation for image generation and segmentation. In CVPR, 2024. 5
[51] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al.
Language models are unsupervised multitask learners. OpenAl blog. 188):9, 2019.
[52] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural informution processing systems, 32, 2019. 3
[53] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, 2021. 3
[54] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024. 3
[55] Mingxing Tan, Ruoming Pang, and Quoe V Le. Efficientdet: Scalable and efficient object detection. In CVPR, 2020.
[56) Haotian Tang, Yecheng Wu, Shang Yang, Enze Xie, Junsong Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Yao Lu, and Song Han. Hart: Efficient visual generation with hybrid autore- gressive transformer. arXiv preprint arXiv:2410. 10812, 2024. 3
[57] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.098/6 2024.7
[58] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances neural information processing systems, 37:84839-84888 2024. 1
13

---

[59] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive mod- eling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.0290) 2024. 8
[60] Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In ECCV, 2020. 3
[61] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023. 3
[62] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[63] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[64] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. 2017. 5
[65] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chich Chen, MaX-DeepLab: End-to-end panoptic segmentation with mask transformers. In CVPR, 2021. 3, 7
[66] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018. 8,8
[67] Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. SOLOv2: Dynamic and fast instance segmentation. NeurIPS, 2020.
[68] Xinlong Wang, Xiaosong Zhang. Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409. 18869, 2024.
[69] Yuqing Wang, Tianwei Xiong, Daquan Zhou, Zhijie Lin, Yang Zhao, Bingyi Kang, Jiashi Feng, and Xihui Liu. Loong: Generating minute-level long videos with autoregressive language models, arXiv preprint arXiv:2410.02757, 2024. 3
[70] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan Yen Lo, and Ross Girshick. Detectron2. https:// gi thub. com/ http:///cccooeeeeeeee/eettcooooo 2019. 8
[71] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018. 8
[72] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurlPS, 2021. 3.8 8
[73] Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun. Upsnet: unified panoptic segmentation network. In CVPR, 2019. 3
[74] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Bureu Karagol Ayan, et Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3):5, 2022.
[75] Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation. In ECCV, 2020. 8
[76] Li Zhang, Dan Xu, Anurag Arnab, and Philip HS Tor. Dynamic graph message passing networks. In CVPR, 2020.
[77] Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. In NeurIPS, 2021.
14

---

[78] Hengshuang Zhao, Jianping Shi. Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. CVPR, 2017. 2, 8
[79]
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang, Philip HS Torr, al. Rethinking semantic segmentation from sequence-to-sequee
perspective with transformers. In CVPR, 2021. 3, 8
[80] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In CVPR, 2017. 7,9
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We include claims and contributions in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitation in the end.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
We have no proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [No]
Justification:
We will disclose the code after submission and acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.o cc/ httee/guiiess//ooeeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececccccccccct http:///ggiieeeeeeeuuuubiiiiiiii
/CodeSubmie sswonnoolll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include these details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our results have been experimented with multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes] We provide the information.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: : /ttp///eerrpp.ee/ee//ee cc cc/ cc/ppbiccc/tticccsseeeleeeee.eee
didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification:
Yes we preserve anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, include it.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
19

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We don't 't include this.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we do.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
20

---

Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
we don't 't't involve this.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Don't involve this.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
21

---

Justification:
We describe its usage.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: : http///rrpss.ccoooofeeeeeeeeee/e for what should or should not be described.
22


Reviews:
[{"review_id": "CmuqvYw91g", "reviewer": "Reviewer_2JK9", "summary": "This paper proposes a framework that casts image segmentation as a conditional autoregressive latent mask generation problem instead of a standard discriminative or diffusion pipeline. It introduces a spatial-aware encoder  that encodes masks into discrete tokens \"seglats\" with position-aware color mapping, an autoregressive transformer decoder -- that sequentially generates mask tokens -- and a multi-stage training regime that aligns image priors with the mask latent space. Experiments on COCO, Cityscapes, and ADE20K show consistent improvements over Mask2Former and prior generative baselines.", "questions": "- how sensitive is the approach to the grid size and colormap encoding in the seglat encoding? Could learned positional embeddings work equally well or is this strictly better? also, the colormap, though taken from prior work, could be expanded in the supplementary for completion \n- the compute intensity is much higher and inference is slower due to autogressive nature, therefore, can the sequential inference be parallelized to offset this?\n- similarly, due to higher inference cost, are there other practical advantages to the autoregressive formulation -- e.g. you mentioned extending to videos", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n- novel formulation: well-motivated argument that sequential autoregression can better model mask dependencies than purely parallel discriminative predictions.\n- unified framework: similar to prior recent work, the same backbone and decoder work for semantic, instance, and panoptic tasks, needing no task-specific heads.\n- empirical improvement: Consistent point gains in all PQ/AP/mIoU across standard benchmarks.\n- careful ablations: The supplementary shows the impact of seglat encoding, latent alignment, and the autoregressive decoder clearly.\n\n## Weaknesses\n- heavy compute cost: The design is substantially larger than Mask2Former and MaskFormer in both parameters and FLOPs (e.g., Swin-L variant: 522M params vs. 216M, 1320G vs. 868G). Inference speed (FPS) is lower due to sequential decoding and additional modules. While the latent space reduces output dimensionality, the net cost is higher.\n- multi-stage complexity: The full pipeline depends on careful multi-stage training, including pretraining of seglat encoder/decoder, latent modules, and final alignment. This is more demanding than standard discriminative setups.\n- though the authors mark ‘statistical significance’ section in the checklist as ‘yes’, but the paper does not report significance tests, confidence intervals, or error bars to support this claim.", "was_revised": true, "final_justification": "I retain my original rating. The rebuttal has resolved some of the issues I raised, as the authors provided additional details / experiments, clarifications. I agree with other reviewers regarding clarity (and architectural complexity) but I expect the authors to significantly improve the writing and figures for the camera-ready version.", "rebuttal": "Thank you for your insightful comments and valuable questions. We appreciate the opportunity to address these points and clarify our work. \n\n\n1.  Compute Cost and Latency\n\nWe acknowledge that even though Seg-VAR has consistent improvement across standard benchmarks, it exhibits higher computational overhead (parameters, FLOPs) and slower inference speed compared to discriminative models like Mask2Former. This is related to our two key innovations: (1) the autoregressive decoder, which sequentially predicts seglat tokens to ensure spatial coherence, and (2) the multi-module architecture (image encoder, seglat encoder/decoder, latent aligners) required for latent space learning.\n\n\nWhile full parallelization of sequential decoding is infeasible, targeted optimizations could reduce latency without sacrificing core autoregressive modeling:\n\n- Hierarchical Parallelism Across Scales\n\nSeg-VAR uses hierarchical decoding, where coarse-scale tokens (establishing global context) are generated first, followed by fine-scale tokens (resolving local details). Coarse-scale tokens could be generated in a compact sequence, after which fine-scale tokens within the same coarse region might be parallelized. For instance, once a global object boundary is defined by coarse tokens, local patches within that boundary could be processed in parallel, as their dependencies are constrained to the local context.\n\n- Cached Attention Mechanisms\n\nThe transformer decoder’s attention over previously generated tokens can be optimized using cached key/value pairs. Instead of recomputing attention scores for all prior tokens at each step, cached values from previous steps can be reused, reducing redundant computations\n\n2. Multi-stage Training\n\nWe acknowledge the training pipeline (pretraining seglat encoder/decoder, latent modules, final alignment) is more complex than standard discriminative setups. However, this staged approach is **necessary** to effectively learn the structured latent space and ensure stable convergence of the autoregressive process. End-to-end training from scratch proved unstable in ablation studies.\n\n|             | Dataset | mIoU |\n|-------------|---------|------|\n| End-to-End  | ADE20K  | 81.2 |\n| Seg-VAR     | ADE20K  | 85.8 |\n\nTable 1. Staged training vs. End-to-end.\n\n\n3. Statistical Significance\n\nWe apologize for the oversight in not explicitly reporting error bars or confidence intervals. While our experiments were run 3-5 times (as noted in the checklist) , we will add in the supplementary material: (1) standard deviations of key metrics (PQ, AP, mIoU) across 5 independent runs on COCO and Cityscapes, and (2) 95% confidence intervals for ablation studies (e.g., Table 5) . This will better support the statistical robustness of our results.\n\n4. Grid Size and Colormap\n\nWe evaluate the robustness of the grid number and Palette on ADE20K. As shown in the table, as the grid number changes greatly,\nthe performance decreases as the number decreases, because the granularity can help the model better\ndistinguish instances. As for the palette size of the colormap, we discover that 6 is the optimal number because\nwe have to balance between a large color space and loss of generality. (Ideally, the size should be larger than the category number).\nOverall, we find that grid numbers are more sensitive than palette size, but they still contribute to the performance gain.\n\n| Grid number(a) | mIoU | Palette size | mIoU |\n|----------------|------|--------------|------|\n| 4              | 84.4 | 124            | 85.4 |\n| 8              | 85.2 | 215            | 85.8 |\n| 12             | 85.8 | 342            | 85.3 |\n\nTable 2. Grid \\& Colormap\n\n5. Practical Advantages of Autoregressive Formulation\nBeyond segmentation quality, the autoregressive design offers unique benefits: \n- Video extension: The sequential token prediction naturally models temporal dependencies, making Seg-VAR adaptable to video segmentation (e.g., tracking instance motion via time-step token conditioning). \n- Unifying generation and perception: The autoregressive formulation in Seg-VAR inherently paves the way for unifying generation and perception, as its design bridges the latent spaces of image content and segmentation masks through sequential modeling. (Which is also recognized by Reviewer avcA)"}, {"review_id": "3YGaMBe3pT", "reviewer": "Reviewer_avcA", "summary": "This paper investigates visual autoregressive (VAR) modeling for image segmentation. The authors propose a novel encoder-decoder framework designed to repurpose the architecture and infrastructure of generative models to handle image segmentation tasks. Their approach leverages autoregressive modeling to sequentially generate segmentation masks, demonstrating improved performance compared to state-of-the-art discriminative segmentation models such as MaskFormer and DETR. The core contribution is the adaptation of generative modeling principles to segmentation. Learning the generative process underlying segmentation masks may provide more nuanced representations than traditional discriminative methods.", "questions": "1.\tClarification on Generative Model:\nCould the authors elaborate on the generative modeling aspect of Seg-VAR? Specifically, does the model use a pure autoregressive transformer, a next-scale prediction VAR, or some hybrid? More detail about the architecture (e.g., sampling procedure, training objective) would be helpful.\n2.\tTokenization and Latent Representation:\nHow is the latent encoder for segmentation related to the traditional VAR tokenization scheme? Does Seg-VAR employ a multi-scale, residual tokenization process, and if so, how are these latent tokens structured and used?\n3.\tConnection to Image Autoencoders:\nIs the segmentation autoencoder in this framework structurally similar to classic VQ-VAE or other image autoencoders? How is the mapping between segmentation masks and discrete latent tokens performed, and does it incorporate spatial or instance-aware encoding?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The central idea of reusing generative model architectures for segmentation is both novel and interesting. This direction can potentially bridge the gap between generative and discriminative paradigms in vision, enabling deeper insight into the structure of segmentation masks.\n- The proposed method achieves better performance than current state-of-the-art segmentation models across several benchmarks, indicating its practical value and significance.\n\nWeaknesses\n- The paper is often unclear and challenging to follow, particularly in the method section. The presentation of the framework, especially regarding the generative model and tokenization process, lacks sufficient detail to ensure reproducibility.\n\t- The connection between the proposed segmentation latent encoder (seglat) and established VAR models (which typically rely on multi-scale residual tokenization) is not clearly articulated. It remains ambiguous whether the proposed latent representations inherit the multi-scale and residual design principles from classic VARs.\n- The manuscript suffers from awkward phrasing, grammatical errors, and poor structural organization, which impedes comprehension and technical assessment.\n- The role of the generative model component is under-explained. It is unclear if the framework uses a pure autoregressive transformer, a next-scale prediction model, or another architecture. The relationship between the autoencoder, tokenization scheme, and segmentation mask reconstruction needs clarification.", "was_revised": true, "final_justification": "I appreciate the author's response to my questions. I adjusted my score to borderline accept. However, I would expect improvement in the clarity of writing in the camera ready version.", "rebuttal": "Thank you for your detailed feedback and thoughtful evaluation of our work. We appreciate the recognition of our novel direction in bridging generative and discriminative paradigms for image segmentation, as well as the constructive criticism that will help us improve the clarity and rigor of the paper. We will reorganize the paper with better structures for readers to comprehend. Below, we address each concern and question raised:\n\n1. Generative Model Architecture and Training Details\n\n    We apologize for the potential ambiguity in describing the generative components of Seg-VAR. Seg-VAR is built on visual autoregressive (VAR) modeling and employs a hybrid design that combines hierarchical autoregressive decoding with next-scale prediction principles. As for the sampling strategy, it refers to an approach of sampling both pixel- and token-level controls for image generation with teacher-forcing guidance, which is an extension of classifier-free guidance by adding controls to the Bayesian-based conditional distribution, as mentioned in ControlVAR. \n    \n    As for the training objectives, the training objective is based on the Evidence Lower Bound (ELBO), optimizing three components: (1) mask reconstruction loss via seglat decoder, (2) KL divergence to align image-derived latents with seglat distributions, and (3) cross-entropy loss for token prediction.\n\n    Overall, this design inherits VAR’s next-scale prediction strengths while adapting it to segmentation via conditional mask generation.\n\n\n2. Tokenization and Latent Representation\n\n    The seglat encoder is tightly linked to traditional VAR tokenization but extends it for segmentation-specific needs:\n\n    1. Like classic VAR, Seg-VAR uses a multi-scale, residual tokenization process: the image encoder generates latent tokens at multiple scales via a multi-scale fusion module (transformer layers + projection), and the seglat encoder (VAR-like structure) maps masks to discrete tokens across these scales.\n    3. Unlike standard VAR, seglat tokens incorporate spatial information via a location-sensitive color mapping. This mapping assigns unique RGB values to instances based on their centroids (gridded into a*a regions), enabling the transformer to distinguish overlapping objects through positional cues.\n\n    Latent encoder for segmentation is implemented to transform segmentation masks (H\\*W\\*K) into corresponding seglats (H\\*W\\*3) to be then fed into the Seglat Encoder \\& Decoder (which refers to the VAR-like structure), itself is a transformer-based structure with colormap encoding.\n\n3. Connection to Image Autoencoders\n\n    Seg-VAR’s segmentation autoencoder (seglat encoder/decoder) shares similarities with VQ-VAE but is specialized for segmentation (add extra image controls into modeling): Like VQ-VAE, it uses discrete latent tokens and a reconstruction loss.\n    \n    As for the latent tokens, they refer to the output from the image encoder, which is (H/d\\*W/d), which is optimized in the image encoder learning to minimize the differences with the latent codes by the seglat encoder (VQ-VAE-like structure). The segmentation masks use the latent encoder to generate seglat tokens with a transformer-based structure and colormap encoding, which incorporates spatial and instance-aware encoding."}, {"review_id": "fSo2JLanYs", "reviewer": "Reviewer_MUQW", "summary": "This paper proposes an innovative architecture for image segmentation under an autoregressive paradigm. Specifically, the authors design three components: 1) an image latent encoder, 2) a feature representation called \"seglat\" for encoding segmentation results, and 3) an autoregressive generator/decoder. Under this framework, the image segmentation task is modeled as a next-token prediction problem. The authors conduct extensive experiments to validate the effectiveness of the proposed algorithm.", "questions": "I rate this article very highly, and would be happy to improve it further if the author could answer my small questions with clarity.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n1. **Clear narrative structure**: The paper presents a coherent storyline with well-aligned claims and supporting evidence.\n\n2. **Superior performance**: The model achieves excellent accuracy with significant performance improvements compared to baseline methods. The ablation studies are comprehensive and well-designed.\n\n3. **Novel paradigm**: Although the approach draws inspiration from VAR and ControlVAR, the proposed autoregressive modeling paradigm represents an innovative contribution to the field of image segmentation.\n\n**Weaknesses:**\n\n1. **Insufficient mathematical formulation**: The seglat representation appears to be the core contribution of this work, and the authors mention that this component references the maskige design from GSS. However, the paper lacks formal mathematical expressions for the seglat modeling process, which hampers the comprehensibility of the proposed method.\n\n2. **Open vocabulary capability unclear**: Given this seglat representation and the complete modeling paradigm, I want to know how well the model performs on open vocabulary segmentation tasks.\n\n3. **Parameter efficiency concerns**: While the model demonstrates significant performance improvements, a lingering question remains: how would the performance compare if the parameter count were kept consistent with the comparison methods?", "was_revised": true, "final_justification": "Thanks to the author for the reply, I will increase my score.", "rebuttal": "Thank you for your insightful evaluation and high appreciation for our work. We greatly appreciate your recognition of the paper’s clear narrative structure, superior performance, and innovative autoregressive paradigm for segmentation. Your constructive feedback on areas for improvement is invaluable, and we address each concern below:\n\n1. Mathematical Formulation of Seglat\n\nSeglat is designed as a structured latent representation that encodes both instance identity and spatial geometry of segmentation masks, bridging low-level mask pixels and high-level semantic tokens. Mathematically, we define it as follows:\nLet $M∈\\{0,1\\}^{H×W×N}$ denote a segmentation mask, where H,W are spatial dimensions, and N is the number of instances. Then, through the spatial convolution and transformer layer, the encoder transforms the original masks into $\\mathcal{S}\\in\\mathbb{R}^{H\\times W \\times 3}$, which can be viewed as a kind of RGB image. To add more explicit identity and positional control, we implement colormap encoding to generate an additional colormap $M_{c}∈\\{0,1\\}^{H×W×3}$. These are added and fed into the seglat encoder as the final seglat $\\mathcal{S}$.\n\nLet $k∈{1,2,...,K}$ denote a scale in the hierarchical transformer structure (where S is the total number of scales). At scale k, image tokens and seglat tokens are tokenized by a shared tokenizer $\\Phi$: \n Image features are \n$X_{k}∈[V]^{h_k×w_k}$, \nand Seglats are $S_{k}∈[V]^{h_k*w_k}$,\n where V indicates the vocabulary size.  A flattening operation\nis adopted to convert the sequence of 2D features into 1D. Full attention is enabled for both control and image tokens belonging to the same scale, which allows the model to maintain spatial locality and to exploit the global context between control and image. $X_k',S_k'=Attention(X_k,S_k,S_k)$. As for the algorithms for tokenizers, \nwe follow the design of VAR, which consists of interpolation, queue push, and lookup operations.\n\n\n2. Open Vocabulary Segmentation Capability\n\n|             | Dataset | mAP  |\n|-------------|---------|------|\n| Mask2Former | LVIS    | 17.1 |\n| GSS         | LVIS    | 18.8 |\n| Seg-VAR     | LVIS    | 24.3 |\n\nWe evaluate the open-vocabulary capability by evaluating on LVIS. By adopting CLIP to the model for classification, our model achieves a preliminary result of 24.3 mAP on novel classes (vs. 17.1 mAP for Mask2Former). This is because the seglat design inherently supports open vocabulary adaptation: Seglat tokens are semantically agnostic—they encode geometry (boundaries, centroids) rather than fixed class labels. \n\n3. Parameter Efficiency\n\n|             | Backbone | mIoU |\n|-------------|----------|------|\n| Mask2Former | R50      | 63.8 |\n| SegVAR      | R50      | 64.2 |\n\nWe adjust the parameters of Mask2Former by extending its transformer layer number so that the parameters can be comparable. As shown in the table, our Seg-VAR still outperforms Mask2Former in COCO panoptic dataset."}, {"review_id": "fr1pGswevF", "reviewer": "Reviewer_BXbx", "summary": "This paper proposes Seg-VAR, a novel autoregressive framework for image segmentation tasks. It rethinks segmentation as a conditional mask generation problem and introduces a spatial-aware latent representation of segmentation masks called seglats. These seglats, along with a multi-stage training strategy and hierarchical autoregressive decoding, allow the model to capture spatial and instance-level structures more effectively. The method demonstrates strong empirical performance across semantic, instance, and panoptic segmentation tasks.", "questions": "1. How sensitive is the performance to the choice of grid size or color palette used in the spatial-aware color mapping? The paper uses a grid-based mapping to assign instance-specific colors, but it's unclear how robust this is to changes in grid granularity or color discretization.\n1. Why are both a seglat decoder and a latent decoder needed? The model has two decoding stages—one for seglats, another for final masks. Could a single decoding stage suffice if trained end-to-end?\n1. What is the role of the [CLS] and [TYP] tokens in the seglat encoder/decoder, and how much do they contribute to performance? These tokens are introduced without much analysis or ablation.", "limitations": "The primary limitation of this method lies in its architectural complexity and computational cost. The framework includes multiple encoders and decoders—image encoder, seglat encoder/decoder, and latent encoder/decoder—but the necessity of each component is not clearly justified. A more thorough discussion or ablation of these modules would help clarify whether the full complexity is essential for the observed performance gains.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths*:\n\n1. The proposed autoregressive approach to segmentation is conceptually compelling. The idea of disentangling semantic features from mask representations via seglats is both intuitive and well-motivated.\n1. The spatial-aware color mapping mechanism for encoding instance-level information into seglats offers a creative and effective solution to a core challenge in generative segmentation—distinguishing overlapping or adjacent instances.\n1. The experimental results are strong: Seg-VAR achieves state-of-the-art performance across semantic, instance, and panoptic segmentation tasks on COCO, ADE20K, and Cityscapes, demonstrating the generalizability and robustness of the approach.\n\n*Weaknesses*:\n\n1. Insufficient component-level ablation: While Table 5 highlights the effect of major training stages, the paper lacks finer-grained ablation studies isolating the contributions of individual modules such as the seglat encoder/decoder or latent decoder/decoder. The comparison to alternative generative backbones (e.g., SD-XL, DALL·E 2, VQGAN) primarily supports the use of VAR itself, but does not clearly attribute the performance gains to the proposed seglat design. Although the supplementary includes FID comparisons, it would strengthen the work to show segmentation performance using vanilla VAR (i.e., without the seglat module).\n1. Pipeline complexity and resource demands: The framework involves multiple interacting components—image encoder, seglat encoder/decoder, latent encoder/decoder—which likely require significant computational resources for training.\n1. Terminology and Figure Clarity: The term Latent Encoder is potentially misleading. According to the paper, its main function is to assign spatially-informed colors to segmentation masks based on instance centroids—more akin to a handcrafted spatial embedding mechanism than a latent feature encoder. If the primary role is color assignment via spatial rules, the use of transformers and the need for learning are not well justified. Additionally, Figure 1 lacks clarity: while Line 132 states that the segmentation masks are binary, the figure shows input images instead of segmentation masks or their encoded seglat representations. A clearer illustration of how seglats are generated and used would greatly aid understanding.", "was_revised": true, "final_justification": "The authors demonstrate the effectiveness of their design, which offers a novel method for segmentation. I recommend that they revise the paper accordingly for clarity.", "rebuttal": "Thank you for your thoughtful and constructive feedback on our work. We appreciate the recognition of Seg-VAR’s conceptual novelty, the effectiveness of seglats, and the strength of our experimental results. Below, we address each concern, clarify ambiguities, and provide additional details to strengthen the work. \nWe will add this refinement to the final version of the paper.\n\n1. Component-level ablation\n\nSorry for the confusion. The ablation in Table 5 is the ablation about each module design: stage 1 represents seglat encoder/decoder, stage 2 refers to latent encoder and decoder, and stage 3 represents the image encoder learning. That is, we discuss the impacts of different module designs. \nSince latent learning is the core idea of Seg-VAR, we keep the latent encoder and decoder in ablation studies. \n\nTo explicitly validate the role of seglats, we conducted additional experiments using a \"vanilla VAR\" baseline (removing latent encoder and decoder, and using plain VAR). \nAs shown in the table, simply using vanilla VAR without seglat modules, the result is 8.4 lower than our Seg-VAR.\n\n|             | Dataset | mIoU |\n|-------------|---------|------|\n| Vanilla-VAR | ADE20K  | 77.4 |\n| Seg-VAR     | ADE20K  | 85.8 |\n\nTable 1. Vanilla VAR v.s. Seg-VAR\n\n2. Pipeline complexity and resources\n\nWe acknowledge the training pipeline (pretraining seglat encoder/decoder, latent modules, final alignment) is more complex than standard discriminative setups. However, our module designs prove their effectiveness in the ablation study, and that staged approach is **necessary** to effectively learn the structured latent space and ensure stable convergence of the autoregressive process. End-to-end training from scratch proved unstable in ablation studies (4.6 decrease in mIoU).\nWhat's more, under comparable parameters (Table 3), our Seg-VAR still outperforms Mask2Former in R50 backbone. These results suggest the efficiency and effectiveness of Seg-VAR.\n\n|             | Dataset | mIoU |\n|-------------|---------|------|\n| End-to-End  | ADE20K  | 81.2 |\n| Seg-VAR     | ADE20K  | 85.8 |\n\nTable 2. Staged training vs. End-to-end.\n\n|             | Backbone | mIoU |\n|-------------|----------|------|\n| Mask2Former | R50      | 63.8 |\n| Seg-VAR     | R50      | 64.2 |\n\n\nTable 3. Comparable parameters\n\n3. Terminology and figure\n\nWe apologize for ambiguities in terminology and figures, which we clarify below: \n- for the latent encoder, we aim to indicate that this structure encodes the segmentation mask\nand output segment latents (seglat). Worth mentioning, this module is a combination of the transformation\nof segmentation masks and color encoding, rather than merely a \"spatial embedding mechanism\". As for the\nfigure, we thank you for your careful suggestions, and we will revise it in the final version!\n(Our intent is to highlight the contents of the image, because binary will lose details.) \n\n4. Sensitivity of grid size and colormap\nWe evaluate the robustness of grid number and Palette on ADE20K. As shown in the table, as the grid number changes greatly,\nthe performance decreases as the number decreases, because the granularity can help modeling better\ndistinguish instances. As for the palette size of the colormap, we discover that 6 is the optimal number because\nwe have to balance between a large color space and loss of generality. (Ideally, the size should be larger than the category number).\nOverall, we find that grid numbers are more sensitive than palette size, but they still contribute to the performance gain.\nThese results confirm Seg-VAR is robust to reasonable variations in grid/color parameters.\n\n\n| Grid number(a) | mIoU | Palette size | mIoU |\n|----------------|------|--------------|------|\n| 4              | 84.4 | 124            | 85.4 |\n| 8              | 85.2 | 215            | 85.8 |\n| 12             | 85.8 | 342            | 85.3 |\n\nTable 4. Grid \\& Colormap\n\n5. Seglat and latent decoder\n\nThe difference in structure naturally prohibits the combination of two decoders, since the seglat decoder is part of VQ-VAE, while latent decoder\nis a transformer-based module.\nWhat's more, the seglat decoder and latent decoder serve distinct roles:\n\n- Seglat decoder: Maps discrete latent tokens (from the image encoder) to seglats, preserving instance-level spatial structure encoded in color mappings.\n- Latent decoder: Refines seglats into pixel-accurate masks, correcting residual errors in color-to-mask conversion (e.g., edge blurring)\n\nEnd-to-end training of a single decoder failed to balance these two granularities (as proved in Table 2), confirming the need for two stages.\n\n6. [CLS] and [TYP]\n\nThese two tokens are required to distinguish between different segmentation tasks. [CLS] provides semantic context for the generated image\n(categories), and the [TYP] token is used to select the type of segmentation tasks. Both tokens are vital to Seg-VAR and can not be \nablated. We will include these details in the revised version.\n\n\n### Conclusion\n\nWe agree that architectural complexity is a key consideration. Our ablations (summarized above) demonstrate that each component (image encoder, seglat encoder/decoder, latent encoder/decoder) contributes meaningfully to performance. Removing any module resulted in significant drops.\n\nWe plan to explore model simplification in future work, such as sharing weights between encoders or using distillation, while maintaining Seg-VAR’s unified capabilities. We thank the reviewer for highlighting opportunities to strengthen the work. The additional ablations, figure revisions, and clarifications outlined above will enhance the clarity and rigor of our submission. \nWe believe these changes address the concerns raised while reinforcing the novelty and effectiveness of Seg-VAR, which could potentially lead to more explorations in unifying perception and generation."}]

Author rebuttal:

Meta-review: a) The authors propose to solve the image segmentation task using a conditional auto-regressive model to sequentially generate a segmentation mask. They use a latent segmentation mask representation combined that is combined with the image in an auto-encoder. The pipeline is trained in a multistage fashion.

b) 
- Originality of the pipeline.
- Works for a variety of segmentation task without adaptation of the pipeline (semantic, instance, and panoptic segmentation tasks)
- Tested on multiple datasets (SOTA results) + ablation studies

c) 
- the clarity of the writing must be improved for the camera-ready version

d) After the rebuttal all reviewers reach a consensus towards accepting the paper under the condition that the authors improve the writing for clarity. Originality of the framework is definitely valuable. This might explain to lack of clarity as it is not a direct application of previous ideas/models. The results are SOTA. This paper deserve to be accepted as a poster.

e) The discussion led some reviewers to change their mind in favor of accepting the paper and the major concerns were resolved. All reviewer have concerns about the clarity but I am confident that the discussion was helpful for the authors to improve the presentation of their work.
Decision: Accept (poster)