FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction
Authors: Siyu Jiao
Gengwei Zhang
Yinlong Qian
Jiancheng Huang
Yao Zhao
Humphrey Shi
Lin Ma
Yunchao Wei
ZEQUN JIE
Venue: NeurIPS 2025 poster ()
Keywords: Autoregressive Modeling
Image generation
Primary area: deep_learning

Abstract: This work challenges the residual prediction paradigm in visual autoregressive modeling and presents FlexVAR, a new Flexible Visual AutoRegressive image generation paradigm. FlexVAR facilitates autoregressive learning with ground-truth prediction, enabling each step to independently produce plausible images. This simple, intuitive approach swiftly learns visual distributions and makes the generation process more flexible and adaptable. Trained solely on low-resolution images (< 256px), FlexVAR can: (1) Generate images of various resolutions and aspect ratios, even exceeding the resolution of the training images. (2) Support various image-to-image tasks, including image refinement, in/out-painting, and image expansion. (3) Adapt to various autoregressive steps, allowing for faster inference with fewer steps or enhancing image quality with more steps. Our 1.0B model outperforms its VAR counterpart on the ImageNet 256 × 256 benchmark. Moreover, when zero-shot transfer the image generation process with 13 steps, the performance further improves to 2.08 FID, outperforming state-of-the-art autoregressive models AiM/VAR by 0.25/0.28 FID and popular diffusion models LDM/DiT by 1.52/0.19 FID, respectively. When transferring our 1.0B model to the ImageNet 512 × 512 benchmark in a zero-shot manner, FlexVAR achieves competitive results compared to the VAR 2.3B model, which is a fully supervised model trained at 512 × 512 resolution.

FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction
Siyu Jiao'
Gengwei Zhang?
Yinlong Qian
Jiancheng Huang
Yao Zhao'
Humphrey Shi
Lin Ma
Yunchao Weilt
Zequn Jie
Institute of Information Science, Beijing Jiaotong University University of Technology Sydney Meituan Georgia Institute of Technology
Code: FAR
512
384
256
384
384
256
128
Figure Generated samples from VeR--ARd24 (1.OB). FAR generates images with various resolutions and aspect ratios, even though it is trained with resolution of 256x256.
Abstract
This work challenges the residual prediction paradigm in visual autoregressive modeling and presents Flex VAR, a new Flexible Visual AutoRegressive image generation paradigm. Flex VAR facilitates autoregressive learning with ground- truth prediction, enabling each step to independently produce plausible images. This simple, intuitive approach swiftly learns visual distributions and makes the generation process more flexible and adaptable. Trained solely on low-resolution images (< 256px), VAR can: Generate images of various resolutions and aspect ratios, even exceeding the resolution of the training images. (2) Support various image-to-imane
tasks, including image refinement, in/out- pointtt.ainiing... and image expansion. (3) Adapt to various autoregressive steps, allowing for faster inference with fewer steps or enhancing image quality with more steps. Our 1.0B model outperforms its VAR counterpart on the ImageNet 256x 256 benchmark. Moreover, when zero-shot transfer the image generation process with 13 steps, the performance further improves to 2.08 FID, outperforming state-of-the-art
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

autoregressive models AiM/VAR by 0.25/0.28 FID and popular diffusion models LDM/DiT by 1.52/0.19 FID, respectively. When transferring our 1.0B model to the ImageNet 512x512 benchmark in zero-shot manner, Flex VAR achieves competitive results compared to the VAR 2.3B model, which is a fully supervised model trained at 512 resolution.
1 Introduction
Autoregressive (AR) models aim to learn the probability distribution of the next to- ken, offering great flexibility by generating tokens of any length. This design brings sig- nificant udvancements in the field of Natural Language Processing (NLP), demonstrating satisfactory generality and transferability
the image latent festure at 1h scale
the resichal latent feuture at 1 scale
Input Img.
Input Img.
Residual -VQVAE
Scalable- VOVAE VQVAE
4 30 33 Concurrently, the computer vi-
sion field has been striving to develop large autoregressive models 22 43 [2 38).
91
91
92
modeling residual
modeling image
9N
These models employ visual tokenizers to discretize images into series of ID to-
VAR-Transformer
FlexVAR- Transorrra
4 or 2D scales and then utilize AR to model the next unit. However, these image autoregressive models typically output im- ages at a single resolution, the flexibility of AR has not yet been realized.
kens
34
22
49
52
(a) VAR
(b) FlexVAR (ouns)
Figure 2: Comparison between VAR 44 and our Flex- VAR. VAR predicts the GT in step and then predicts the residuals relative to the GT in all subsequent steps. Our FlexVAR predicts the GT at each step.
Recently, in image generation, VAR 44] has pioneered scale-wise autoregressive modeling, com- pleting image autoregression based on 2D sequences. This approach predicts the next scale rather than the next token, thereby preserving the 2D structure of images and mitigating the issue of limited receptive fields in ID causal transformers. Specifically, VAR predicts the ground-truth (GT of the smallest scale in the first step. Subsequently, at each step, it predicts the residuals of the current scale and the prior one. Finally, the outputs of each scale are upsampled to a uniform size and undergo weighted summation to generate the final output, as illustrated in Fig. D(a). Successors [5]] 37 25
have all adopted the residual design, assuming it to effective. A Although this technique achieves commendable performance, it encounters primary challenge: The residual prediction relies on rigid step design, restricting the flexibility to generate images with varying resolutions and aspect ratios, thus limiting the adaptability and flexibility of image generation. Meanwhile, residuals at different scales often lack semantic continuity, and this implicit prediction approach may limit the model's capacity to represent diverse image variations.
In this work, we examine the necessity of residual prediction in visual autoregressive modeling. Our intuition is that, in scale-wise autoregressive modeling, the ground-trude value of the current scale can be reliably estimated from the prior series of scales, rendering residual prediction (i.e., predicting the bias between the current scale and the preceding one) unnecessary. Notably, predicting GT ensures semantic coherence between adjacent scales, making more conducive for modeling the probability distribution of the scale. Additionally, this structure can output reasonable results at any step, breaking the rigid step design of the residual prediction and endowing autoregressive modeling with great flexibility.
Motivated by this, systematically design the paradigm of visual autoregressive modeling without residual prediction, referred to as FlexVAR, Within Flex VAR, the ground -ruun is predicted at each step instead of the residuals. Specifically, we design scalable VQVAE tokenizer with multi- scale constraints, enhancing the VQVAE's robustness to various latent scales and thereby enabling image reconstruction at arbitrary resolutions. Then, the Flex VAR Transformer learns the probability distribution of a series of multi-scale latent features, modeling the ground-truth of the next scale, as shown in Fig. (b). Additionally, we propose scalable 2D positional embeddings, which incorporate
'To avoid confusion, we use ground-truie truth (GT) to represent image latent feature, and residual to represent residual latent feature.

---

2D learnable queries initialized with 2D sin-cosing weights. This approach enables the scale-wise autoregressive modeling to be extended various resolutions/steps, including those beyond the resolutions/steps used during training, shown in Fig. [
In nutshell, this non-residual modeling approach ensures continuous semantic representation between adjacent scales. Simultaneously, it avoids the rigid step design inherent in residual prediction, significantly expanding the flexibility of image generation. Flex VAR can (1) generate images of various resolutions and aspect ratios, even exceeding the training resolutions; (2) support image-to- image tasks such as in/out-painting, image refinement, and image expansion without the need for fine-tuning: (3) enjoy flexible inference steps, allowing for accelerated inference with fewer steps or improved image quality with more steps.
2 Related Work
VQ-VAE [46] introduces a groundbreaking two-stage image generation paradigm: (1) encoding the image into latent space and quantizing it to the nearest code a fixed-size codebook; (2) modeling the discretized code using PixeICNN 45), which predicts the probability distribution of each code in raster scan order. This two-stage paradigm has laid the foundation for many subsequent works.
Raster-scan Manner Building on the aforementioned foundation, 12 33 perform autoregressive learning in latent space with Transformer architecture. VQVAE-2
35 and RQ--rrasffrmmer [22]
use extra scales or stacked codes for next-image-toke -onkag prediction. These works further advance the 26 utilize a GPT-style next-token- next-token-prediction
field and achieve impressive results. Recently, [41
strategy to achieve high-quality image generation.
further improves this paradigm by introducing mixture of autoregressive models, while (33 incorporates Mamba structure [33 to accelerate image
generation. 48 (3 [4] combine diffusion processes into autoregressive modeling to address the
information loss caused by quantization, which potentially degrades the quality of generated images.
Random-scan Manner Masked-prediction models learn to predict masked tokens in a BERT-style manner 16 2] They introduce a bidirectional transformer that predicts masked tokens by attending to unmasked conditions, thus generating image tokens in a random-scan manner. This approach enables parallel token generation at each step, significantly improving inference efficiency. and text-to-image generation, respectively. MagVit ViT series [50] (2 adapts this approach videos by introducing a VQVAE for both images and videos. NOVA B first predicts temporal frames and then predicts spatial sets within each frame to achieve high-quality image/video generation.
Specifically, 6 (5) apply masked- prediction models in class-to-
masked-prediction
Scaling-scan Manner VAR establishes a new generation paradigm that redefines autoregressive next-scale-prediction VAR in parallel predicts image tokens at one scale, significantly reducing the number of inference steps. Following VAR. VAR-CLIP [51) achieves text-to-image generation by converting the class condition token into text tokens obtained from the CLIP. In terms of operational efficiency, 77] introduces an efficient decoding strategy, (37) incorporates linear attention mechanisms to accelerate image generation, and [25 designs a lightweight image quantizer, significantly reducing training costs. Regarding generation quality, 42 36 optimize image details by using continuous tokenizers in combination with flow matching diffusion model. Infinity [15] redefines the visual autoregressive model under bitwise token prediction framework, remarkably enhancing generation capability and detail.
learning on images from next-token-predicttin
3 Methodology
3.1 Scale-wise Autoregression (Preliminary)
Scale-wise autoregressive models tokenize the input image into a sequence of multi-scale discrete image token maps T' {ti, tn,..., where t, is the token map with the resolution of h; x w; downsampling from t,, Rh,,,,,,, Each autoregressive step generates an entire token map, rather than a single token. Compared to next- soattttkken-peee prediction,, which contains one token at each step, t; contains h; w; tokens and is able to maintain the 2D structure.
Previous approaches only regress the ground truth at the first scale (g1). while subsequent tth scale, the residual between
37 25 42
typically follow a residual prediction paradigm. They
3

---

the preceding scale (9:-1) and current scale (g:) is predicted. We formulate residuals (fr;)?-2) as:
ri 9: Upsample' (gi-1);
here Upsample' represents upsample gi-1 to the ith scale. The autoregressive likelihood is:
plg, 22, , Ta) = p(r: 22, (n.......
attention mechanisms (e.g., Transformer 47 are utiliizd instantiate this modeling. During the ith autoregressive step, all preceding residuals are merged in the autoregressive model, which then pre- dicts the probability distribution of r, The h; x w, tokens in r; are generated parallel, conditioned on all preceding units. Thus, image token maps can be redefined as: T {gi, T2, ra...,nn.. Finally, each image token map in I' is upsampled to R 1%, and summarized for image generation.
3.2 Overview of FleXVAR
Our Flex is flexible visual autoregressive image generation paradigm that allows autoregressive learning with ground-trus truuth prediction rather than residual, enabling to generate reasonable images at any step independently. Within input images into tokens at various scales and reconstructs images, as detailed in Sec. 3.3 (2) A Flex VAR transformer is trained via scale- wise autoregressive modeling, with the removal of residuals, as detailed in Sec. 3.4
approach: (1) A scalable VQVAE tokenizer quantizes the
Quantize reconstruct images various scales
Mainstream VQVAE tokenizers perform well at single resolution. However, when scaling the latent space, they often fail to reconstruct images shown in Fig. 33). This observation motivates to explore scalable tokenizer that quantizes input images into tokens various scales and reconstructs images. Specifically, the proposed scalable tokenizer first encodes an image into multi-scale latent space, and then uses quantizer to convert latent space features into discrete tokens, finally a decoder used to reconstruct the original images from the discrete tokens at each scale.
Encoding, Given an input image I E R R#x an autoencoder E(-) [12 is used to convert into latent space f:
f f= E Rxxxx
We then downsample ʃ at K random scales to obtain multi-scale latent features F = f2.,,,k f& represents represents the latent feature of the 1 downsample from f. fx matches the original resolution of
here h 16 W 16
Quantizing. The quantizer 2(-) includes codebook Z E RVC containing learnable vectors.
The quantization process g = Q()) is implemented by finding the Fuclidean nearest code o (k.Ad) of each feature vector (k in multi-scale latent features F:
(k...) (argmin,ggy///$eeet((,, - F( E [V]
(4)
where Select(Z, [2 v) denotes selecting the th vector in codebook Z. Based on F,, we extract all (k.555) and minimize the distance between g and f to train the quantizer Q.
Decoding, The multi-scale images 7 12, Ix } are reconstructed using the decoder D(.) 12
given g (k.3.)) We follow Llamagen 41 to adopt the same loss functions (Lrae) train {E, Q, D) each scale without special design. Therefore, the final loss function can be formulated:
C
Σ Luae (Uss, Ia), (Jxs (k))
(5)
3.4 Visual autoregressive modeling without residual
reconceptualize the next-scale-predittin progress from residual prediction to GT prediction. illustrate in Fig. (b). Here, each autoregressive step predicts the GT of current scale, rather than

---

the residual. We start by sampling N multi-scale token maps (91.92: .2............ 9NV from latent feature f. each an increasingly higher resolution h, x Wa. culminating in INv matches the original feature map's resolution Rxxxx, The autoregressive likelihood is reformulated as:
plg1.92. 92. In. II p(g. 92.92..11 ................................ In-1).
(6)
During the 7t autoregressive step, 9: E Rhxx contains hg x tokens are generated in parallel, conditioned on all preceding scales {91. 901.92....
Scalable Position Embedding. VAR utilizes fix fix-length Position Embedding (PE) by adding learnable queries each step and h-w coordinates. This requires both training and inference follow fixed number of steps and resolutions, which limits the flexibility of the autoregressive process.
In our VAR, we design 2D scalable PE (P E Rdx2hx2u) adding to the h-w coordinates. It contains 2h 2w learnable queries with channels. At the 1th step, P is upsampled/downsamppled to match the scale of gi- To ensure stability during linear interpolation across various scales, we set P to 2x the size of the max latent space in traning. P initialized using 2D sin-cosine PE [ to ensure the 2D positional correlation. Additionally, prediction paradigm, incorporating PE for step embeddings is unnecessary (Tab. (6). Therefore, we remove the step embeddings to ensure the flexibility of steps in autoregressive modeling.
experimentally find that in our ground-truth
Step sampling. During training, we randomly sample the scale size each step to enhance Flex VAR's capability to perceive any scale. Specifically, we set the maximum number of steps to 10, fixing the scale size of the first step to 1x and the last step to 16x 16 (corresponding to 256x256 input images), and randomly sampling the scale sizes for the intermediate steps. Each step is dropped with 5% probability, with a maximum of steps being dropped. Thus, the number of steps during training is from 6 to 10. During inference, we use default of 10 steps: {1, 3, 4, 5, 6, 8, 10, 13, 16) (same as VAR), Our experimental results show more steps yield better performance (Fig. 6].
4 Experiments
4.1 Implementation details
FAe tokenizer. Our scalable VQVAE tokenizer is configured with a downsampling factor of 16 and is initialized with the pre-trained weights from LlamaGen [41). the codebook size is set 8912, and the latent space dimension is set to 32. The quantization of each scale shares the same codebook. We follow the VQVAE training recipe LlamaGen. The training is on OpenImages (21) with constant learning rate of 10 Adam W optimizer with 3 = 0.9, 32 = 0.95, weight decay 0.05, batch size of 128, and for 20 epochs. K is set 5 by default, indicating that each latent space is randomly sampled into 5 different resolutions.
FAR transformer. We provide Flex VAR in three scales, with detailed configurations for each scale provided in Tab Flex VAR is trained on the ImageNet-1K K 256x256 using 80GB A100 GPUs The training process employs the AdamW optimizer with 31 = 0.9, 3n 0.95, and a weight decay rate of 0.05. The learning rate is set to le-4, with the training epochs varying between 180 and 350 depending on the model scale.
Model name
Layers
Params.
Heads
Dims.
Epoch
Flex FlexVAR-d1e
16
310M
16
1024
180
Flex VAR-
20
600M
20
1280
250
FlexVAR-d24
24
1.0B
24
1536
350
Table 1: Configuration of Flex
4.2 Overall Comparison
We compare Flex VAR with existing generative methods on the ImageNet- -I benchmark, including GAN, diffusion models, random-scan, raster-scan, and scaling- autoregressive models. As shown in Tab. 2 ensure fair comparison, we only present models with a size smaller than IB. Our Flex VAR achieves state-of-the-art performance in all generative methods, and performs remarkably well compared to the VAR counterparts. Specifically, we achieve -0.45, -0.56, and -0.12 FID improvement compared with VAR at different model sizes.
5

---

Table Generative model comparison on class-conditional ImageNet 256x256. Metrics include Fréchet inception distance (FID), inception score (IS), precision (Pre) and recall (rec). Step: the number model runs needed to generate an image. Time: the relative inference time of VAR-d30
44
We present models with a size I IB.
Model
FID
18t
Pret
Rect
Param
Step
Time
Generative Adversarial Networks (GAN)
BigGAN
K
6.95
224.5
0.89
112M
GigaGAN 20
3.45
225.5
0,84
0.61
569M
1
StyleGan-XI.
40
2.30
265.1
0.78
0.53
166M
0.2
Diffusion Models
ADM 10
10.94
101.0
0.69
0.63
554M
250
118
CDM 18
4.88
158.7
8100
LDM-4-G 39
3.60
247.7
400M
250
D/2-L/2
B2
5.02
167.2
0,75
0.57
458M
250
2
DiT-XI//2 52
2.27
278.2
0.83
0.57
675M
250
Random-scan Manner (Mask Prediction)
MaskGIT 6
6.18
182.1
0.80
0.51
227M
8
0.4
RCG (cond.)
24
3.49
215.5
502M
20
1.4
Raster-scan Manner (Token-wise Autoregressive)
VQGAN-re
18.65
80.4
0.78
0.26
227M
256
RQTran. 22
13.11
119.3
821M
68
LlamaGen-XI
2.62
244.08
0.80
0.57
775M
256
27
AiM 23
2.56
257.2
0.81
0.57
763M
256
12
Scaling-scan Manner (Scale-wise Autoregressive)
VAR-d16 44
3.55
280.4
0.84
0.51
310M
10
0.2
Flex VAR-d16
3.05
291.3
0.83
310M
0.2
VAR-d20 44
2.95
302.6
0.83
0.56
600M
10
0.3
-AR-220
2.41
299.3
0.85
0.58
600M
10
0.3
VAR-d24 44
2.33
312.9
0.82
0.59
1.0B
10
0.5
Flex
2.21
299.1
0.83
0.59
1.0B
10
0.5
FlexVAR-224
2.08
315.7
0.83
0.59
1.0B
13
0.6
4.3 Zero-shot Comparison
Zero-shot inference with steps. We use 13 steps for image generation without training, shown in the last row of Tab. 2 FlexVAR can flexibly adopt more steps to improve image quality. By using 13 inference steps, FAe further enhances the performance 2.08 FID and 315 IS, manifesting strong flexibility and generalization capabilities. The specific steps design is detailed in the Supplementary Material.
Zero-shot inference on ImageNet
Model
Training Free
FID
IS
Params.
512x512 benchmark. We
BigGAN
X
8.43
177.9
112M
FAcxVAR-d24 to generate 512x512
ADM 10
X
23.24
101.0
554M
images and evaluate on ImageNet-512
DIT
X
3.04
240.8
675M
benchmark without training, as shown in Tab. Surprisingly, our Flex VAR- d24 exhibits competitive performance when compared VAR, despite Flex- VAR being trained only on resolutions x56x 256 and having only 1.0B pa- rameters.
MaskGIT 6
X
7.32
156.0
227M
VQGAN
X
26.52
66.8
1.4B
VAR-d36 12
X
2.63
303.2
2.3B
Flex VAR-A24 (ours)
4.43
314.4
1.0B
Table 3: Zero-shot inference on ImageNet 512x512 con- ditional generation. Training Free indicates whether the model is trained at the 512x512 resolution.
4.4 Ablation study
We conduct ablation studies on various design choices in VAR Due to the limited computational resources, we report the results trained with short training scheme in Tab. i.e., 40 epochs (~ 70K iterations).

---

Component-wise ablations. To understand the effect of each component, we start with standard VAR and progressively add each design, as shown in Tab.
Baseline: VAR uses residual prediction paradigm, exhibits decent performance (1*t result), but its flexibility in image generation does not meet expectations (as described in Sec. D
Prediction type: It is infeasible directly convert the prediction type GT, seen in the 2nd and 3rd results. We employ the VQVAE tokenizers from VAR and Llamagen, both of which yield inferior performance. This not surprising, the current tokenizers lack robustness to images with varying latent space, while we force these tokenizers to obtain multi-scale latent features during training (we provide a detailed analysis in Fig. 3
Tokenizer: Our scalable tokenizer obtains reasonable multi-scale latent features during training, resulting an improvement of -3.87 FID (the 4th results). However, flexible image generation is not accomplished yet.
Position embedding: As shown the last result in Tab. 4 the introduction of our scalable Position Embedding (PE) provides high flexibility for image generation, and further enhances the performance to 3.71 FID.
255
TEE
24
Pred. type
VQVAE
PE
FID
IS
Residual
VAR
fixed-length
4.00
226.04
Unsteperiimmed result
GT
VAR
fix-length
N/A
N/A
GT
Llamagen
fix-length
17.75
234.12
recondrect pood rosk at one
OUTS
fix-length
3.82
229.35
Llamagen
GT
OUTS
scalable
3.71
230.22
Table 4: Ablation of diverse designs. We use the next-seale-predicton paradigm, explore the ef- fects of different prediction types (residual/GTT) VQVAE tokenizers (Llamagen/VARR) positional embedding (fix- tix-lengtth/scalable).. N/A denotes the model does not converge during train- ing. We report the results with model scale -d20 trained 40 epochs on ImageNet-1e
Ours
Figure 3: Compared with VQVAE 4
41
for
and
multi-scale reconstructing images, we downsam- ple the latent features in VQVAE to multiple scales and use the VQVAE Decoder to recon- struct images. We upsample images < 100 pix- els using bilinear interpolation for a better view.
Reconstruct images with different VQVAEs. In Fig. 3 we reconstruct multi-scale images by scaling the latent features in VQVAE tokenizers. Existing VQVAE tokenizers typically do not support scaling the latent features across range of small large scales. VAR's VQVAE 44 residual-ber based training recipe, directly applying it to non-residual image reconstruction does not yield the anticipated results (the 1s row). The VQVAE tokenizer from Llamagen [41) shows excellent reconstruction performance only at the original latent space, indicating that it is not feasible for scale- wise autoregressive modeling (the nnd row).
Depth
Atten, type
FID
IS
Params.
Time
Step
h-w coordinates fixed-length
learnable
FII)
IS
-116
Transformer
4.32
209.3
310M
0.2
fix length
True
3.82
229.35
Mamba
4.22
200.04
370M
0.2
fixed-length
True
3.87
224.25
Transtiomer
3.71
230.22
600M
0.3
sealcuble
False
3.74
224.04
Mamba
3.80
216.45
700M
0.3
scaleable
True
3.71
230.22
Table 5: Ablation of the Mamba architectural.
Table Ablation Position Embedding.
X
Models trained 40 epochs (~~ 70K iterations).
denotes that the corresponding PE is removed.
Transfer FAR Mamba. Recent work, AiM 23 uses the Mamba architecture for token- wise autoregressive modeling. Inspired by this, we modify VAR with Mamba to evaluate the performance (Tab. 5). With similar model parameters, Mamba demonstrates competitive results compared to transformer models, indicating the GT prediction paradigm can effectively adapt to linear attention mechanisms like Mamba. However, considering that this Mamba architecture does not reflect the speed advantage, do not integrate Mamba into our final version.
7

---

Mamba's 's inherent unidirectional attention mechanism prevents image tokens from achieving global attention within the same scale. To address this issue, we employ 8 scanning paths in different Mamba layers to capture global information. The specific Mamba architecture is detailed in the Supplementary Material.
Position Embedding. In Tab. 66 we experiment with several types of step PE and x-y coordinate PE. To make the model robust inference steps and enable generate images any resolution, remove the fixed-length step embedding (results in the second row), and the performance showed only slight changes. We adopt a non- con-aaametric variant, similar to ViT I. which shows 0.03 FID difference compared to the learnable variant.
512
384
256
128
80
96
128
160
256
Figure 4: Generated samples from 80px to 512px. Flex VAR demonstrates strong consistency across var- ious scales and can generate 512px images, despite the model being trained only on images 256px.
Figure 5: Generated samples with various as- pect ratios. Flex VAR-d24 used. VAR demonstrates good visual quality across im- ages with various aspect ratios.
VAE Performance Comparison. We com-
Epoch
Batch-Size
rFID
pared the reconstruction performance VAR and our proposed VAE in Tab. 7 'he im- age reconstruction quality is measured by r-FID, reconstruction-FID on ImageNet validation set. We train FAR VAE with fewer epochs and smaller batch sizes. We observe its reconstruction quality inferior to VAR-VAE. Thus the improvement is not due to the quality of the discrete tokens, using a more robust Flex VAR-VAE might further improve the quality of generated images
VAR- VAE Flex VAR- VAE (ours)
1.92
3.79
Table 7: VAE Performance.
4.5 Analysis and Discussion
Generate images at various resolution. We show generated images at different resolutions using By controlling the inference steps, our Flex can generate images at any resolution, despite being trained only on images with resolutions < 256px. The generated images demonstrate strong semantic consistency across multiple scales, and the higher resolutions exhibit more detailed clarity. See the Supplementary Material for more zero-shot high-resolution generation samples and step designs.
VAR-d24 in Fig 1 4
Generate images at various ratio. We use Flex VAR-d24 to generate samples with various aspect ratios in Fig. and Fig. [ By controlling the aspect ratio at each step of the inference process, our Flex VAR allows for generating images with various aspect ratios, demonstrating the flexibility and controllability of our GT prediction paradigm. We control the height and width at each scale through approximate rounding. e.g., to generate an W0 and w Tβ. We adopt VAR's default set of 10 steps (K = {1, 2, 4, 5, 6, 8, 10, 13, 16)) to determine the size at each scale. As a result, the H x W image corresponds to ten scales with sizes {int(h x ).), int(w (w 16 h.ek.
of size H x W, the corresponding VAE
latent feature size is h x w, where h
Generate images at various step. In Fig. 6 investigate the FID and IS for generating 256x256 images from 6 to 16 steps with different sizes (depth 16. 20, 24). As the number of steps increases, the quality of the generated images improves. The improvement is (e.g., Flex VAR-d24), as larger transformers are thought able to learn more complex and fine-grained image distributions. During training, we up 10 steps avoid 0OM (out-of-memory) problem. Surprisingly, in the inference stage, using 13 steps results in performance gain of 13 FID. This observation indicates that our Flex VAR flexible with respect inference steps, allowing for fewer steps to speed up image generation or more steps to achieve higher-quality images. The details of various step designs are provided in the Supplementary Material.
significant in larger models
8

---

FID flower betcer)
IS (higher better)
2.08
(Celaalt)
Figure 6: Zero-shot image generation different steps (from to 13 steps). FID and IS are used for evaluation. We use I 10 steps for training, and Flex VAR can zero-shot transfer to 13 steps during inference and achieve better results.
Refine image high resolution. In
256px
256px
Fig. [7] we input low-resolution im-
ages (e.g., 256px x 256px) and en- able FlexVAR-d24 to output high- resolution refined images. Despite be- ing trained only on S 256px images, VAR effectively refines image de- tails by increasing the input image olution, such as the eyes of the dogs in the example. This demonstrates the high flexibility of Flex VAR in image- to-image generation.
512px
Input
512px
Ipput
Output
Output
Figure Zero-shot image refinement high resolution. Zoom in for a better view.
Ingut
Generated
In Deppaiiinng
Out paiciinnng
Inpur
Inpur
Figure 8: Zero-shot evaluation in/out-painting. The results show that FlexVAR can generalize novel downstream tasks without special design and finetuning.
Figure 9: Zero-shot evaluation image expansion. The results show that Flex VAR can generalize novel downstream tasks without special design and fine-tuning.
Image in in-painting and out-painting, For in-painting and out- out-painting, we teacher-force ground- truth tokens outside the mask and let the model only generate tokens within the mask. Class label information is also injected. The results are visualized Fig. Without modifications to the architecture design or training, FlexVAR achieves decent results on these image-to-image tasks.
Image extension. For image extension, we extend images with aspect ratio of 1:2 for the target class, with the ground-truth tokens forced to be in the center. The results are visualized in Fig. 9 FlexVAR shows decent results in image extension, indicating the strong generalization ability and flexibility of our architecture.
Failure case. FlexVAR fails to generate images with resolution 3x more than the training resolution, as illustrated in Fig. [0] These cases typically feature noticeable wavy textures and blurry areas in the details. This failure is likely due to the overly homogeneous structure of the current training dataset. i.e., ImageNet- 1K generally lacks multi-scale objects ranging from coarse to fine, leading to errors in generating details of high-resolution objects.
hypothesize that training the model with a more complex dataset that includes images with fine-grat finengrained details, the model might become robust for higher resolutions.

---

768
768
1024
Figure 10: Failure cases at high resolutions (768 & 1024). FlexVAR shows wavy textures when generating high-resolution images. Zoom in for better view.
5 Conclusion
In this paper, introduce FlexVAR, flexible visual autoregressive image generation paradigm that allows autoregressive learning without residual prediction. We design a scalable VQVAE tokenizer and TeexVAR-Transffrne for this purpose. This ground-truth prediction paradigm endows the autoregressive model with great flexibility and controllability, enabling image generation at various resolution, aspect ratio, and inference step, beyond those used during training. Moreover, it can zero-shot transfer to various image-to-image generation tasks. We hope Flex VAR will serve as solid baseline and help ease future research of visual autoregressive modeling and related areas.
We observe that when generating images with a resolution 2 3x larger than the training image, noticeable wavy textures appear (Fig. 10p. This issue may be attributed to the homogeneous structure of the ImageNet- IK training set. We will investigate this further future work to explore how to ensure stability in zero-shot image generation at higher resolutions.
Limitations.
Broader Impact. This research strictly follows established practices for class-to-image (c2i) model training and evaluation. Similar to most generative models, our approach may inherit biases present in the training datasets. We advocate for the responsible use of this technology and caution when deploying it in real-world scenarios.
Acknowledgments and Disclosure of Funding
This work is supported by the National Natural Science Foundation of China (No. 92470203, U23A20314, U24B20179, 6120106009) the Beijing Natural Science Foundation (No. L242022), and the Fundamental Research Funds for the Central Universities (No. 2024XKRC082).
References
[1] Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexci A Efros. Sequential modeling enables scalable learning for large vision models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 22861-22872, 2024.
[2] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint 2rXiv:2106.08254, 2021.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arxiv 2018. arXiv preprint 1rXiv:1809.11099 1809.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leamerr.. Advances in neural information processing systems, 33:1877-1901, 2020.
[5] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Hsuan Yang. Kevin Murphy, William T Freeman. Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint 0rX1023301.00004 2023.
10

---

[6] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patiern Recognition. pages 11315-11325, 2022.
[7] Zigeng Chen, Xinyin Ma, Gongfan Fang, and Xinchao Wang. Collaborative decoding makes visual auto-regressive modeling efficient. arXiv preprint arXiv:2411.1778. 2024.
[8] Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang. Autoregressive video generation without vector quantization. arXiv preprint 2111:24422.1169, 2024.
[9] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 10Xiv:8110.0490000 2018.
[10] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[11] Alexey Dosovitskiy. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11922 11929, 2020.
[12] Patrick Esser, Rohin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[13] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint 2xXiv22//..0052,222 2023.
[14] Jiatao Gu, Yuyang Wang, Yizhe Zhang, Qihang Zhang, Dinghuai Zhang, Navdeep Jaitly, Josh Susskind, and Shuangfei Zhai. Dart: Denoising autoregressive transformer for scalable texl-to-image generation. arXiv preprint 2rXiv:2410.08159, 2024.
[15] Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yugi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. arXiv preprint uxXiv:211.004431222 2024.
[16] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16000-16009, 2022.
[17] Wanggui He, Siming Fu, Mushui Liu, Xierui Wang, Wenyi Xiao, Fangxun Shu, Yi Wang. Lei Zhang, Zhelun Yu, Haoyuan Li, et al. Mars: Mixture of auto-regressive models for fine-grained text-to-image synthesis. arXiv preprint arXiv:2407.07614 2024.
[18] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffiusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
[19] Vincent Tao Hu, Stefan Andreas Baumann, Ming Gui, Olga Grebenkova, Pingchuan Ma, Johannes Fischer, and Björ Ommer. Zigma: A dit-style zigzag mamba diffusion model. In European Conference on Computer Vision, pages 148 166. Springer. 2024.
[20] Minguk Kang, Jun-Yan Zhu, Richard Zhang. Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10124-10134 2023.
[21] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, al. The open images dataset v4: Unified image classitication, object detection, and visual relationship detection at scale. International journal of computer vision, (28(7):1956) 1956-191111, 2020.
[22] Doyup Lee, Chibeon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11523-11533 2022.
[23] Haopeng Li, Jinyue Yang, Kexin Wang, Xuerui Qiu, Yuhong Chou, Xin Li, and Guoqi Li. Scalable autoregressive image generation with mamba. arXiv preprint a40iv:2408. 000008./22445 12245, 2024.
[24] Tianhong Li, Dina Katabi, and Kaiming He. Return of unconditional generation: A self-supervised repre- sentation generation method, In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
11

---

tasks. In The Eleventh International Conference on [25] Xiang Li, Kai Qiu, Hao Chen. Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregressive image generation with folded tokens. arXiv preprint erXiv:2410.01756 2024.
[26] Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng Ii, and Peng Gao. Lumina-mgpt: Illuminate flcxible photorealistic text-to-image generation with multimodal generative pretraining. arXiv preprint arXiv:2408. 02657, 2024.
[27] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unitied-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26439-26455, 2024.
[28] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and Learning Representations, 2022.
[29] Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang. Limin Wang, and Ying Shan. Open-magvitee An open- source project toward democratizing auto-regressive visual generation. arXiv preprint arXiv:2409.044 2024.
[30] OpenAL. Introducing chatgpt. htttp:///openaa.ccm///ooo///cche
2022.
[31] OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[32] William Peebles and Saining Xic. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages -195-4205- 2023.
[33] Aditya Ramesh, Mikhail Paylov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821-8831. Pmlr, 2021.
[34] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals, Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[35] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[36] Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, and Liang-Chine Chen. Flowar: Scale-wise autoregressive image generation meets flow matching, arXiv preprint arXiv:2412.15205, 2024.
[37] Sucheng Ren, Yaodong Yu, Nataniel Ruiz, Feng Wang. Alan Yuille, and Cihang Xie. M-var: Decoupled scale-wise autoregressive modeling for high-quality image generation. arXiv preprint arXiv:2411.10433. 2024.
[38] Zhongwei Ren, Yunchao Wei, Xun Guo, Yao Zhao, Bingyi Kang. Jiashi Feng, and Xiaojie Jin. Videoworld: Exploring knowledge leaming from unlabeled videos. arXiv preprint wxXiv:2501.09750 2025.
[39] Robin Rombach, Andreas Blattannn, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[40] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan large diverse datasets. volume abs/2201. 2000220010023,, 2022.
[41] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autore- gressive model beats diffusion: Ilama for scalable image generation. arXiv preprint arXiv:2406.06522, 2024.
[42] Haotian Tang, Yecheng Wu, Shang Yang. Enze Xie, Junsong Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Yao Lu, and Song Han. Hart: Efficient visual generation with hybrid autoregressive transformer. arXiv preprint arXiv:2410. 10812, 2024.
[43] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. Stti:24055098/45 2024.
arXiv preprint
[44] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404. 02905. 2024.
[45] Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference machine learning, pages 1747-1756. PMLR, 2016.
12

---

[46] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning, Advances in neural information processing systems, 30, 2017.
[47] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[48] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu. Zhijie Chen, Zhenheng Yang. and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 00..008..22228. /2528, 2024.
[49] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv:21 10.04427. 2021.
[50] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsual Hsuan Yang. Yuan Hao, Irfan Essa, et Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10459-
10469, 2023.
[51] Qian Zhang, Xiangzi Dai, Ninghua Yang, Xiang An, Ziyong Feng, and Xingyu Ren. Var-clip: Text-to- image generator with visual auto-regea regressive modeling. arXiv preprint 21X1012408.01111, 2024.
[52] Chuanxia Zheng. Tung-Lowe Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for sigh-iidelity image generation. Advances in Neural Information Processing Systems, 35:23412-23423 2022.
[53] Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. 11039, 2024.
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction outline our contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in the Conclusion section.
14

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical assumptions and claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The detailed information about model designs and experimental settings in the paper makes it possible for researchers reproduce the model with the same public dataset.
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code will be released. The datasets are all public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://niis.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttppp.cc/pbllc/guudeeee http://oeeubiiissioooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental setting is presented in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: It is uncommon for top conference papers in this area to report error bars similar statistical measures. Our paper aligns with this standard of other SOTA papers, which are the key criteria for evaluation in this domain.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: This paper provides sufficient information on the computer resources (type of GPU, inference time)
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////oorpss...ppppb../iiiit
Answer:
[Yes]
Justification:
The research conducted in the paper conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
This paper provides a discussion potential socictal impacts in the Conclusion
section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [No]
18

---

Justification:
The datasets used are public datasets from existing papers.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: his paper cites the original papers for their code or datasets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
19

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology in this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non otanndaddrrd components.
Please refer to our LLM policy (https://nnn) ccc//ooeeeenccc//222//////////// for what should or should not be described.
20

---

A Inference steps
In Tab. 8 we list the scales corresponding to different inference steps. The scales in each step are fixed and can be flexibly adjusted during inference. Note that during training, we only limit the maximum number of steps to 10 and randomly sample the scale for each step, so the scales during training process do not follow Tab. g
Reso
Step
Scale
6
(1, 4, 10, 16)
7
11,2,335,8. 16)
8
(1,2,3,4,6, 10, 13, 16)
9
(1,2.3,4.5,7,100, 13, 16)
256px
10
81,2,3,4,5,6.8, 10, 16)
11
1,223,445,66799111111113 16]
12
[1.2.3.4.556,7,88
10, 14, 16) [1.2334455666.88899100)0)
13
14, 16)
384px
11
(1,2,3,4.5,6,
13,
12
6, 8, 10, 13, 16, 23, 32] 3, 4. 5, 6, 7, 8, 9, 10, 12, 14, 16, 23, 32} [1.
512px
15
Table Scale configurations of various inference steps.
B Extent visual autoregressive modeling with Mamba
Unlike attention mechanisms that utilize explicit query-key-vee -ey- (QKV) interactions to integrate context, Mamba faces challenges handling bi-directional interaction. Therefore, prior Mamba- based visual autoregressive work (37 only used Mamba to model the unidirectional relationship between scales, relying on additional Transformer layers process tokens within one scale.
In this work, we adopt network. Specifically, we utilize a Zigzag scanning strategy [19] over the spatial dimension. We alternate between eight distinct scanning paths across different Mamba layers (as shown in Fig. which include:
strategy to obtain global information in Mamba
(a) top-left to the bottom-right.
(b) top-left to the bottom-right.
(c) bottom-left to the top-right.
(d) bottom-left to the top-right.
(e) bottom-right the top-left.
bottom-right to the tof-left.
gg) top-right to the bottom-left.
(h) top-right to the bottom-left.
C
Qualitative results with different steps.
we show some generated samples with (6, 8, 10, 12) steps. Our Flex VAR uses up to 10 steps for autoregressive modeling during training to avoid OOM (out-of-memory). while it can naturally transfer to any number of steps during inference. The samples generated with different steps are highly similar, differing only in some details. Generally, more steps result in better image details.
In Fig. 12
21

---

g1
92
Mamba layer 1
Mamba Louey N
Path
Path
Path
Path 8
g1
g2
Figure 11: Sptial scan paths for Mamba.
D Qualitative results with various resolutions.
Fig. 13 shows some generated samples with [256. 384, 512) resolutions. FlexVAR uses up to 256 256 resolution images for training, it can generate images with higher resolutions such as 384 and 512. The generated images demonstrate strong semantic consistency across multiple scales, and the higher resolutions display more detailed clarity.
Qualitative results with different VQVAE tokenizers.
Image reconstruction. We compare more image reconstruction results in Fig. 14 First, encode the image into the latent space and performe multi-scale downsampling, then reconstruct the original image through the VQVAE decoder. It is evident that only our scalable VQVAE can perform image reconstruction at various scales.
Generate images with GT prediction. We visualize the generated samples with VQVAE tokenizers from VAR. Llamagen, and ours, corresponding to the 2nd 3rd and 5th results in Tab. 6 in the main paper. As shown in Fig. 15| the VAR tokenizer, trained with residual paradigm, fails to generate images under GT prediction; the generation samples Llamagen's 's tokenizer are not up to the mark, due to its discrete tokens at intermediate steps being suboptimal.
F Additional Visual Results.
We show more generated samples in Fig. 16
22

---

10 (default)
12
256px
384px
512px
Figure 12: Some generated samples with (6, 8, 10, 12) steps. Note the model is trained with steps < 10. More steps typically result in better image details. Zoom in for better view.
Figure 13: Some generated samples with {256. 384, 512] resolutions. Note the model is trained with resolution of 256x256 Zoom in for better view.
266
VAR
Llamagen
Ours
VAR
Llamagen
hex11912)
Outs
Llamagen
ours
Figure Compared with different VQVAE tok- enizers 44 41 for multi-scale reconstructing im- ages, we downsample the latent features in VQVAE to multiple scales and then use the VQVAE Decoder to reconstruct images. We upsample images < 100 pixels using bilinear interpolation for a better view.
Figure 15: Some generated samples with differ- ent VQVAE tokenizers (Llamagen & VAR), cor- responding to the 2nd and 3rd results in Tab. 6 in the main paper. We report the results with model scale -d20 trained 40 epochs (~ 70K iterations) ImageNet- I Zoom in for a better view.
23

---

Figure 16: Some generated 256 samples.
24


Reviews:
[{"review_id": "rHrBy8nAL4", "reviewer": "Reviewer_zR9e", "summary": "The paper proposes FlexVAR for image generation which builds on the VAR (visual autoregressive modelling) framework. Similar to VAR, the discrete token sequences are formed by using VQ tokens of different image scales. Different from VAR, the proposed VQVAE directly predicts the ground-truth of the image instead of the residual between different image scales. Experimental results show FlexVAR achieves similar performance to VAR.", "questions": "1. Could the authors clarify over the claim of FlexVAR outperforms VAR? VAR paper reports 2.09 FID on ImageNet, not 2.33 as reported in Table 2. At equivalent inference budget, FlexVAR performs worse than VAR. The better result requires more inference compute. In addition, if same technique of step dropping is applied to VAR during training, VAR might achieve similar or even better performance?\n\n2. Could you provide a quantitive comparison of VQVAE performance trained by predicting groundtruth to that trained by predicting residual? Maybe the improvement is solely due to the quality of the discrete tokens? \n\n3. Does FlexVAR scale similarly to VAR? From the trend of FlexVAR 3.05->2.41->2.21 v.s. VAR 3.3->2.57->2.09, it seems that FlexVAR is not making good use of the additional model capacity? Could the authors provide some discussions?\n\n4. Could you explain in more detail how the aspect ratio is controlled during the inference process?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is clearly written and experimental results seem strong.\n\nBy using scalable positional encoding, the paper demonstrates the ability of the model to generate images of different aspect ratios. \n\nIt confirms that using ground-truth works out similarly to using residual, as least up to a certain scale.\n\n**Weaknesses**\n\nThe motivation is not super clear. Is there a theoretical reason why predicting ground-truth is better or worse than predicting residuals? Is predicting ground-truth unlock additional capabilities? The flexibility of generating images of different aspect ratios comes from using scalable positional embedding. Adaptive to more steps comes from randomly dropping steps during training time. These techniques are equally applicable to VAR.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments, and here are the answers to your concerns.\n\n\n**W1: Motivation.**  \nResidual modeling cannot achieve the flexibility of generating, as it relies on a weighted sum of all scales. This dependence requires a consistent scale design during both training and inference. Additionally, residual modeling mandates using every scale in the weighted sum, thus preventing step dropping during training.\nWe analyze the differences between two structures in L50-L52: residuals at different scales often lack semantic continuity, and this implicit prediction approach may limit the model’s ability to represent diverse image variations.\nFurthermore, the ablation study in Tab. 4 experimentally demonstrates that predicting ground truth is an effective approach.\n\n\n**Q1: VAR performance clarification.**  \n1) We perform evaluations using the official VAR model, and the results presented in Tab. 2 align with the performance reported by the VAR team on GitHub. Additionally, concurrent work M-var [36] also verifies the performance of VAR (Tab. 2 in [36]). Therefore, at an equivalent inference budget, FlexVAR demonstrates superior performance compared to VAR.   \n2) Step dropping is to enable the model to adapt to varying steps, which does not result in significant performance gains, as discussed in **W1**.\n\n\n**Q2: VQVAE performance.**  \nDue to limited training resources, we train FlexVAR-VAE with fewer epochs and smaller batch sizes. We observe its reconstruction quality slightly inferior to VAR-VAE. Thus the improvement is not due to the quality of the discrete tokens, using a more robust FlexVAR-VAE might further improve the quality of generated images.\n\n|      | **Epoch** | **Batch-Size** | **rFID** |\n|------|-----------|----------------|----------|\n| VAR-VAE  | 20        | 768            | 1.92     |\n| FlexVAR-VAE (ours) | 10        | 128            | 3.79     |\n\n\n**Q3: Scaling capacity**\n\nAs shown in Tab. 2, the FID trends when scaling up FlexVAR and VAR are similar (VAR: 3.55 -> 2.95 -> 2.33). Additionally, according to the FID formula, the decrease in FID score is non-linear, indicating that it becomes increasingly challenging to achieve further reductions as the score lowers:  \n$$\n\\text{FID}(P, G) = \\|\\mu_P - \\mu_G\\|^2 + \\text{Tr}(\\Sigma_P + \\Sigma_G - 2 \\cdot (\\Sigma_P \\Sigma_G)^{1/2})\n$$\nWhere $\\mu$ and $\\Sigma$ represent the mean and covariance matrix of two distributions, respectively, and $\\text{Tr}(\\cdot)$ denotes the trace of the matrix.\n\n\n\n\n**Q4: Explication of aspect ratio generation.**  \nWe control the height and width for each scale through approximate rounding. *e.g.*, to generate an $H × W$ image, the corresponding VAE latent feature size is $h × w$, where $h = \\frac{H}{8}$, $w = \\frac{W}{8}$. We use VAR's default 10-steps ($K = ${$ 1, 2, 3, 4, 5, 6, 8, 10, 13, 16 $}) to determine the size of each scale. Consequently, the H × W image corresponds to ten scales with sizes {$(\\text{int}(h \\times \\frac{i}{16}), \\text{int}(w \\times \\frac{i}{16}))$}$_{i \\in K}$. We will add more detailed explanations in the final version."}, {"review_id": "Ifwlq4f7Dh", "reviewer": "Reviewer_B8QN", "summary": "This paper challenges the dominant residual-prediction paradigm in VAR modeling. The authors propose FlexVAR, a new framework that instead predicts the ground-truth image representation at each generative step. This seemingly simple change, enabled by a novel scalable VQVAE and scalable positional embeddings, unlocks remarkable flexibility. Trained only on low-resolution images (≤256px), FlexVAR can zero-shot generate images at higher resolutions and arbitrary aspect ratios, perform various image-to-image tasks like inpainting and refinement, and dynamically trade speed for quality by varying its inference steps, achieving state-of-the-art results on ImageNet.", "questions": "1. Regarding the GT-prediction paradigm and error propagation: How does the model behave when it makes a significant error at a coarse scale (e.g., step 2 or 3)? Does the model learn to \"correct\" or ignore this incorrect context in later steps, or does the error tend to compound, leading to a flawed final image? An analysis of failure cases related to error propagation would be very insightful.\n\n2. Regarding the GT-prediction paradigm and error propagation: How does the model behave when it makes a significant error at a coarse scale (e.g., step 2 or 3)? Does the model learn to \"correct\" or ignore this incorrect context in later steps, or does the error tend to compound, leading to a flawed final image? An analysis of failure cases related to error propagation would be very insightful.\n\n3. The ability to use more inference steps to improve quality is a key advantage. Figure 6 shows FID improving up to 13 steps. What happens at the high end? Is there a point of diminishing returns (e.g., at 20 or 30 steps), or does performance begin to degrade due to compounding minor errors over a long generative chain? Understanding the practical upper limit of this flexibility would be valuable.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1.  This work makes a compelling case against a core assumption (the necessity of residual prediction) in a popular and powerful class of generative models. By proposing and successfully implementing a direct GTprediction paradigm, the paper offers a new, simpler, and more intuitive foundation for visual autoregressive modeling. This conceptual shift has a high potential impact on the field.\n\n2. The ability of a model trained exclusively on low-resolution (≤256px) images to zero-shot generate high-quality 512px images (Table 3), handle arbitrary aspect ratios (Figure 5), and perform a suite of image-to-image tasks without any fine-tuning (Figures 7-9) is a significant breakthrough and a testament to the power of the GT-prediction paradigm.\n\n3.  FlexVAR not only introduces a more flexible paradigm but also achieves excellent quantitative results. The 1.0B model surpasses its direct VAR counterpart and, when using more inference steps (a unique capability of the method), achieves a 2.08 FID on ImageNet 256x256, outperforming strong diffusion (DiT-XL/2) and autoregressive (AiM) models (Table 2). This demonstrates the method's effectiveness without sacrificing quality.\n\n**Weaknesses:**\n\n1. The paper's core paradigm, ground-truth prediction, lacks an explicit error-correction mechanism. In the residual-based VAR, each step can be seen as correcting the upsampled version of the previous scales. In FlexVAR, if the model generates an imperfect latent map at an early, coarse scale (e.g., predicts the wrong object silhouette), this error is fed forward as conditioning information to all subsequent, finer scales. The paper does not analyze or discuss this potential for compounding errors, which could lead to instability or inconsistencies in the final output, especially for complex scenes.\n\n2. The proposed solution to make the VQVAE robust to varying latent scales is to train it on reconstructions from multiple, randomly downsampled versions of the latent map. While empirically shown to work (Figure 3), this approach feels more like a data augmentation trick than a fundamental architectural innovation. \n\n3. In Table 2, the \"Time\" column provides a relative inference time comparison, which is very helpful. However, the best FID score for FlexVAR-d24 (2.08) is achieved with 13 steps and a relative time of 0.6, while the baseline models it is compared against, such as AiM, have a much higher relative time of 12. This comparison, while favorable for FlexVAR, does not compare models under an iso-time budget. A more direct comparison would involve evaluating baseline models under a similarly constrained time budget to better contextualize FlexVAR's efficiency.", "was_revised": true, "final_justification": "After reading the rebuttal, almost all concerns have been resolved. Therefore, I keep my original positive score.", "rebuttal": "Thank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments, and here are the answers to your concerns.\n\n\n**W1 & Q1: Analysis of error-correction**  \nVAR predicts residuals at each step and weights the sum of all step residuals.  These residuals are not equivalent to errors, thus there is **no theoretical basis for VAR explicitly correcting errors**. Both VAR and our FlexVAR can see all previous scale tokens, allowing them to **implicitly correct previous errors**. From this perspective, predicting ground truth and predicting residuals show no significant difference in error accumulation. Additionally, Tab. 2 demonstrates that predicting ground truth is an effective approach.  \nTo further analyze error propagation,  we progressively added Gaussian noise within 1% of the maximum response scale at the early steps (*e.g.*, step \\{1\\} or steps \\{1,2,3\\}) to simulate the error generation process during inference. The FID results under different noise patterns are shown in the table below. It is evident that our FlexVAR consistently produces better quality than VAR under varying noise patterns.\n\n|         | **{-}** | **{1,}** | **{1,2}** | **{1,2,3}** | **{2,3}** |\n|------|---------|----------|-----------|-------------|-----------|\n| VAR-d20     | 2.95    | 3.47     | 3.54      | 2.91        | 3.18      |\n| FlexVAR-d20 | 2.41    | 2.35     | 2.33      | 2.49        | 2.53      |\n\nNote: FID is used for evaluation, where a lower score indicates better performance.\n\n\n\n**W2: The proposed VAE lack innovation.**  \nWe do not emphasize VAE as a fundamental architectural innovation. Our VAE follows the architecture of Llamagen and VQGAN, and the training of VAE is aimed at fitting the architecture that removes residuals. Its training primarily relies on data augmentation methods rather than structural improvements.\n\n\n\n**W3: The relative inference time favorable for FlexVAR.**  \nThere may be some misunderstandings about the relative inference time. We would like to clarify it as follows. \nRelative time is a time evaluation metric used by VAR, which **provides an intuitive reflection of inference time**. This metric is not favorable for FlexVAR.  Relative time represents the multiple of inference time required in comparison to VAR-d30 (as shown in Tab. 2 caption). *i.e.*, on an A100 GPU, the inference time for VAR-d30 is 0.5s. Therefore, multiplying 0.5 seconds by the corresponding relative time metric yields the actual image generation time for each model.\n\n\n**Q2: Zero-shot inference with more steps.**  \nA 256-resolution image corresponds to a VAE latent feature of 16 × 16, allowing for a maximum of 16 sampling steps, ranging from 1 to 16. We present the FID and IS of FlexVAR-d24 during inference at steps 14, 15, and 16 in the table below, along with results for 6-step, 10-step, and 13-step for comparison. The results indicate that both FID and IS achieve optimal values at step 14, followed by a slight decline in performance with more steps.\n\n|      | **6** | **10** | **13** | **14** | **15** | **16** |\n|------|-------|--------|--------|--------|--------|--------|\n| FID  | 3.09  | 2.21   | 2.08   | 2.07   | 2.10   | 2.15   |\n| IS   | 260   | 299    | 315    | 316    | 316    | 312    |"}, {"review_id": "TZ76zPNaar", "reviewer": "Reviewer_EER8", "summary": "This paper enhances the flexibility of VAR, which exhibits exceeding resolution generation, various image-to-image task support, and various autoregressive step adoption. The scalable tokenizer is robust to arbitrary resolution reconstructions and the flexVAR transformer models the ground-truth of each scale, which gets rid of the residual paradigm. The experiments show the effectiveness of the design.", "questions": "The design is simple yet interesting, providing new insights for efficiently generating high-resolution images. Check the weakness above.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1) The simple yet effective design of modeling each scale instead of the residuals enhances the flexibility of VAR. \nEquipped with multi-scale representation learning in VQ-VAE and scalable position embedding,  the visual autoregressive modeling on the GT images further enhances generation performance and speed.  \n2) The experimental results are sufficient and effectively convey the rationality of the design. \n\n**Weakness**\n1) Line 141 indicates the random scale sampling. Will it cause bias in specific resolution training. Will it cover high-resolution training.\n2) When inference, how the attention mask is set. Will it be conditioned on all previous scale features. I suppose only the previous n-1 scale features are enough. \n3) In Table 2. I am curious about why shifting the modelling paradigm from residuals to multi-scale modeling can bring benefits on gFID.\n4) Line 170 indicates the training flops will be relatively high if sampling the resolution higher than 512.", "was_revised": true, "final_justification": "After reading the rebuttal from other reviewers, I decide to maintain my original score since I think it provides a new insight in into how to generate a high-resolution image with only training on low-resolution ones.", "rebuttal": "Thank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments, and here are the answers to your concerns.\n\n\n**W1: Concerns about bias in specific resolution**  \nThe randomness inherent in the sampling process during training prevents the introduction of bias towards specific resolutions. Additionally, sampling only includes scales with resolutions $\\leq$ 256. This approach does not encompass higher resolutions, thereby ensuring zero-shot inference at those higher resolutions.\n\n**W2: Attention mask during inference**  \nDuring inference, each scale is conditioned on all previous scale tokens. We believe that the information from previous scales is valuable, as it helps prevent error accumulation.\n\n\n\n**W3: Why shifting the paradigm to GT modeling brings benefits to gFID.**   \nWe analyze the differences between two structures in L50-L52: residuals at different scales often lack semantic continuity, and this implicit prediction approach may limit the model’s ability to represent diverse image variations.\nFurthermore, the ablation study in Tab. 4 experimentally demonstrates that predicting ground truth is an effective approach.\n\n\n**W4: Concerns about the training flops.**   \nWe perform sampling at resolutions $\\leq$ 256, deliberately avoiding larger scales such as 512.  As a result, compared to VAR, **no additional computational overhead** is introduced. Specifically, while VAR employs 10 fixed steps for sampling, we utilize random sampling within a range of 6 to 10 steps. Consequently, the training FLOPs may vary within a certain range but will not surpass the computational demands of VAR."}, {"review_id": "pquSi9Jytj", "reviewer": "Reviewer_c8ru", "summary": "FlexVAR overhauls multi-scale autoregressive image synthesis by skipping the usual residual-based updates and learning the actual latent codes at every resolution. It pairs a VQ-VAE tokenizer—built with multi-scale constraints so it can faithfully compress and reconstruct images at any size—with a Transformer that uses trainable 2D position embeddings to arrange those codes in sequence, all without being tied to a preset step order. The unified FlexVAR pipeline can then crank out semantically consistent images across a spectrum of resolutions and flexible generation schedules.", "questions": "1. Can you provide the performance of the densest scale {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16} and sparsest scale like {1,4,16} or {1,2,4,8,16}? Will the results be upper bound and lower bound respectively?\n\n2. Can your provide more analysis why you for adopting VAR’s scale configuration by default?", "limitations": "Yes", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.\tFlexVAR can produce images in arbitrary sizes and aspect ratios.\n2.\tFlexVAR achieves greater gains as additional scales are added.\nWeakness:\n1.\tMy main concern is the claim on lines 49–50—“The residual prediction relies on a rigid step design, restricting the flexibility to generate images with varying resolutions and aspect ratios.” In fact, VAR’s limitation comes from its fixed-length positional embeddings, not from residual prediction itself. The tokenizer of VAR with residual prediction can also reconstruct across arbitrary scale sets like {1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 14, 16} (randomly picked).  FlexVAR’s added flexibility simply stems from removing those fixed-length 2D embeddings.\n2.\tFlexVAR offers no clear advantage over VAR. Although Table 2 shows marginal FID improvements at similar parameter counts, its IS scores are weaker, and Precision and Recall remain virtually unchanged. These results suggest that replacing residual prediction with direct ground-truth modeling does not yield meaningful performance gains, weakening the authors’ critique of the residual prediction paradigm.\n3.\tIt seems FlexVAR’s image visual quality lags behind VAR’s—could you present side-by-side visual comparisons of FlexVAR and VAR outputs at the same generation step?\n4.\tFlexVAR’s scalability isn’t fully demonstrated: the authors only train up to 1 billion parameters, still trailing the top performance of VAR-d30.", "was_revised": true, "final_justification": "My ranking stays at rebuttal. My viewpoint  is not altered by  other reviewers or the rebuttal. FlexVar has no clear advantage over Var. Scaling is not clear.", "rebuttal": "Thank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments, and here are the answers to your concerns.\n\n**W1: Concern about the flexibility.**  \nResidual modeling cannot achieve the flexibility of generating, as it relies on a weighted sum of all scales. This dependence requires a consistent scale design during both training and inference. Additionally, predicting residual does not allow for dropping scales because each scale is utilized during the weighted sum stage. \nWe analyzed the differences between residual modeling and GT modeling in L50-L52: residuals at different scales often lack semantic continuity, and this implicit prediction approach may limit the model’s ability to represent diverse image variations.\n\n\n**W2: Replacing residual with ground-truth modeling does not yield meaningful performance gains**  \n1) FlexVAR demonstrates higher Inception Scores (IS) across different scales (d16, d20, d24) when compared to VAR. Our FlexVAR enhances IS further by utilizing zero-shot inference with more steps.   \n2) Precision and Recall metrics do not accurately reflect image quality; *i.e.*, the Precision score of VAR across different scales (VAR-d16, VAR-d20, VAR-d24) shows a decreasing trend: 0.84 -> 0.83 -> 0.82. Therefore, critiquing the ground-truth modeling paradigm based on IS, Precision, and Recall is not accurate.\n\n\n\n**W3: FlexVAR’s image visual quality lags behind VAR’s**  \nDue to NeurIPS guidelines prohibiting anonymous links, we are unable to present side-by-side visual comparisons at this time. However, we have included a selection of generated samples in the *supplemental material* to showcase the visual quality of FlexVAR's images.\n\n\n\n**W4: FlexVAR’s scalability isn’t fully demonstrated**  \nFlexVAR at scales d16, d20, and d24 outperforms the corresponding VAR models, which sufficiently demonstrates the scalability. The performance of FlexVAR-1B being weaker than VAR-d30 (2B parameters) should not be seen as our weakness. Due to limitations in computational resources, we are currently unable to train FlexVAR-d30. Therefore, to ensure a fair comparison, we only present models with parameters less than or equal to 1B, as explained in Line 198-199.\n\n\n**Q1: Performance of the densest scale and sparsest scale**  \nWe present the FID and IS of FlexVAR-d24 during inference with 3-steps ({1, 4, 16}), 5-steps ({1,2, 4, 8, 16}), 10, 13, 14, 15, and 16-steps in the table below. The results indicate that using too few inference steps (*e.g.*, 3 steps) results in poor performance, but there is a noticeable improvement in image quality with 6 steps. Both FID and IS achieve optimal values at step 14, followed by a slight decline in performance with more steps.\n\n|      | **3**  | **5**  | **6**  | **10** | **13** | **14** | **15** | **16** |\n|------|--------|--------|--------|--------|--------|--------|--------|--------|\n| FID  | 29.13  | 7.61   | 3.09   | 2.21   | 2.08   | 2.07   | 2.10   | 2.15   |\n| IS   | 75.65  | 159    | 260    | 299    | 315    | 316    | 316    | 312    |\n\n\n**Q2: Why adopt VAR’s scale configuration by default**  \nWe use the same scales as VAR for a fair comparison."}]

Author rebuttal:

Meta-review: This paper presents a compelling and practical advancement in visual autoregressive modeling with FlexVAR. The core contribution is demonstrating that a simple but important shift—from residual to direct ground-truth prediction—and changing the positional encoding enables significant flexibility for image generation without compromising performance. This is a valuable finding for practitioners in the field.

There has been some difficulty in quantifying the performance of the baseline VAR model because the baseline paper publication and their updated github repository report conflicting metrics (2.09 vs 2.33 FID). For the purpose of this review I am willing to side with the authors; considering 2.33 the more trustworthy baseline. 

The paper's main thesis is that abandoning the residual prediction paradigm is the key to unlocking flexibility. However, as Reviewer c8ru convincingly argues, the flexibility gains may instead stem from the paper's use of scalable positional embeddings, a feature distinct from the prediction target. The authors' rebuttal did provide some clarity, however I think additional ablations disentangle these two factors more carefully.

In summary however FlexVAR is a well-executed study that delivers a clear, useful result. The method's enhanced flexibility is a solid contribution and the discussion of pros/cons of ground-truth prediction will benefit the community. I am happy to recommend this paper for acceptance and look forward to seeing the discussion it inspires.
Decision: Accept (poster)