Next Visual Granularity Generation
Authors: Yikai Wang
Zhouxia Wang
Zhonghua Wu
Qingyi Tao
Kang Liao
Chen Change Loy
Venue: ICLR 2026 Poster (2026)
Keywords: image generation
Primary area: generative models
TL;DR: Our framework decomposes images into sequences of increasing structured visual granularity and guiding generation through structure-aware mechanisms, not only improving fidelity but also opening new possibilities for structure-controlled generation.

Abstract: We propose a novel approach to image generation by decomposing an image into a structured sequence, where each element in the sequence shares the same spatial resolution but differs in the number of unique tokens used, capturing different level of visual granularity.
Image generation is carried out through our newly introduced Next Visual Granularity (NVG) generation framework, which generates a visual granularity sequence beginning from an empty image and progressively refines it, from global layout to fine details, in a structured manner.
This iterative process encodes a hierarchical, layered representation that offers fine-grained control over the generation process across multiple granularity levels. We train a series of NVG models for class-conditional image generation on the ImageNet dataset and observe clear scaling behavior. Compared to the VAR series, NVG consistently outperforms it in terms of FID scores (3.30 $\rightarrow$ 3.03, 2.57 $\rightarrow$ 2.44, 2.09 $\rightarrow$ 2.06). We also conduct extensive analysis to showcase the capability and potential of the NVG framework. Our code and models are released at https://yikai-wang.github.io/nvg.

Published as a conference paper at ICLR 2026
NEXT VISUAL GRANULARITY GENERATION
Yikai Wang', Zhouxia Wang', Zhonghua Wu', Qingyi Tao?, Kang Liao', Chen Change Loy" S-Lab, Nanyang Technological University: 2 SenseTimm Research yi-kai, wangBoutlook, com acloy@ntu.eduu edu.sq
Project Page: httts::/yikaiiiiee
gi gig.u.......... io/nvg
Construction of Visual Granularity Sequence
1
2
4
16
32
64
128
256
Next Visual Granularity Generation
(a) Construction of the visual granularity sequence 256 image and the next visual granularity generation in the 16 latent space. Top-to-bottond bottooo:: Number of unique tokens, structure map, generated image.
(b) Our model can generate diverse and high-fidelity images.
(c) The generated images align well with the generated binary structure map.
(d) can reuse structures from reference images (wallaby, flamingo) to generate new ones (rabbits, heron).
We propose Next Visual Granularity (NVG) generation framework, representing images with varying number unique tokens, naturally forming different granularity levels. The induced structure maps reflect how these tokens assigned across different spatial locations. The structure maps and unique tokens are iteratively generated to gradually refine the generated image.
Figure 1:
ABSTRACT
We propose novel approach to image generation by decomposing an image into a structured sequence, where each element in the sequence shares the same spatial resolution but differs in the number of unique tokens used, capturing different level of visual granularity. Image generation carried out through newly

---

Published as a conference paper at ICLR 2026
introduced Next Visual Granularity (NVG) generation framework, which generates a visual gramularity sequence beginning from an empty image and progressively refines it, from global layout to fine details, in a structured manner. This iterative process encodes a hierarchical, layered representation that offers fine-grained control over the generation process across multiple granularity levels. We train a series of NVG models for class-conditional image generation on the ImageNet dataset and observe clear scaling behavior. Compared to the VAR series, NVG consistently outperforms it in terms of FID scores (3.30 - 3.03, 2.57 - 2.44, 2.09 + 2.06). We also conduct extensive analysis showcase the capability and potential of the NVG framework. Our code and models are released at https: /yykkii-wand .ithubt ho/iii/obboooggggggg
INTRODUCTION
How do generative models understand images? Different generative models interpret images in distinct ways: Token-based models (such as autoregressive (Chen et al.) 2020 Esser et al. 2021 and masked token modeling methods (Chang et al. (022 treat images as visual sentences, processing them similarly to how they handle language. GANs (Goodfellow et al., 2014), diffusion (Ho et al.) 2020), and flow (Liu et al.) [023 Lipman et al. 2023) models see images as samples of a high- dimensional probability distribution over the raw pixel space learned latent space. Visual autoregressive models (Tian et al.; 2024) break images down into multiple resolutions using a residual visual pyramid. In these cases, the image generation process is framed as either modeling a conditional probability distribution over previous tokens/scales or a stochastic process between the real data distribution and a random distribution.
While these approaches have led to powerful generative models, each comes with limitations in how they view and handle images: Treating an image as a sequence (like a sentence) or a pyramid often ignores the rich and complex spatial structure of images. The autoregressive methods rely on unidirectional generation, which neglects the inherent 2D spatial structure when generating early tokens, and suffer from error accumulation, also known as exposure bias (Ranzato et al., 2016). Visual autoregressive methods may mix up nearby visual information from distinct semantics and have to handle miscellaneous information. Modeling images purely as distributions often requires significant fine-tuning or extra modules (Zhang et al. 2023b) to control the generation process.
In this paper, introduce structured visual granularity, representing images as structured sequences. Figure Ta) provides an illustration of this sequence. At each stage, the image is described using different number of unique tokens in the same spatial size. This allows to create structure map that shows how the tokens are arranged across the latent space. The structure map naturally captures the image's s granularity at different levels. This structured sequence can be used train an image generation model that can generate images more naturally and with greater structure control.
We propose data-driven method build this visual granularity sequence. We use a bottom-up strategy, repeatedly clustering the most similar tokens until all tokens are merged into single cluster that represents the entire image. Ideally, as the number of unique tokens decreases, the image structure gradually emerges: progressing from fine details to parts of the object, to objects, then to basic backgrock separation and ending with single cluster.
Based on this sequence, we present a new Next Visual Granularity (NVG) generation process that mirrors the intuitive, coarse-to-fin progression commonly observed in art painting. Specifically, starting from an empty image, we gradually add more details in structured manner by generating the structure map and corresponding tokens. In this way, we begin with coarse structures like foreground and background, then add object shapes, object parts, and finally fine details.
We train a series of NVG models of varying sizes on the ImageNet class-conditional image generation task. Our results reveal a clear scaling trend: performance consistently improves with larger model sizes, highlighting the scalability of our framework. Compared to other sfate-of-the-ae art image generation models, NVG achieves comparable or superior performance. In particular, when compared to VAR, our model consistently achieves better FID, IS, and recall scores across all model sizes. Figure [b) shows examples of diverse generated images. They align well with the generated binary
2

---

Published as a conference paper at ICLR 2026
Stage 0 {Co, So
Stage 1 S1
Stage
Stage K
Content C E 2
S2
Sx]
Stage
Structure
5/ E [0, xxw
X
X1
X
Figure 2:
The relationship between current generated images each stage H;, the final generated image and the content and structure 8; of each stage in the visual granularity sequence.
structure maps, shown in Figure By reusing the structure maps, NVG transfer the structure from one image to another, generating new images with varied content, as shown in Figure [d).
The gen- eration process follows natural progression of visual granularity. We introduce a structure map that explicitly controls the level of granularity in the latent space. Explicit structure control: Each generation step controls specific level of granularity. This structured approach enables natural control during generation itself, rather than relying on extra conditional modules trained post-hoc.
Our approach offers several key advantages: (1) Structured coarse-to-fine generation:
Compared to auto-regressive models, NVG uses residual modeling inspired by VAR, where each stage predicts the quantization error from the previous stages to the ground truth. This naturally reduces exposure bias. Compared to diffusion models, we introduce an explicit structure-come centture-contoolledd generation pipeline, making structure control an integral part of the process rather than an extra post-hoc module. Compared to VAR, our visual granularity decomposition alleviates representation ambiguity the early stages, where a single token represents a large and semantically diverse image region. The granularity -ased decomposition improves the meaning of each token, leading to better performance in both reconstruction and generation.
NEXT VISUAL GRANULARITY GENERATION
In this paper, we present the concept structured visual granularity, where images are represented structured sequences composed of varying numbers of distinct tokens distributed in the latent space at each stage. An illustration of this sequence is shown in Figure [a). Our method organizes the token sequence inherently capture image granularity across multiple levels. This structured representation allows the model to generate images in a more organized and interpretable manner.
In the following, first describe how construct the visual granularity sequence by training multi-granularity quantized autoencoder (Sec. 2.1). We then explain how to generate this sequence for structured image generation (Sec. 2.2
2.1 VISUAL. GRANULARITY SEQUENCE
Tokenization. We propose multi granulartyy quantized autoencoder that represents images as structured sequences across multiple levels of granularity. Specifically, an image encoded into a latent representation Z xxx where h and denote the spatial dimensions and is the channel dimension. We propose constructing a visual granularity sequence that represents quantized latent representation. In particular, the sequence consists of content and structure pairs of multiple stages K, denoted as T {ci, 8; { where the contents of different stages are derived from a shared codebook V E Raxe
In stage i, the latent of size x w is represented by n, unique tokens, dubbed as contents, i.e., |cl T4; C; V. The structure s; is a matrix of size h w. indicating the arrangement of each token in the latent space of size h x w where the value of each position is the index in the corresponding content token, 8; C (0, I...., ........- n4- 1) hxw, See Figure 2 for an illustration. The latent x is quantized as K a(c, st) where is the assignment operator that places the e; into the latent space according to the arrangement of 8;, K denotes the final quantization stage.
3

---

Published as a conference paper at ICLR 2026
Finberdingg
Embeedding
Embeedinn
420033
4,0M5
4,003
4,000
4,003
Stage
Stagg
Slage2
Figure 3: We use a K-dim vector to encode the structure across all stages. At stage O, all locations belong to single cluster, we pad the vector with all 1s. For stages O, the embedding is inherited from the parent and extended with one cxtra bit (0 or 2) to distinguish between child labels.
Structure Construction. A structure map defines the arrangement of tokens across the latent space. To demonstrate the flexibility of our model, we propose a fully data-driven clustering approach to construct T. Starting from the finest granularity stage, where each position is assigned a unique token, i.e., |ck| = lew, we progressively group visually similar points. This can be achieved using methods like k-nearest neighbors clustering (Fix & Hodges, 1989), graph cuts (Greig al. 1989). or linear assignment (Kuhn 1955). As an initial exploration, we adopt a simple yet efficient greedy strategy, grouping tokens into equal-sized clusters.
Specifically, we begin by computing pairwise lg distances between all tokens and grouping the top-k most similar tokens into cluster. We then remove the clustered tokens from the pool and repeat the process on the remaining tokens until all tokens are assigned to a cluster. This yields the penultimate token label map s effectively reducing the number of unique tokens by k. We repeat this clustering process iteratively until all tokens are merged into a single cluster, resulting in a hierarchy of multi-stage label maps {s; We set k 2, reducing the number of tokens by half in each stage. With the structure maps defined, we now derive the corresponding content tokens.
Content Construction. We construct the multi-stage tokens in a residual manner, forming visual pyramid similar to VAR (Tian et al., 2024]. However, unlike the spatial resizing used in VAR. our compression is guided by the induced structure map. With this design, we have a sequence of unique tokens with a count of { for latent space of size 162. Algorithm | summarizes our construction algorithm. The detailed implementation provided in App. A.2
Structure Embedding. We introduce compact hierarchical structure embedding for multi-stage 1 label maps {/ that preserves parent-child relations, distinguishes stages within a unified space, and avoids embedding cluster ID order. An example of this embedding process is shown in Figure 3 Each stage adds a bit (0 or 2) to the parent's 's K Kiddmmnsionnl bit-style vector, with as padding; stage 0 is fully padded. This integer- valued design is RoPE-compatible and enables simple, efficient embedding construction from class and stage IDs. Padding reveals the stage, while bit patterns separate clusters. We discuss this design in App. A.3
2.2 NEXT VISUAL GRANULARITY GENERATION
Generation Pipeline. We denote the generation of content tokens as content generation. and the generation structure maps as structure generation. To support these, we design the Next Visual Granularity (NVG) generation framework. At each stage, we first generate the structure, followed by the corresponding content. This design allows users to optionally provide a preferred structure to guide and control the generation process. An illustration of our generation pipeline is shown in Figure 4 with the detailed algorithm provided Algorithm 2 and structure generation, respectively. We provide the architecture details in App. A.4
We use separate models for content
2.2.1 STRUCTURE GENERATOR
As we explicitly represent each stage using separate channels in the structure embeddings, directly generating the whole hierarchical structure in one-step challenging. Furthermore, the structure generator must produce full binary cluster map of the image in first step. This task is complex and plays a crucial role in guiding successful image generation. We regard this as the "cold-start" issue.
On the other hand, compared to content generation, which typically uses large codebook and high-dimensional channels, structure generation is simpler. It only needs to generate 8-channel embeddings, with maximum complexity of 2 256. Intuitively, for structure generation while applying more advanced techniques to better model the structure distribution and balance quality with efficiency.
use smaller model
4

---

Published as a conference paper at ICLR 2026
Partial Noise
Boudano Hantt
Newckffic
Predicted
Sirudtre Embadiing
Stucture Transforme
Trandfrrmer
Stracuuee
S1-1
Predicted Final Canvas
Class
<dog>
xy(l 2)
X
Trandarmes
Content
Trandormet
Content
6/2
X/-1
Figure 4: Left: Overview of our generation pipeline. At each stage, we first generate the structure and then generate the content based on that structure. Both steps guided by the input text, the current canvas, and the current hierarchical structure. Right: Overview of how we obtain the target from the network predictions. (1): The structure generator predicts the overall structure embedding, from which we extract the channel for the next stage. (2): The content generator predicts the final canvas. We compute the residual between the predicted final canvas and the current canvas -; and use it to obtain the next-stage content prediction.
Based on these considerations, we propose using a lightweight rectified flow model (Liu et al., 2023 Lipman et al. 2023) for structure generation. To this end, we design a unified structure generation process that spans all stages. This approach allows training from later stages to also inspire the early stages, helping to avoid cold-start issues.
Input. We use the structure embeddings 8g (see Figure 3 for illustration) as the initial input. We then add the noise to the embeddings, leading (t) t + +1 - t) 8.. In the later stages of generation, the outputs from earlier stages are already known and fixed. Hence, for these known parts, use ground-truie -rutt embeddings instead of noised embeddings, i.e.,
2. (1) 0 = :
This makes our structure generation task structure "inpainting" task. We concatenate the text and current generated image in the spatial channel as the condition for structure generation.
Structure Prediction. We use the u-prediction (Salimans & Ho 2022) to estimate the velocity during the denoising process. During sampling, in stage O, we start with s, filled with 1. We then use random noise as input and gradually denoise to generate the final hierarchical structure embedding &g. In the early stages, the generated full hierarchical structure embedding may be inaccurate because it relies only on the image produced so far. Therefore, at stage given the structure maps from stages generate the full structure embedding, then utilize the generated structure map of stage to update 8, as:
0 to i 1 and the image refined from stage i
se [ 1] < i
(2)
Note that all we need for the generated structure is to split each parent label evenly in the latent space. To this end, instead treating the generated structures as binary splitting decisions, we interpret them unnormalized probabilities of belonging to particular sub-cluster. Using Gumbel-top-k sampling (Kool et al., 2019), we could sample half of the locations for one sub-cluster and assign the remaining half the other. This increases the diversity of the generated structure.
2.2.2 CONTENT GENERATOR
Unlike VAR TTan et al. 2024). which models token relationships in an image-indes -ineee-iddeeendent manner, our method captures image-specific relationships that convey richer structural information. To train effectively, the model must be aware of the content's hierarchical structure. Without this, such as when unique tokens from different stages are flattened into a ID sequence as in VAR, the model cannot be trained effectively. We address this issue through progressive canvas refinement, where iteratively refine the canvas by producing more structured details in different visual granularity.
Specifically, define the canvas the latent we generated so far by accumulating all previous stages, 2; Σj= a(ci, 8:). Then the content generator is trained to directly generate the final
5

---

Published as a conference paper at ICLR 2026
2K- The difference between the final canvas (model output) and the current canvas (model input), fz(2,) - Z, simulates the quantization target for the current stage's 's content tokens C; during the VGS construction. An illustration of the relationship between the final canvas æ, current canvas Et and current content e. can be found at Figure 2 This strategy is similar to diffusion and flow models, but in contrast to denoising the image, we aim to refine the canvas iteratively.
canvas fz(
Inputs. The input canvas is represented as a flattened sequence, concatenating with the text condition. The text embedding and stage embedding are summed and input the norm layer.
Structure-Aware RoPE. To help the model understand the token structure, we extend RoPE (Su al. 2024) to encode the hierarchical structure. For the attention feature dimension of 64, we split it into: [8] for text/image identification, [2] x 8 for structure encoding, [20] x 2 for image spatial locations. In this structure-aware RoPE, for each stage, tokens within the same cluster are treated as being at the same structure position. Vice versa, if they are from different clusters, they are treated as different structure positions. the stages accumulate, the model knows the entire hierarchical structure of the tokens. The structure IDs are the structure embedding shown in Figure]
Content Prediction. We train the model to predict the final canvas æ. This approach unifies the training objectives across different stages and helps prevent overfitting by providing an informative supervision signal. Next, we compute the difference fe(24) az between the predicted final canvas and the current input canvas. We then average the features corresponding to each unique token at this stage to obtain the feature vector for each unique token. linear layer maps these token features to logits é, forming distribution over all possible token candidates. These logit vectors serve the same role as those in (visual) autoregressive models, supporting token sampling during generation.
Content Generator Training. We use MSE loss supervise the generation of the final canvas and cross-entropy loss to supervise the content prediction, as
f(x:) | fe( CE(C,cc))
(3)
We train the model with 10% null condition. We apply RePA (Yu et al. 2025c) at the 8-th layer.
EXPERIMENTS
We train NVG models of varying sizes for clas-coonitional image generation on the ImageNet dataset. We compare NVG with other models and provide further analysis of NVG. The training and sampling
details and reconstruction comparisons are given in App. A.5 A.6
and App. B.I respectively.
3.1 GENERATION RESULTS
Quantitative Comparison. We compare our NVG series with state-of-the-art image generation models Table The competing models are grouped into the following categories: GANs, diffusion models (Diff), masked regressveee models (Mask), standard auto-regressive models (AR), auto- regressive variants (X-AR), and VAR. We evaluate all models using standard metrics: FID, Inception Score, precision, and recall, calculated with OpenAl's evaluation tool introduced in Dhariwal & Nichol (02II. We also report each model's size, training steps, and generation steps.
Overall, our NVG models perform comparable or better than all competing methods, while requiring fewer training steps and using fewer parameters. NVG consistently outperforms VAR on FID, IS, and recall. These results highlight the strength of our framework. As scale up the NVG model, we see consistent improvements in FID and Inception Score, showing the strong potential of our approach.
Qualitative Visualization. We visualize the generated images Figure 5 (1) In the top rows, we display representative examples. The first unique token sets the overall color tone, while the first binary structure map defines the initial layout. As the generation progresses, the structure map becomes more detailed, guiding finer aspects of the layout. Meanwhile, the image itself is refined step-by-step, starting with object shapes, then object parts, and finally visual details. (2) Our clustering algorithm is based feature similarity, so the binary structure map often reflects rough separation between foreground and background. However, the generator can interpret this map flexibly. This is evident in the middle row: while the generator generally follows the structure, may merge separated regions to form one object (c.g., the chimpanzee's 's chest). It may also split a single region, such as the foreground, into multiple distinct objects (e.g., several ostriches). (3) The

---

Published as a conference paper at ICLR 2026
Table 1: Generation performance on class-conditional ImageNet 256 256.
Type
Model
FID(J)
IS(1)
Pre(t) Rec(t) #Para
#Train'
#Step
BigGAN (Brock et al.
2019
6.95
224.5
0.89
0.38
112M
GAN
GigaGAN Kang et
2023
3,45
225.5
0.84
0.61
569M
920K
StyleGan XL Sauer et al
2022
2.30
265.1
0.78
0.53
166M
CDM (Ho et
2022)
4.88
158.7
2.1M
8100
LDM-4-G Rombach et al.
2022
3.60
247.7
400M
178K
250
Diff
DIT-XL/2 (Peebles & X1e
2023
2.27
278.2
0.83
0.57
675M
7M
250
SiT-X et 2024)
2.06
270.3
0.82
0.59
675M
7M
250
MaskGIT Chang et al.
2022
6.18
182.1
0.80
0.51
227M
300c
8
RCG (cond.)
20231
3.49
215.5
502M
200c+800e
20
Mask
TTTok-S-128
Yu al.
2024b
1.97
281.8
287M
300e
MAGVIT-v2 Yu al
2024a
1.78
319.4
307M
1080e
64
MAR-H (Li et al. 2024
1.55
303.7
0.81
0.62
943M
800e
256
VQGAN (Esser ct al.
2021
15.78
74.3
1.4B
2.4M
256
RQTran. (Lc al. (2022)
7.55
134.0
3.8B
68
ViTVQ Yu
2022)
4.17
175.1
1.7B
450K
1024
LlamaGen-L. (Sun
2024
3.07
256.1
0.52
343M
300e
576
AR
LlamaGen-XXI. (Sun el al
2024
2.34
253.9
0.80
0.59
1.4B
300e
576
Open- Open-MAGVIT2-XX (Luo et
2024
2.33
271.8
0.84
0.54
1.5B
300e~3506
256
IBQ-XL (Shi et al.
2024)
2.14
279.0
0.83
0.56
1.1B
300e~450e
256
IBQ-XXL (Shi et al
2024)
2.05
286.7
2.1B
300e~
256
DART-FM (Gu al. 2025
3.82
263.8
820M
500K
16
SAR-XL (Liu al. 20244
2.76
273.8
0.84
893M
200e
256
RandAR-L (Pang el al. 2025)
2.55
288.8
0.81
0.58
343M
300e
88
X-AR
RandAR-XXI (Pang el al. 2025
2.15
322.0
0.79
0.62
1.4B
300e
88
D-AR-XL (ao & Shou 2026)
2.09
298.4
0.79
0.62
775M
300c
256
EAR-H (Shao al.
2025
1.97
289.6
0.81
0.59
937M
800c
64
CausalFusion XL (Deng et al.
2024)
1.77
282.3
0.82
0.61
676M
800c
250
VAR-d16 Tian
2024
3.30
274.4
0.84
0.51
310M
200e
VAR
VAR-d20 Tan et al.
2024
2.57
302.6
0.83
0.56
600M
250e
10
VAR-d24
Ian et al.
2024
2.09
312.9
0.82
0.59
1.0B
350e
10
NVG-d16 (255M+64M)
3.03
279.2
0.82
0.54
320M
200e
9
Ours
NVG-d20 (497M+125M)
2.44
310.4
0.80
0.60
622M
250c
9
NVG-d24 (856M+215M)
2.06
317.0
0.79
0.61
1.1B
350e
9
content
generation as one step per stage. Counting separately gives
steps
paper, we use n 25 and we leave generating structures with one-step flow models as future work.
This columm shows training iterations/epaccs. K: thousand iterations, M: million, e: epochs. With batch size 768, 100e cu 167K.
The training itcrations for NVG are 333K, 444K, and 519K, respectively.
bottom highlight the diversity and quality of our results. We provide qualitative comparison with other methods in App. B.3
3.2 FURTHER ANALYSIS
We provide ablation study in App. B.2 generation variation analysis in App. B.4 and complexity analysis in App. B.5
Grieterr-Guieed Generation. One key advantage our framework is we can use an explicit structure map to guide generation. We test our framework with very simple binary structure maps based on basic geometric shapes, such as circles placed in different positions
rectangle. The pructure-guided guided generation results at the bottom Figure 6 The model follows the given structure maps closely. Because the structure maps continuous, the generated background tends to be simple. The model fills in the foreground according to the provided class label, though some cases interprets the structure map differently but still in reasonable way. In practice, one could use segmentation maps semantic layouts to create structure maps, enabling more detailed and controlled generation. This demonstrates the benefit of framework: it supports flexible control right out of the box without additional training.
results are shown in Figure [ We also provide reference structure-
7

---

Published as a conference paper at ICLR 2026
Figure 5: Visualization of generated images. Top: We show several representative examples to illustrate the iterative generation process. Middle: The generated binary structure maps align well with the final images. Bottom: Our NVG-d24 model can generate diverse and high-quality images.
Extreme Cases Analysis. In this section, we examine how NVG behaves in several extreme cases. Representative examples are shown in Figure
(a) Unclear structures: In some situations, the generated structures appear unclear or less interpretable. NVG handles these cases as follows: (1) Rough object shapes: When realistic object boundaries do not align perfectly with the generated structures. NVG first produces coarse but plausible structure and then progressively refines into more accurate shapes (first row of Figure 7a). (2) Uninterpretable structures but coherent images: Sometimes the structures look ambiguous to humans, but the resulting images still show a clear spatial layout (second row of Figure 7a We argue that these structures
remain meaningful to the generator. (3) Extreme cases: When the object has color very similar to the background or is extremely small, the content generator first adjusts the overall layout and textures in the early stages. Once more flexibility is allowed in later stages, precise and realistic objects emerge and are quickly refined (last row Figure
(b) Multi-object cases: Images containing multiple objects particularly challenging. NVG typically handles them in three ways: (1) Small objects: When objects are very small, they are often merged into single large region. NVG then refines this region in the final stages produce clear
8

---

Published as a conference paper at ICLR 2026
Figure Visualization of structure-guit guidee generation results. Top: Each row shows generated images based the given geometric binary structure map. Bottom: Each group of three images includes one reference image and two generated images that follow its structure.
(a) Our NVG can still generate realistic images even when the structure maps rough (first row), unclear to humans (sccond row). in extreme imbalance (third row).
(b) Our NVG can generate multiple objects of various sizes like small ones (first row) as well as moderate ones (second row), and those with structures that unclear humans (third row).
Figure 7: NVG can handle several extreme cases where equal-size, value-based clustering or the generated structures are not ideal.
object boundaries (see the first row of Figure 7b moderate size, their structural patterns emerge early and are refined in the middle stages. This leads to clearer object separation, shown in both the structures and images in the second row of Figure 7b
(2) Moderate-sizze objects: When objects have

---

Published as a conference paper at ICLR 2026
(3) Unclear structure maps: In extreme cases where the structure map is unclear to humans, NVG still captures the overall scene layout, as seen in the early refined images. As refinement continues, the model gradually separates individual objects and enhances textures (last row of Figure 7b).
Overall, NVG shows strong robustness handling complex scenes and imperfect structures.
RELATED WORK
The underlying modeling of current image generation models can be grouped into holistic, fragmented, and cascaded image modeling processes. We provide a discussion of these methods in App. C existing approaches lack explicit modeling of the structural composition inherent in images. We aim to address this gap by prompting
Most
as a fundamental aspect in image generation.
CONCLUSION AND DISCUSSIONS
This work advances image generation by explicitly modeling hierarchical visual structure, addressing a key limitation of existing approaches that often treat images as flat, unstructured data. By decompos- ing images into sequences of increasing granularity and guiding generation through structure-awan mechanisms, our framework not only improves fidelity but also opens new possibilities for structure- controlled generation. The strong empirical performance demonstrates that NVG framework can scale effectively with model size, suggesting a promising path toward more controllable generative systems. Beyond the benchmarks, the ability separate and iteratively refine structure and content offers practical advantages in domains such as design, scientific visualization, and any scenario where structure and hierarchy in the generation process are essential.
Limitations and Future Work. In this paper, we focus on verifying the feasibility of our generation framework and developing an effective learning paradigm. We also provide fair comparisons with competitors to better demonstrate the capability of our NVG models. Our study centers on standard class-conditional image generation using the ImageNet benchmark dataset. For future work, there are several exciting directions to explore in visual granularity based generation:
Better Structure Design: In this paper, we derive the structure directly from data. (1) Depending on the task, the structure map can be designed to include expert knowledge about the image, such as multi-level segmentation, frequency decomposition, or resolution scales. (2) When the structure is learned from data, more advanced algorithmss can be used to split the image into more interpretable regions, such as more accurate clustering methods or flexible clustering approaches that explicitly handle imbalanced regions in the image. We believe all these variants can be effectively trained within our proposed framework.
Region-Aware Generation: Our approach enables direct generation using visual granularity sequences defined by domain-specific annotations. These controls are not just conditions. They define the structure the model follows. Our multi-stage setup also allows fine-grained control specific regions and generation stages.
Physical-Awadd Aware Video Generation: Structured image regions can be tracked over time, enabling video generation that is more coherent and physically realistic. By tracking the evolution of each region over time, we can enforce structural constraints and physical laws during generation, rather than applying post-hoc supervision to the generated frames.
Hierarchical Spatial Reasoning: The pioneering work Spatial Reasoning Models (Wewer et al. 2025 explores visual spatial reasoning using patch-wise diffusion process with predicted patch generation order such that the reasoning enabled when generating unobserved patches conditioned on observed patches. Our method can also be used to generate structured global-to-local eyvide-and-conque reasoning chain for spatial reasoning.
Reproducibility Statement. We have made every effort to ensure work is reproducible. The main paper provides a complete description of the framework and pipeline, while the appendix includes detailed information the implementation, architecture, and optimization. The full training and inference source code and models are available in littts::. ://::////iiiii/ -ang.. qit github. io/ nvg
10

---

Published as a conference paper at ICLR 2026
Acknowledgments. This study is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). It is also partially supported by NVIDIA Academic Grant. We thank Hanze Dong, Ziang Zhou, and Junqiu Yu for thoughtful discussions.
REFERENCES
Roman Bachmann, Jesse Allardice, David Mizrahi, Enrico Fini, Oğuzhan Fatih Kar, Elmira Amirloo, Alaaeldin El-Nouby, Amir Zamir, and Afshin Dehghan. Flextok: Resampling images into 1d token sequences of flexible length. In ICML, 2025.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. JMLR, 2003.
Black- Baack-Forstt-Lab.. Flux.
lateesstlabb//o flux 2024.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. NeurlPS, 2020.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In ICML, 2023.
Chaorui Deng, Deyao Zhu, Kunchang Li, Shi Guang, and Haoqi Fan. Causal diffusion transformers for generative modeling. arXiv preprint 2rXiv:24/2.12099 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In /CML, 2024.
Evelyn Fix and J. L. Hodges. Discriminatory analysis. nonparametric discrimination: Consistency properties. International Statistical Review, 1989.
Ziteng Gao and Mike Zheng Shou. D-AR: Diffusion via autoregressive models. In ICLR, 2026.
Ian J Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farlcy, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NewrIPS, 2014.
Dorothy Greig, Bruce T Porteous, and Allan Scheult. Exact maximum a posteriori estimation for binary images. JRSS: B, 1989.
Jiatao Gu, Yuyang Wang. Yizhe Zhang, Qihang Zhang, Dinghuai Zhang, Navdeep Jaitly, Joshua M. Susskind, and Shuangfei Zhai. Denoising autoregressive transformers for scalable text-to-image generation. In ICLR, 2025.
11

---

Published as a conference paper at ICLR 2026
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurlPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. JMLR, 2022.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang. Yuxiang Huang, et al. Miniepm: Unveiling the potential of small language models with scalable training strategies. COLM, 2024.
Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In CVPR, 2023.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In NeurlPS, 2022.
Wouter Kool, Herke Van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. In ICML, 2019.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 1955.
Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR, 2022.
Tianhong Li, Dina Katabi, and Kaiming He. Self-conditioned image generation via generating representations. CoRR, 2023.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. In NeurlPS, 2024.
Tianhong Li, Qinyi Sun, Lijie Fan, and Kaiming He. Fractal generative models. 7MLR, 2025.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR, 2023.
Wenze Liu, Le Zhuo, Yi Xin, Sheng Xia, Peng Gao, and Xiangyu Yue. Customize your visual autoregressive recipe with set autoregressive modeling. arXiv preprint arXiv:2410. 105/1, 2024.
Xingchao Liu, Chengyue Gong, et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR, 2023.
Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing auto-regressive visual generation. arXiv preprint arXiv:2409.044100 044400 2024.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Hunden-Eijnden and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In ECCV, 2024.
Ziqi Pang, Tianyuan Zhang, Fujun Luan, Yunze Man, Hao Tan, Kai Zhang, William T Freeman, and Yu-Xiong Wang. Randar: Decoder-only autoregressive visual generation in random orders. In CVPR, 2025.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In ICLR, 2016.
12

---

Published as a conference paper at ICLR 2026
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In NeurlPS, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In SIGGRAPH, 2022.
Chenze Shao, Fandong Meng, and Jie Zhou. Continuous visual autoregressive generation via score maximization. In ICML, 2025.
Fengyuan Shi, Zhuoyan Luo, Yixiao Ge, Yujiu Yang, Ying Shan, and Limin Wang. Taming scalable visual tokenizer for autoregressive image generation. arXiv preprint arXiv:2412.02692, 2024.
K Simonyan and A Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR, 2021.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. 2024.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 065i:22400..65555, 2024.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. NeurIPS, 2024.
Christopher Wewer, Bartlomiej Pogodzinski, Bernt Schiele, and Jan Eric Lenssen. Spatial reasoning with denoising models. In ICMI., 2025.
Hu Yu, Hao Luo, Hangjie Yuan, Yu Rong, and Feng Zhao Frequency autoregressive image generation with continuous tokens. arXiv preprint arXiv:2503.05300 2025a.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. In ICLR, 2022.
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu. Alexander Hauptmann, et al. Language model beats diffusion-tokenizerr is key to visual generation. In ICLR, 2024a.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang- -Chihh Chen. image is worth 32 tokens for reconstruction and generation. In NeurlPS, 2024b.
Qihang Yu. Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chich Chen. Randomized autoregressive visual generation. In ICCV, 2025b.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. ICLR, 2025c.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In ICLR, 2023a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023b.
13

---

Published as a conference paper at ICLR 2026
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In CVPR, 2018.
Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vq-gan to 100,000 with utilization rate of 99%. In NeurlPS, 2024.
A METHOD
A.1 ALGORITHM
We present our construction and generation algorithms in Algorithm 1 and Algorithm
respectively.
Algorithm 1: VGS Construction
Algorithm 2: VGS Generation
Inputs: Encoded latent Z. K 8, token down
Inputs: Class condition Exx,, Es ((O for = 0, 0....
factor latent size I. 16 16;
Initialize: a: 0, 8g [1]4*K; do
Initialize: 8x range( L), Co from Eq.
for K 1,... do
if < < K then
Zu, 8. Cluster Zk+1, Sk-1,
8, +- ftatt, æ, 8e, S,, k);
for = 1,... do
# No need to predict 8K-1 + aK
Reorganize #y:
Get st from Eq. 7
Calculate Rg from Eq.
fe(tat, S.. k):
Calculate C from Eq.
x + 5: (zx)
Return: VGS {c,, 8,
10 Return: T {ca, 81 1 or
A.2 CONTENT CONSTRUCTION ALGGORITH
In stage-O, we initialize the quantization target Rg as the latent Z and quantize it to obtain content token Co as:
(Avg R2, 8:)), where Q(z) = miny, IV z112. Avg is an average operator applied within the locations the same cluster ID. After getting the tokens of each stage, we calculate the quantization error via
(4)
R. Z Σ R oa(c,)
(5)
where a(-) is an assign operator that assigns the unique tokens the original latent space following
the structure map s;, and o; is a single conv refiner layer as in VAR (Tian et al. 2024 The R; serves as the target for quantization in the next stage.
A.3 STRUCTURE EMBEDDING CONSTRUCTION
Structure maps play crucial role in our generation framework they determine the layout of details at different granularity levels. To this end, We propose compact, hierarchical representation encode the multi-stage label maps { at each spatial location. The structure embedding is designed to satisfy the following criteria: (1) Parent-child relationships across stages must be preserved; (2) Embeddings from different stages should remain distinguishable within unified embedding space; (3) The ordering of cluster IDs should not be incorporated in the embedding, as they serve only distinguish between clusters. practice, we also expect the implementation the mapping from cluster ID and stage ID to the embedding is simple and efficient.
We define the structure embedding as bit-style vector in the most-to-least significant bit order, with bit values 0 and 2 and padding value 1 as midpoint. An example of this embedding process is shown in Figure
We use -ddimensional vector to encode the structure across all stages. At stage 0, since all locations belong to single cluster, pad the vector with all 1s. For stages > 0, the embedding is inherited from the parent and extended with one additional bit (either 0 or 2) to distinguish between child labels. All values non non-negative integers for easy integration with RoPE IDs (Su et al. 2024). We apply this structure embedding in both the content and structure generators in our framework.
14

---

Published as a conference paper at ICLR 2026
During construction, reindex the structure map sequence such that class j in stage becomes classes 2j and 2j + 1 in stage i + 1. With this implementation, we can derive the embedding by converting the structure label into bits using the formula:
(s, 1), 2([s/2'-1~ mod 2)1j<r 1,26 vo S K,v K, Vs [0,2").
(6)
The original structure class ID can be directly recovered from the structure embedding via
Σ sebix x
Our structure embedding strictly follows the intended design criteria. Specifically; (1) Each stage is
represented by one channel. Child embeddings are derived from their parent embeddings by adding new bit to distinguish between child labels, effectively preserving the parent-child relationship. (2) The number of padding channels directly indicates the stage of the structure embedding, making the
hierarchical structure clearly visible across channels. (3) The bit-style embedding clearly separates different clusters within unified space, without relying the order of cluster IDs.
A.4 ARCHITECTURE
Content Generator We follow FLUX Forest-Labss 2024) to use the self-attention block with parallel linear layers (Dehghani al. (2023) as our basic building block. We follow VAR for network configuration, setting network width w, attention heads / and dropout rate dr with model depth as:
64d, h= d, dr 0.1 d/24.
(8)
The major number of parameters in the backbone grows with model depth according to:
(d) = d
+
5w
3w = 15dw 61, 440d
(9)
okv mip_
proj _uuut
modulation
This is 5/6 the size of VAR since we adopt simpler modulation design inspired by FLUX.
Structure Generator Since the structure embeddings have lower dimension (8 v.s. 32) and fewer unique tokens (up to 256 v.s. 4096) than the content embeddings, learning the structure is comparatively easier. Therefore, we reduce the model width to 1/2 that of the content generator while keeping other architectural aspects the same, leading to
w 32d, h = d/2, dr = 0.1 d/24, ((d) = 15du = 15, 360d
(10)
This results in a small model that is 1/4 the size of the content generator. When combined, our full generator has only about 1/24 more parameters than VAR. We also adopt the structure-aware RoPE in the structure generator, where the first part is extended to text/imag//structure identification.
A.5 TRAINING SETUP
We train the auto-encoder using the same loss functions as Open- MANn-22 Luo et al. 2024). but replace the original discriminator with DINO (Zhang al. 2023a). following the setup used in VAR. To improve codebook initialization, we apply IBQ (Shi et al. (024) during the first epoch as a warm-up. We set the base channel size to 128 instead of 160 which used in VAR. The codebook has a size of 4096 x 32. We train the autoencoder for 100 epochs with batch size of 256 and learning rate of 1e 4 with cosine decay
The generator is trained with base learning rate 1e 4 and a batch size of 256, scaled linearly based on the actual batch size. We use the WSD learning rate schedule (Hu et al., 2024): We linearly increase the learning rate during the first 1,000 steps as a warmup, then keep constant for most of the training, and finally decrease it linearly to 1/10 of the original rate by the end of training. We train NVG-d16 for 200 epochs, NVG-d20 for 250 epochs, and NVG-d24 for 350 epochs to make fair comparison with VAR. For NVG-d16 and NVG-d20, the learning rate starts to decay after 80% of the training epochs. For NVG-d24, we found that the model's 's behavior stabilizes between 120 and 200 epochs, so we begin decaying the learning rate at epoch 200 to avoid wasting more computing resources. We train our models on ImageNet (Deng al. 2009 The generation step for structure generator set as 25.
15

---

Published as a conference paper at ICLR 2026
Table 2: Reconstruction performance on ImageNet validation dataset. Results of VAR are reproduced, while other competitors are from IBQ. LPIPS are calculated via VGG (Sioonyan & Zisserman 2015
Tokenizer
#Tokens
Ratio
Codebook
iFID())
LPIPS(()
Usage (1)
VQ-GAN (Esser
2021
16
16
1,024
7.94
44%
SG-VQGAI Rombach ct a 2022
16
16
16
16,384
5.15
VQGAN- LOGAN-LCAN (Zhu et al.
2024
16
16
16
100, 000
2.62
0.2212
99%
MaskGIT CCang
2022
16 16
16
1,024
2.28
LlamaGen (Sun
124
16 16
16
16,384
2.19
0.2281
97%
Open-MAGVII22 Lio ct
2024)
16 16
16
262,144
1.17
0.2038
100%
IBQ (Shi et al. 2024)
16 x 16
16
262,144
1.00
0,2030
84%
VAR Lian el al. 2024)
680
16
4,096
1.06
0.1863
100%
NVG
511"
16
4,096
0.74
0.1875
100%
deline
number unique consider number
comparison.
uses 256 tokens, showing that our tokenizer achieves quantizarion tokens.
Table 3: Ablation NVG-d12. Models trained for 40 epochs, tested with constant CFG scale of 1.5 for both content and structure generators without setting TOP _PPKoon or TOP P.
Content Input
Structure Input
Structure RoPE
FID(L)
IS(t)
Precision(1)
Recall(*)
Pure Noise
38.94
44.0
0.48
0.53
27
Pure Noise
37.89
44.6
0.50
0.53
Partial Noise
X
39.03
43.5
0.49
0.52
Partial Noise
37.59
45.1
0.50
0.53
A.6 SAMPLING METHOD
For content generation, approaches to sample the tokens to improve diversity. This sampling approach is effective in the early stages, where the model focuses on the overall contents of the image. However, because we use residual learning approach, the model is trained to predict the difference between the current canvas and the final canvas, i.e. fix the error. Hence in later stages, the process becomes less about sampling and more about accurately correcting errors. Inspired by this, we propose control the available candidates by reducing the TOP P parameter from 100% (the full codebook) down to 50% (the most confident candidates) in logarithmic rate. This strikes balance between promoting diversity early on and ensuring high fidelity the later stages. We follow previous methods by linearly increasing the CFG scale: from 1 to 2.5 during structure generation, and from 1 to 3.5 during content generation. This helps strengthen guidance in the later stages of generation.
learning the token distribution, might follow autoregressive
B EXPERIMENT
B. RECONSTRUCTION COMPARISON
We compare our tokenizer with state-of-the-art methods in Table 2 Our tokenizer achieves better rFID (Heusel et al.) 2017) and comparable LPIPS (Zhang al. 2018 scores, while using much smaller codebook and maintaining a high utilization rate. Our tokenizer significantly outperforms VAR's in reconstruction quality with less number of unique tokens, showing the advantage of our granularity- -based approach over scale-based tokenization. Furthermore, in the first stage, VAR's codebook utilization rate is 25.39%, while ours is 68.55% indicating more balanced codebook.
B.2 ABLATION STUDY
We conduct ablation studies NVG-d12. The results summarized in Table 3
(a) Content Inputs: We evaluate three types of content inputs: (1) Current Canvas (a:): Similar to
autoregressive modeling, where the current canvas is used directly; (2) Noised Canvas (x + te): Adds Gaussian noise, following the approach of EDM (Karras et al., 2022), trying to incorporate more randomness during generation; (3)
Noised Canvas ((1 -)x + (e): A
16

---

Published as a conference paper at ICLR 2026
linear interpolation between the canvas and noise, inspired by rectified flow models (Liu et al. 2023 Lipman et al. ).23). The noise level is linearly scheduled as t = 1 (i - 1)/9 for stage i.
During training, the preservvng version performed significantly worse in predicting tokens compared to other approaches. As a result, it is considered ineffective, and we terminate its training. These results suggest that using an autoregressive approach is more effective for generating the VGS, indicating that treating content generation as conditional modeling is better.
(b) Structure Inputs: We test two variants: (1) Pure Noise: Uses standard Gaussian noise for the entire structure input. (2) Partial Noise: Replaces known parts of the Gaussian noise with ground-truth values for known stages, mimicking structure inpainting task. The results show that preserving known stages leads to better performance.
(c) Stracture-Aware RoPE: : We compare models with and without structure-awar RoPE to evaluate its impact. When only spatial-aware RoPE is used without structure-aware RoPE, the model struggles to capture structural relationships between tokens, resulting in weaker generation quality.
(d) Content Prediction: We also tried directly predicting the next content without predicting the final canvas. However, this method starts to overfit after about 25 epochs and is therefore considered ineffective and we terminate its training. This further highlights the advantage of providing richer supervision through the final canvas.
B.3 QUALITATIVE COMPARISON
In this section, show random examples generated by our method and several of-te-of-the-art baselines, including diffusion models (SiT-X et. (024)), autoregressive models [IQ-XXL (Shi) al. 2024h), and VAR (Tian al. (024), shown Figure and Figure [0] These results show that N VG achieves improved or comparable image quality and diversity compared with these SOTA methods.
B.4 WISE GENERATION VARIATION
We analyze generation variation in different stages by fixing the structure and content from earlier stages and generate the remaining stages. The results are shown Figure
(a) In-domain controlled generation: In the left column, we guide the generation using the in-domain class "standard poodle". The provided structure and content align with this class label.
(1) The first unique token has a strong influence on the main semantics of the image, such as the presence of a dog and a grassy background. The first binary structure map determines the overall layout. When only the structure map is fixed, the generated images show dogs with similar poses but different colors and backgrounds. When the first unique token is fixed, the overall semantic content and color tones stay consistent across generations. This doesn't 't mean the first token directly specifies colors or specific objects, but it provides strong prior that guides the generator towards that direction. We can confirm this by using different class guidance in the right column. When we change the class condition, the same fixed content and structure can lead to noticeably different outputs, as the model "interprets" them differently. A similar effect is seen with changes the structure map.
(2) As more stages are fixed, the variation in the generated images decreases. In stages 2-3, layout details are fixed. This reduces variation in the dog's pose and how the background is arranged. These constraints come from both the fixed content and structure. If only the structure is fixed, the generator still creates multiple plausible variations that match the structure. In stages 4-5, detailed aspects like the dog's log's head orientation become fixed. Content tokens clearly encode information such as the dog facing forward. If only structure is fixed, the interpretation of the generator varies, but the results remain consistent and will not change when fixing more stages. The final stages refine smaller details, such as the texture of the dog's 's fur in the face or the color of flowers the background. These results show clear trend: each stage controls a different level of visual information, from general layout and semantics to fine- grained appearance details.
(b) Out-of domain controlled generation: In the right column, guide the generation process using distinct class labels such as "Siamese cat" and "Indian elephant". These results demonstrate the model's ability to produce diverse outputs and correct emror..
17

---

Published as a conference paper at ICLR 2026
2
7
Generate 1-8
Generate 3-8
Generate 5-8
Generate 7-8
Generate 1-8
Generate 3-8
Generate 5-8
Generate 7-8
Figure 8: Visualization of stage-wise controlled generation results. Top row shows how the reference image is reconstructed at each stage. Middle row shows generated images where both the content and structure from previous stages are preserved. Bottom row shows generated images where only structure from previous stages is preserved. The left column uses the class "standard poodle" as guidance, which closely resembles the reference image. The right column uses the classes "Siamese cat" and "Indian elephant" as guidance, representing -iitt--ddittiuttiion (OOD) examples. In the case of "Generate i-8", the structure of i-stage provided by the reference image.
(1) When both content layout are fixed the early stages, the overall color and layouts are also fixed. However, if the target class differs from the reference image, the model struggles to maintain layout control when only the first stage fixed. This suggests that different classes tend to follow different structural patterns. The generator tends to interpret the layout based what it has learned during training. For example, cats often appear in indoor scenes. However, we fix the first unique token, the generated image changes
outdoor scene with grass in the background.
(2) The structure maps can be interpreted in multiple valid ways. Even when the structure map comes from single image, the generator can still produce diverse results. But once the content is fixed
18

---

Published as a conference paper at ICLR 2026
Table Comparsion of inference cost.
Model
#Para
Time Cost (s)
Memory Cost (MiB)
SiT-X
675M
1.403600
4099
IBQ-XL
1.1B
5.848799
5563
VAR-d16 (w/ KV cache)
310M
0.115876
2521
NVG-d16 (w/o structure generation)
255M
0.073220
1563
NVG-d16
320M
0.723339
1687
VAR-d20 (w/ KV cache)
600M
0.137775
3817
NVG-d20 (w/o structure generation)
497M
0.084362
2253
NVG-d20
622M
0.841313
2495
VAR-d24 (w/ KV cache)
1.0B
0.159784
5539
NVG-d24 (w/o structure generation)
856M
0.112932
3001
NVG-d24
1.1B
1.122031
3471
along with the structure, this variety disappears. Interestingly, even when the first three stages of dog image are fixed, our model can still generate an image of different class. This shows that our framework has strong error-correction ability. This is major advantage over autoregressive models, which cannot revise what they have already generated.
B.5 COMPLEXITY ANALYSIS
In this section, we analyze the time and memory usage of NVG and compare it with state-of-the-art generation models in Table 4 All models are evaluated under the same hardware, software, data type, acceleration techniques, and CFG settings. The VAR series is further accelerated using KV cache, following their official implementation. The reported runtime and memory usage cover the full generation process in the latent space well as the final decoding stage. Because NVG generates structure maps with unique tokens while VAR can be regarded as using fixed structure map, we also include variant of NVG that uses fixed structure map for a more fine-grained comparison.
(1) Content generation. Thanks to our efficient generation sequence (9 steps v.s. in VAR) and improved architecture, NVG achieves noticeably faster runtime than VAR. VAR consumes more memory mainly due to the KV cache uses for acceleration, while NVG does not rely on this mechanism.
(2) Structure generation. Introducing the structure generator, currently implemented with a flow- matching model, increases the runtime. Because the structure model is small, the additional memory overhead remains limited.
(3) Overall comparison. Although structure generation makes method slower than the VAR series, is still much faster than standard diffusion models (SiT-X) and autoregressive models (IBQ-XI).. Notably, method also much less memory than current state-of-the-ar models. Speeding up the structure generation step is an important direction for future work, and we believe it is achievable given the recent rapid progress in few-step flow-matching models.
C RELATED WORKS
Holistic Image Modeling. These models treat the image as unified high-dimensional distri- bution in pixel/latent space, aiming to generate the entire image simultaneously. To approximate this distribution, models typically start from a simple prior, such as a Gaussian distribution, and transform it into the image distribution. Prominent modeling approaches include generative ad- versarial networks (Goodfellow al.) 2014), diffusion models (Ho al.) 2020). and flow-based methods (Liu et al.] [023) Lipman et al. 2023). GANs generally perform one-step image generation, whereas diffusion models and flow-based methods often rely on multi-step generation processes. In particular, diffusion and flow-based models are by design require hundreds or thousands of gen- eration steps. They reduce the number generation steps during inference via specific sampling algorithms (Song al. (021). These models are known for producing high-fidelity high-quality generation results (Kang et al. 2223) Rombach et al.) 2022 Esser et al. 2024) and can be conditioned auxiliary inputs through extra module (Zhang al. 2023b) light-weight fine-tuning (Hu et al.
19

---

Published as a conference paper at ICLR 2026
2022). However, the holistic nature of the generation process introduces challenges for precise control. They usually need to train additional components to control the generation process.
Fragmented Image Modeling. These models conceptualize image set of discrete, often non-overlapping patches, analogous to a 2D sentence visual words. This approach is inspired by autoregressive modeling in natural language processing (Bengio et al. 2003 Brown al. 2020).
The model learns to predict the distribution of unknown patches conditioned on observed ones. The generation order can follow raster-scan pattern (autoregressive) (Chen et al. 2020 Esser et al. 202 or a model-defined sequential rule mron-autoregressivv) (Chang et al. [2022] Pang et al. 2025 Yu al. 2025b). The conceptual alignment with modeling facilitates integration into multi multi-modal architectures. These models also need hundreds or even thousands of steps to produce results. To
speed up, non-autoregressive models often generate multiple tokens in one step, while autoregressive models mainly rely on engineering techniques like the KV-cache.
Cascaded Image Modeling. These models adopt hierarchical, coarse-to- generation strat- egy. Typically, low-resolution image is generated first, followed by subsequent upsampling and refinement to achieve high-resolution output (Razavi et al., 2019 Ho al.) 2022). Recent work starts decomposing images at the same spatial resolution. The pioneer work visual autoregressive models (Tian et al.
2024) uses spatial compression to control the density visual information in different stages. Other kinds of decompositions like the number of tokens (Bachmann al.
2025 Gao & Shou 2026) or frequency filter YYu et al. 2025a) have also been explored. Li et al. (2025) explores generating image in divide-and-conquee way. Cascaded models benefit from faster generation due to their few-step design, making them generally more efficient than diffusion or autoregressive models without further exploration on sampling algorithms engineering techniques.
D STATEMENTS
Use of Large Language Models. We follow ICLR's policy and LLMs only general-purpose assistive tools. In our work, they were used to polish the writing in some paragraphs and to suggest improvements for code. They were not involved in research ideation, experimental design, or the development of core technical contributions.
20

---

Published as a conference paper at ICLR 2026
SIT-X FID: 2.06, IS: 270.3
IBQ-XXL FID: 2.05, IS: 286.7
VAR-d24 FID: 2.09, 312.9
NVG-d24
FID: 2.06. IS: 317.0
Figure 9: ualitative comparison with state-of-the-art diffusion (SIT-X Ma et al. 2024)), auto- regressive (IBQ-XXL (Shi al. 2024 and VAR (Tian et al. 2024) models.
21

---

Published as a conference paper at ICLR 2026
SIT-X FID: 2.06, IS: 270.3
IBQ-XXL FID: 2.05, IS: 286.7
VAR-d24 FID: 2.09. 312.9
NVG-d24 FID: 2.06. IS: 317.0
Figure 10: More qualitative comparison with state-of-the-art diffusion (SiT-X (a et al. 2024 auto-regressive (IBQ-XXL (Shi et al. 2024)) and VAR (Tian al. 2024) models.
22


Reviews:
[{"review_id": "MlVPRqxMav", "reviewer": "Reviewer_enua", "summary": "This work introduces content and structure pairs to assist the generation task. In particular, the structure guidance is from the features, without another pretrained network. The structure guidance can improve the generation performance.", "questions": "Please see the weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work explores the structural information for autoregressive-based image generation. It generates a structural mask by a cluster-based algorithm, without any pretrained method. The experimental results verify its effectiveness.", "weaknesses": "- The structured conditional guidance appears to have limited novelty, as similar ideas have been widely explored in text-to-image generation, such as in ControlNet [A].\n- There is no ablation study evaluating the effectiveness of the structure and content guidance. In addition, qualitative comparisons with state-of-the-art methods are missing, as are qualitative results for the ablation studies.\n\n[A] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. \"Adding conditional control to text-to-image diffusion models.\" ICCV 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "98kcAL7fCn", "reviewer": "Reviewer_bnRk", "summary": "This work proposes a new scheme for image generation, namely next visual granularity generation (NVG). In each granularity, the image’s latent is represented by the different binarized tokens. Each digit of the binarized token represents the grouping information of the pixel under each granularity levels. During the generation, the model start with the most coarse granularity level, where all pixels are separated into two groups. After each prediction stage, the number of the groups is doubled. This process is repeated until that all pixels have a distinct group. The model is trained in two parts. One model predicts the structure and the other predicts the content. In each prediction stage, the model first predicts structure and then predicts the content of each pixel.", "questions": "Please see my weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The reviewer likes the general idea of generating image from coarse-to-fine structure and content.\n* The model shows its scalability, indicating it can be a potential model for foundational generative models.\n* By leveraging the structure from reference image, the proposed method can conduct guided generation from reference image.", "weaknesses": "My major concern are related to some detail in the method and figures, the current manuscript makes the reviewer being confused.\n* From the Fig 2, it seems that the models (structure and content) are predicting the residual results (that will be aggregated to final result). However, in Fig 4, the output of the structure/content model are final structure/canvas (and ln 269 also mentioned that content generator directly generated final canvas). And the “Residual” is the difference between the final canvas and the input. This is quite confusing, why two models outputs final structure and canvas, and why the residual is computed after the final canvas is obtained. In addition, there are different colored lines used in the Fig 4, the reviewer would like author to provide clear instructions on what does these colors mean.\n* From the ln 239, it seems the model is conducting a flow-matching-like prediction. the reviewer has two questions here:\n    * why does the author adopt this type of input?\n    * is this strategy applied on each stage? If it is, does it mean it takes a flow-matching-like prediction for each step?\n* The reviewer is quite not understand the ln 263-265, “To train effectively, …, the model cannot be trained effectively”. What is the thing that hampers the model training?\n* The reviewer does not find the candidate pools of the content tokens. Since ln 289 mentions that “forming a distribution over all possible token candidates”, the reviewer feels like there is a candidate pool like text generation. However, the image generation is usually a continuous task and the tokens are usually continuous. I wonder that are the tokens here. Also, in the ln 269-270, the author mentioned the x-f(x) is the quantized target. I wonder what quantization there is and why use the difference as target.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yNo8qVYkPC", "reviewer": "Reviewer_aGuw", "summary": "The paper introduces Next Visual Granularity (NVG), a generative framework that models image synthesis as a coarse-to-fine process guided by hierarchical structure maps. It constructs these maps through a greedy clustering of token embeddings, progressively merging similar regions to capture visual granularity from details to global structure. NVG alternates between structure generation and content generation at each stage, offering interpretable and controllable image synthesis. Experiments on ImageNet show competitive or superior performance to VAR across FID, IS, and recall, demonstrating improved structure awareness and scalability.", "questions": "1.\tSince NVG heavily relies on the accuracy of the structure maps at each stage, would errors in early-stage structure generation propagate and amplify through subsequent stages, leading to degraded image quality?\n2.\tIn cases involving multiple objects within a scene, how effectively can the structure map distinguish between different foreground objects? Does the current clustering-based construction capture such object-level boundaries?\n3.\tAlthough Table 1 reports a similar number of generation steps as VAR, NVG introduces separate structure and content generation at each stage. Does the observed performance gain come at the cost of increased computational load or slower inference speed in practice?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed structure prediction follows an intuitive and interpretable coarse-to-fine process, aligning well with how humans perceive visual composition.\n2.\tThe framework effectively separates structure and content generation, enabling controllable and interpretable image synthesis.\n3.\tThe model demonstrates strong scalability and competitive performance across metrics such as FID and Inception Score compared to state-of-the-art baselines.", "weaknesses": "1.\tSemantic Misalignment in Clustering\nNVG constructs hierarchical structures via greedy clustering based on token embedding similarity.\nHowever, embedding-space similarity does not always reflect semantic consistency; tokens with similar color or texture may be grouped together even if they belong to different objects, leading to inaccurate granularity segmentation.\n2.\tIncreased Computational Cost\nCompared with VAR, NVG requires two sequential generation steps, structure and content at each stage.\nThis design improves interpretability but likely increases inference time and computational complexity, offsetting VAR’s efficiency advantage over traditional AR and diffusion models. It would be helpful if the paper reported the actual inference-time difference between NVG and VAR at a similar number of generation steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qLlOdHSxcM", "reviewer": "Reviewer_82ro", "summary": "This paper proposes Next Visual Granularity Generation (NVG), a hierarchical autoregressive image generation paradigm that progressively increases the token number (the “visual granularity”) with spatial resolution fixed in multiple stages. Specifically, during the inference process of each stage, a structure map (granularity structure information) is predicted with a diffusion model and then AR model is leveraged to transfer the stage-specific image to a fake final image with this structure, producing new stage residual content. On class-conditional ImageNet, NVG shows consistent but modest FID/IS improvements over VAR-style next-resolution autoregression and presents appealing spatial controllability via structure map guidance.", "questions": "1. Sensitivity about stage number. How sensitive are results to the number of granularity stages? \n\n2. Fake Final Canvas Prediction. Why does the model not produce the residual directly (like a diffusion process that predicts denoising noise)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel Concept. Using the diversity token and explicit structure map to decouple and capture the different-level visual granularity, achieving good explainational exploration and controllable generation for AR. \n\n2. Good Methodological Techniques. The integration of diffusion model and AR with structure-aware positional embedding for explicit structure map generation and granularity visual content producing is neat and well-motivated. \n\n3. Empirical Experiments. On ImageNet class-conditional generation, NVG exhibits clear scaling trends with model size. Visualization of the structure-map controllability demonstrates a practical interface for layout control.", "weaknesses": "1. Structure construction assumptions. In this paper, the structure map plays a core role during the generation process. Since the structure map is constructed with a hierarchical clustering algorithm as ground-truth labels, such construction method may misalign with semantic boundaries in some situations (e.g., crowded/occluded object scenes) and then leads to failure generation. Robustness to such cases is unclear. Additionally, how about the generation results with a noisy or bad structure map during inference? \n\n\n2. Limited improvement magnitude. Gains over strong VAR baselines are small (e.g., FID reductions on the order of ~0.2–0.3 absolute). The authors may further explore the advantages of structure-map controllability for other applications. \n\n3. System metrics are underreported. The two-stage pipeline (structure and content generation) and multi-stage decoding likely incur additional costs, but this paper lacks memory footprints and time cost curves.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)