NFIG: Multi-Scale Autoregressive Image Generation via Frequency Ordering
Authors: Zhihao Huang
Xi Qiu
Yukuo Ma
Yifu Zhou
Junjie Chen
Hongyuan Zhang
Chi Zhang
Xuelong Li
Venue: NeurIPS 2025 poster ()
Keywords: Image Generation
Auto-regressive model
Frequency analysis
Primary area: deep_learning
TL;DR: Images inherit a hierarchical frequency structure from the physics of light (light fields decompose by frequency, properties are frequency-dependent). This principle underpins our proposed low-to-high frequency image generation approach.

Abstract: Autoregressive models have achieved significant success in image generation. However, unlike the inherent hierarchical structure of image information in the spectral domain, standard autoregressive methods typically generate pixels sequentially in a fixed spatial order. To better leverage this spectral hierarchy, we introduce Next-Frequency Image Generation (NFIG). NFIG is a novel framework that decomposes the image generation process into multiple frequency-guided stages. NFIG aligns the generation process with the natural image structure. It does this by first generating low-frequency components, which efficiently capture global structure with significantly fewer tokens, and then progressively adding higher-frequency details. This frequency-aware paradigm offers substantial advantages: it not only improves the quality of generated images but crucially reduces inference cost by efficiently establishing global structure early on. Extensive experiments on the ImageNet-256 benchmark validate NFIG's effectiveness, demonstrating superior performance (FID: 2.81) and a notable 1.25x speedup compared to the strong baseline VAR-d20.

NFIG: Multi-Scale Autoregressive Image Generation via Frequency Ordering
Zhíhao Huang 1.2 Xi Qiu?
Yukuo Ma 2,4 Yifu Zhou 12 Junjie Chen?
Hongyuan Zhang
Chi Zhang Xuelong Li
Northwest Polytechnical University 3 University of Hong Kong 4 "Beihang University
2 TeleAI, China Telecom
huangzhihaoommail. cawpu.edus cn, hyzhang380gmail.com com,
2tangcc22@chiinatellecoom en, beelong_l@iiest
Abstract
Autoregressive models have achieved significant success in image generation. How- ever, unlike the inherent hierarchical structure of image information in the spectral domain, standard autoregressive methods typically generate pixels sequentially in a fixed spatial order. To better leverage this spectral hierarchy, we introduce Next- Frequency Image Generation (NFIG). NFIG is novel framework that decomposes the image generation process into multiple frequency guided stages. NFIG aligns the generation process with the natural image structure. does this by first gener- ating low-frequent frequency components, which efficiently capture global structure with significantly fewer tokens, and then progressively adding higher-frequene details. This frequency-awan paradigm offers substantial advantages: it not only improves the quality of generated images but crucially reduces inference cost by efficiently establishing global structure early Extensive experiments the ImageNet-256 benchmark validate NFIG's effectiveness, demonstrating superior performance (FID: 2.81) and notable 1.25x speedup compared to the strong baseline VAR-d22 The source code is available at https://ithub..om//iid hortubbbb.commiidddduuua//EECCC
Introduction
The synthesis of images has emerged as a fundamental challenge computer vision 1
7 g 10 11 [2]. Rapid progress in this field has been propelled by deep generative models, such as autoregressive models (AR) [13, Generative Adversarial Networks (GANs), and diffusion models (SD) [4].
Despite remarkable advances in existing methods, AR models for image generation still face several fundamental challenges. On the one hand, due to their inherently local and sequential nature, most current AR models struggle to effectively capture long-range dependencies and global structure [15]. For example, PixelCNN [9] generates an image by predicting each pixel in raster scanning sequence, which neglects the global image structure and relationships with distant elements. On the other hand, the generation process is computationally intensive and time-consuming, always generate pixels or patches sequentially in predetermined order, with each new element requiring the computation of conditional probabilities based on all previously generated content [16] 17 For instance, ViTVQ [18] requires more than seconds to generate 256 x 256 image over
AR models
"Corresponding authors ?Completed during internship at TeleAI.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Next-Pater Prediction (Taming TF, 2022)
Next-Frequency Prediction (NIFG, 2025)
Image Spatial Correlation Oversight
Fast Fourier Transform
Prediction
Frequency
(VAR, 2024)
High
Cross-spectral Dependency Blindness
Spectral-aware Multi-scale Alignment
Figure I: Illustration of three autoregressive image generation frameworks. The figure demonstrates three prediction approaches: Next-Patch Prediction (patch-based progression), Next-Scale Prediction (coarse-to- to-tie resolution generation), and Next- Next-Frequency Prediction (NFIG), which performs image generation by progressively predicting and synthesizing frequency components from low to high, resulting in coarse-to-fie -oneee spatial reconstruction.
1024 steps, making it impractical for real-time applications. Most importantly, AR models face fundamental challenge defining a meaningful autoregressive sequence. Traditional AR models using raster scanning predefined arbitrary orders fail to reflect the natural hierarchical structure and dependencies images [1]] This improper sequence design makes difficult for models to capture the true causal relationships between different image components, ultimately affecting the coherence and visual quality generated outputs.
To address these limitations, recent works have explored incorporating various improvements into the generation process. For example, Taming Transformer (20) partially addresses long-range dependency challenges through its discrete latent space and transformer architecture, but still suffers from computational inefficiency. Fast PixeICNN+ [2] speeds up generation in convolutional autoregressive models by caching hidden states avoid redundant computation, achieving up to 183x speedups, yet it doesn't 't fundamentally change the autoregressive sequence design. VAR [19] leverages the Laplacian Pyramid as prior to guide autoregressive image generation across different resolutions, achieving improved generation quality with reduced computational overhead. However, these methods do not fully exploit the potential of natural priors inherent in raw images guide the generation process and improve the efficiency of AR models.
In fact, the natural structure of images follows hierarchical frequency distribution-low fre- quencies encode global structures while high frequencies contain local details. This organization suggests efficient autoregressive generation sequence from low to high frequencies, aligning with visual information's s natural structure. Since low-frequency components require fewer tokens to represent, this approach enhances computational efficiency. Similar have proven effective in diffusion models, which build from low-frequency foundations before adding higher-freee frequency details 22
progressive principles
Motivated by this insight, we propose a Next-Frequency Image Generation (NFIG) framework for AR models that: (1) first generates low-frequency image with few tokens capture global structure; (2) then progressively adds gggeuerfreeuency components conditioned on the low-frequency foundation. This process has been shown in Figure Grounded in information theory, this approach efficiently represents information across the frequency spectrum using our Frequency-guided Residual-quaatized VAE (AR-
Key contributions of the NFIG framework include:
We introduce Next-Frequency Image Generation (NFIG) framework that incorporates frequency analysis into AR image generation. To our knowledge, this work is the first to guide autoregres- sive generation using the image's s frequency spectrum, associating low frequencies with lower resolutions and high frequencies with higher resolutions:
To demonstrate the feasibility of the NFIG paradigm, we design Frequency-guides Residual- quantized VAE as our image tokenizer. FR-VAF separates low and high-frequency components in the representation learning process, with low frequencies encoding global structure and high

---

frequencies preserving local details. Experiments show FR-VAE achieves a reconstruction FID of 0.85, validating image content preservation capability;
Through extensive experimentation, we show that our approach achieves state-of-the-an art image generation quality, evidenced by an FID of 2.81 from relatively small model. This improvement paves the way for more effective and efficient AR image generation models, making them more practical for real-world applications.
Table
Main Notation Table
Symbol
Meaning
Dimension
Input image
H W x
Reconstructed image
H X W x 3
Feature map from VAE encoder
H' X W C
M
The i-th frequency selection mask Set of i-th frequency component feature maps Scaled feature map for i-th frequency component Quantized representation for i-th frequency component
H' X W X C
fr
H' X W X C
h X W.
h X WT C
R
Cumulative signal residual through level The learnable codebook of FR- VAE The i-th frequency band
H' X W
Z
K C
F.
N/A
Related Work
Autoregressive Image Generation Autoregressive image generation has demonstrated remarkable capabilities in producing high-quality images by modeling the joint distribution of image tokens as a product of conditionals [23]. PixelCNN 9 generates images sequentially, processing pixels one by one (typically top-left to bottom-right). It employs masked convolutions that the generation of each pixel depends solely on the pixels already generated. Taming Transformer (20) introduces an autoregressive approach that generates high-resolution images by predicting the next latent patch token in a discrete compressed space learned through vector quantization. Emu3 [4] patchifies an image into series of tokens and generates images by predicting tokens in raster-scan scan manner. VAR [99 incorporates the prior knowledge of Laplacian Pyramid into transformer architecture and generates images in next-resolution manner. MAR [25] improves the quality of generated images by replacing discrete tokens with continuous features, recognizing that autoregressive models primarily need per-token probability distributions. FAR (26 attempts enhance MAR performance through frequency- approaches, yet lacks critical insights into the distinctive information characteristics across different frequency bands. Infinity 225 combines autoregressive modeling with Bit-wise Modeling to enhance visual details in high-resolution image synthesis. ImageFolder (27) utilizes folded image tokens to generate high-quality images in next-scale prediction manner, achieving superior performance. These approaches collectively demonstrate the evolution of autoregressive image generation techniques. progressing from pixel-level prediction more sophisticated methods involving latent spaces, hierarchical structures, and physical priors. Despite their differences in implementation, all these methods share the fundamental autoregressive principle sequentially generating image elements conditioned on previously generated content.
Image Tokenizer Image tokenizers, which transform continuous image data into discrete repre- sentations, have become a critical component in modern image generation systems. VQ-VAE (28) introduces vector quantization into VAE, reducing the pressure on the downstream generative model by transforming the continuous latent space into a discrete one. VQ-VAE-2 (29] extends this idea with hierarchical framework and multi-scale codebooks, where top-level codes capture global structure, and bottom- level codes model local details, enabling higher-qualin reconstruction and generation at increased resolutions. However, VQ- VAE- methods often suffer from codebook collapse, where only few codebook entries are effectively used. FSQ [30 attempts to address this issue by utilizing finite scalar quantization to learn the codebook, but it does not learn a meaningful feature of images. RQ-VAE (31] employs residual quantization with a shared codebook to enhance reconstructed image quality. XQGAN 32 introduces feature product decomposition and residual quantizer to enhance VQ- VA-'s performance, leading to improved image generation results. While these approaches have
3

---

E RH
Low
High
Frequercy
Encoder
FER
Frequency- guided Decomposer
[í RR H''W"xxxx" Xx
(b) Frquency-guide Decomposer and Composer
Residual Quantizer
{ug E R n
Frequency-guict Composer
f "''''xxx
Frequency-aware Transformer
Decoder
I E R HxW
bos
.vok
Mick
(a) Frequency-guidee Residual-quantizer VAE
(c) Next-Froqueacy Prediction
Figure 2: Overview of the Next- Fext-Frequency Image Generation (NFIG) Framework: (a) The Frequency- guided Residual-Quael Quantization VAE encodes images into and decodes from frequency-guide residual quantized representations; (b) The image is decomposed into frequency components (low to high) and reconstructed progressively by merging these components for coarse-to-fin process; (c) Next- Frequency Prediction model employs a frequency-ave token sequences, with each block with same color representing specific frequency band, enabling sequential image synthesis from low to high frequencies. Naok h,w; is the number of image tokens used for the ith frequency band.
Transformer to auto-regressively generate
made significant progress in image tokenization, they often neglect the inherent multi-scall structure of natural images, which is crucial for efficient and effective representation learning.
Next-Frequency Image Generation
To provide a comprehensive understanding of our NFIG methodology, this section delves into its intricate architectural structure. The essential operational sequence, illustrating the flow and interaction of the system's key components, is clearly visualized in Figure 2 The details of loss function have been listed in Appendix B.I.
3.1 Frequency-guided Residual-quantized VAE
The workflow of Frequency-guided Residual-quantized VAE has been shown in Figure (a). To generate images in trequency-aware manner, we propose Frequency guided Residual quanttized VAE (FR- VAE) with VQ-GAN framework. The key idea is represent lower-frequency signals with fewer tokens and higher-frequency components with more tokens.
3.1.1 Frequency-guidee Reconstruction
As illustrated Figure (b), raw images can be decomposed into components across different frequency bands: low frequencies encode the global structure, while high frequencies retain fine details. Utilizing the Frequency-guided Decomposer and Composer, these components can be recombined without loss, ensuring a complete and accurate visual representation.
Frequency-guided Decomposer. Given a image æ E "#xx*x x3 and a encoder E(-), there is image latent feature f E(x) and H' xW'xC FR-VAE decomposes f into several component with

---

different frequency by Frequency- guided Decomposer via Fast Fourier Transform (FFT);
(1) Here, @ represents the element wise product. F signifies the FFT operation. _F1 indicates the inverse FFT, and M, is the i-th frequency mask used to select the desired frequency range, n the total number of frequency masks, fi is the component corresponding to M,.
f M.). Vi [L......))
Frequency-guider Composer. Frequency-guidder Composer reconstructs the raw image by inter- polating different frequency components to a uniform size and merging them into a single image, as illustrated in Figure (b).
11 == I(
where I(-, H'' W") is the interpolation function, which enables the Frequency-guided Composer to process images of different frequencies at varying resolutions.
3.1.2 Frequency-guided residual-quantization
To efficiently represent images with minimal tokens, we implement frequency-gui -uidedd residual quan- tization approach that addresses information loss during downsampling, Our method progressively captures different frequency components of an image through residual learning scheme.
Residual Token Extraction. Given sequence of feature maps with different dimensions {(h1, ".) (h,, wn)]: }. where h2 2 by and ny 2 wy if 2 j. and h = H' and to, = W", we supervise the learning process using accumulated signals from the lowest frequency the current frequency band.
The residual R E C'''xC and representation U E Xmxxx of the i-th frequency component can be computed follows:
f. I(vi, HI', H'' ('),
i= 12 1
R₂
(3)
Ri-1 + (f: I(u; H', W")),,
arg mille; [1ʃ: I(v;, H' W arg ming, [|(R,-1 + fi) I(u,, H' W')l, 121 1
i= 0
(4)
where I(U;, H', W is the interpolation function that upsamples v; to the original feature map size, and R² represents the difference between the accumulated frequency components up to the i-th level and the learnable features.
In general, autoregressive models utilize the discrete tokens to generate image. To achieve this goal, we take simple vector quantization transform the continuous token into discrete tokens.
Vector Quantization.
We define a quantizer Q with a learnable codebook Z R KCC this codebook, the quantizer Q transforms a continuous feature map V; E X into set of
xC containing K code vectors. Using
discrete tokens (1(1.1) : ((1.2) ..... ((fh..u))), where each token e(5.k) has the corresponds to
_(ih)) E CR. The process of finding the optimal code representation involves:
{(,)) = lookup Z, arg min ||2 (1.) o (1.8) (12)
(5)
2 2(it) EZ
From Uú the quantized feature map of E X and set of discrete tokens ((( are obtained
through quantization using the codebook Z. Here, lookup(Z, x) is function that finds the index of the closest entry in codebook Z.
3.2 Autoregressive Image Generation
To generate images progressively from low high frequency components, we implement decoder- only transformer framework and block-wise causal attention [19].
5

---

Next-Frequency Image Prediction Unlike conventional autoregressive image generation models that employ "token-by-token prediction" strategy. which often neglects spatial relationships and inherent image structure. NFIG adopts "Coarse-to-Fine Generation" approach, first synthesizes the fow-freuueccy components of an image, then iteratively incorporates higher-frequency details, progressively refining the generated output at each step, as shown in Figure (c). The generation process for nexi-frequency prediction given by the autoregressive factorization:
p(T,, .. T.) = II (TIT, 1. (T--)
where T; E [K]ha KK/uxx: is the matrix of code indices for the i-th frequency component, and the set [K] (1,2, ...., K} represents all available index values.
Frequency Band Division Strategy We treat the lower frequency components as the foundation for generating T,, represented by {Ti T2. T,-1), According to information theory principles, lower-frequency signals contain less information and require fewer tokens, while frequecy components carry more detailed information and need more tokens for accurate representation.
Consequently, we establish increasing scale sequence {h1 w1). (h2,, W2). .(,, wn)} } for com-
ponents with increasing frequency bands {Fi, F2, ........ F = {[0, 01), [1, (2) o2).... [on-1 on Here, maz denotes the maximum frequency of the entire image feature map f, with on = man- We divide the frequency bands based on their corresponding resolution as:
h
0, 0;-1 + Σ
mmaz-
jj=1 by Wj
This rreuenncy-guider progressive approach allows our model to capture and prioritize salient components at each stage. The method improves both computational efficiency and image quality by explicitly modeling the multi-scale frequency structure inherent in natural images.
4 Experiment
This section details our experimental methodology, covering datasets, evaluation metrics, comparison baselines, and implementation specifics. We then evaluate NFIG against state-of-the-e af-thh-art approaches on image generation benchmarks. Subsequently, ablation studies and motivation verification experiments are conducted analyze the impact of different components and validate design decisions.
4.1 Experimental Settings
For the purpose of experiments, we the ILSVRCC 2012 subset of ImageNet [33] which comprises total of 1.2 million training images, 50k validation images, and 100k test images. This subset focuses on 1k object categories, with each category having approximately 1.2k training images, 50 validation images, and 100 test images.
Dataset.
Evaluation Metrics. We adopt four metrics for quantitative evaluation: Fréchet Inception Distance (FID) which measures distribution similarity between generated and real images, Inception Score (IS) which assesses quality and diversity, and Precision (Pre) and Recall (Rec) which evaluate sample fidelity and diversity coverage respectively.
Baselines. Our method is benchmarked against several leading image generation techniques, includ- ing generative adversarial networks (GAN), diffusion models (Diff.), mask diffusion (Mask.), and autoregressive models (AR). These approaches have demonstrated strong performance various image synthesis tasks and serve as robust comparators.
Implementation Details. Our model was implemented using the PyTorch framework and trained on NVIDIA H100 graphies cards. To ensure the reproducibility of our experiments, our implementation is built upon open-source research code, while incorporating improvements specific to this study. For the image tokenizer, the TR-VAB incorporates VQGAN architecture with a DINO discriminator. The image encoder is initialized with pretrained weights from DINOv2-base. Since VAR's s image tokenizer training code is not open-source, we adopted XQGAN's implementation strat- egy. The frequency residual quantizer employs multiple scaling factors [1, 2, 3, 4, 5, 6, 8, 10, 13, 16)

---

Table 2: Performance class-conditional ImageNet 256x256 for image generative model. rFID represents reconstruction FID, while gFID indicates generation FID. "L" or "I" indicate lower or higher values are better. "#Step": the number of model needed to generate image. Wall-clock inference time relative to NFIG is reported. Models with the suffix "-re" used rejection sampling. For comprehensive evaluation, separately compare autoregressive
i: taken from MaskGIT 34
(AR) and non-autoregressive (non-AR) models, with the best metries highlighted in bold.
Type
Model
rFID
gFIDL
ISt
Pre
Rec
#Para
#Step
Time
GAN
BigGAN
B5
6.95
224.5
0.89
0.38
112M
1
GAN
GigaGAN
36
3,45
225.5
0.84
0.61
569M
1
GAN
StyleGan-XI.
2.30
265.1
0.78
0.53
166M
1
0.75
Diff.
ADM
10.94
101.0
0.69
0.63
554M
250
420
Diff.
CDM 38
4.88
158.7
8100
Diff.
LDM-4-G 39
3.60
247.7
400M
250
Diff.
DIT-L/2
40
0.9
5.02
167.2
0.75
0.57
458M
250
77.5
Diff.
D/T-XL/2
40
278.2
0.83
0.57
675M
250
112.5
Diff.
L-DiT-3B
41
0.9
2.10
304.4
0.82
0.60
3.0B
250
> 112.5
Diff.
L-DiT-7B
41
0.9
2.28
316.2
0.83
0.58
7.0B
250
> 112.5
Mask.
MaskGIT
R4
2.28
6.18
182.1
0.80
0.51
227M
8
1.25
Mask.
RCG 42
3.49
215.5
502M
20
4.75
AR
VQVAE-21
34
2.0
31.11
45.0
0.36
0.57
13.5B
5120
AR
VQGANf
20
7.94
18.65
80.4
0.78
0.26
227M
256
47.5
AR
VQGAN
20
7.94
15.78
74.3
1.4B
256
60
AR
ViTVQ
18
1.28
4.17
175.1
1.7B
1024
>60
AR
ViTVQ-re
227.4
1.7B
1024
>60
AR
RQTran.
43
1.83
7.55
134.0
3.8B
68
52.5
AR
RQTran-fre
45
1.83
3.80
323.7
3.8B
68
52.5
AR
FAR-B 26
4.26
248.9
0.79
0.51
208M
10
FAR-B
26
3.45
282.2
0.80
0.54
427M
10
AR
FAR-H 26
3,21
300.6
0.81
0.55
812M
10
AR
XQGAN-310M 32
0.78
2.96
310M
10
AR
VAR-d16 19
0.9
3.55
274.4
0.84
0.51
310M
10
AR
VAR-d20
0.9
2.95
302.6
0.83
0.56
600M
10
1.25
AR
NFIG(Ours)
0.85
332.42
0.77
0.59
310M
10
across different frequency bands, resulting in vocabulary size 680 tokens. The VA- codebook size of 4096 utilized. The image generator employs a VAR Transformer backbone with depth of 16, enabling multi-scale image prediction. Optimization was performed using the Adam optimizer, setting the learning rate 10 and the batch size 768. Training of the model ran for 350 epochs on the ImageNet dataset. For inference, we configured CFG to 4.5 and top_k k to 990.
4.2 Main Results
Table 2 provides detailed comparison of our approach against leading image generative models evaluated on ImageNet 256 x 256. The findings indicate that NFIG achieves superior performance within the AR model family while establishing itself formidable competitor among diverse generative methods across different paradigms.
AR Model Comparison. NFIG achieves the best gFID (2.81) and IS (332.42) scores, significantly outperforming other AR models. Compared to VAR-16 (gFID: 3.55, IS: 274.4), our approach reduces FID by 0.74 and improves IS by more than 21%. XQGAN-31 1GGAN-310M has better image tokenizer with rFID 0.78 with gFID 2.96. This indicates that NFIG's performance improvement is not solely due to a good image tokenizer, but more importantly, to the injection of image fiequeecyy priors. Additionally, the proposed approach outperforms VAR-d20, a relatively larger model, while delivering 25% faster inference speed.
Cross-family Comparison. NFIG achieves competitive performance with the best models from other families. NFIG outperforms the best mask diffusion model RCG, which has a gFID score of 3.49 and score of 215.5. While some GANs like StyleGAN-XL. have lower gFID scores (2.30) with moderate IS (265.1), 1),, and diffusion models like DİT-L/2 show excellent gFID (2.27) and strong IS (278.2), NFIG uniquely balances both metrics at high levels (gFID: 2181, IS: 332.42). This establishes NFIG as not only the leading AR model but also strong competitor across all model types.
7

---

qualitatively shows NFIG's impressive ability to generate diverse ImageNet 256 256 images across a wide variety of categories. Appendix B.5 show the Failure case of NFIG.
Qualitative Results. Figure 3
Scaling Up. To validate NFIG's scaling behavior, train 310M and 600M parameter models for 55 epochs under computational constraints. The results are shown in Table 3
Performance of Different Epochs. VAR sets different training epochs for models of different sizes: 200 epochs for 310M, 250 epochs for 600M, 300 epochs for IB, and 350 epochs for 2B. Limited by computational resources, focus training 310M model. As Table 4 shows, NFIG outperforms VAR at matched epochs and demonstrates superior parameter efficiency. At 200 epochs, NFIG already outperforms VAR-d16 of the same size. With extended training, NFIG-310M achieves performance comparable to or better than VAR-d20-600M (twice the parameters) while using significantly fewer resources.
Table 3: Performance of NFIG with different parameters at 55 epochs.
Model
FID
1St
Precision
Recallf
Epoch
Para
steps
Time
NFIG-310M
5.47
224.20
0.7569 0.7184
0.4914
55
310M
10
NFIG-600M
5.07
225.16
0.5546
600M
10
1.25
Figure 3: Generated 256
256 examples by NFIG trained Imagenet.
Table 4: The performance of NFIG and VAR different epochs.
Model
Epochs
FID
ISt
Pret
Rect
Params
Steps
Time
VAR-d16
200
3.55
274.4
0.84
0.51
310M
10
VAR-d20
250
2.95
302.6
0.83
0.56
600M
10
1.25
NFIG(ours)
200
3.35
309.2
0.79
0.55
310M
10
NFIG(ours)
250
3.16
311.9
0.78
0.56
310M
10
NFIG()urs)
300
2.93
325.9
0.79
0.56
310M
10
NFIG(ours)
350
2.81
332.4
0.77
0.59
310M
10
4.3 Ablation Study
To evaluate the contribution of various components within our proposed NFIG model, we perform comprehensive ablation analysis on the ImageNet validation set. Table summarizes the results of

---

this study, with performance evaluated using rFID and gFID. We start with the baseline AR model with sequence length of 256, which achieves an rFID of 1.62 and a gFID of 18.65.
Image Tokenizer. We incrementally add components to progressively improve the model's per- formance. First, incorporating Trequency-guider residual quantization into the VAR framework reduces the rFID to 1.40. Next, we integrate the DINO discriminator from VAR's ' tokenizer, which substantially improves the rFID from 1.40 0.85.
Transformer. We then utilize the image tokenizer (FR-VAE) to train the transformer model. Without Top k and Classifier Free Guidance (CFG), NFIG achieves a gFID of 9.7. The addition of Top k sampling strategy further reduces the gFID to 6.83. Finally, incorporating CFG yields the best overall performance, maintaining the rFID 0.85 while dramatically improving the gFID 2.81.
Our experimental results demonstrate that the combination of FR-VA FAE with CFG provides optimal generation quality. Moreover, we observe that both DINO discriminator and FR- VAE contributes significantly to improving rFID for the image tokenizer. Additionally, Top_k sampling and CFG prove essential for reducing gFID. These results underscore the significance of discriminator guidance and conditional generation strategies for improving image generation quality.
4.4 Motivation Verification
Frequency Distribution Analysis. The experimental results in Figure demonstrate the progressive refinement generated images and the effective capture and synthesis of multi-scale visual features by FR-VAE The frequency spectrum visualizations reveal the model's ability to hierarchically incorporate information from low to high frequencies, resulting in generated images with rich details and natural appearance. Appendix B.2 compares the frequency keep ability of NFIG and VAR. These
Step 2
Step 4
Step 6
Step 8
Step 10
Spectrum
Spectrum
Spectrum
Spectrum
Spectrum
Figure 4: Generated images at different steps 2, 4, 8, 10 of 10-step process by TR-VAE, with corresponding frequency spectrum. In these spectrograms, brightness (red/yellow) indicates higher frequency energy while darker colors (blue) represent lower energy components. The center each plot shows low-frequency information, with frequencies increasing radially outward, revealing the evolving distribution during the generation process.
results demonstrate that NFIG's frequency-guide approach enables more effective feature learning, particularly at lower resolutions, by maintaining balanced loss values throughout the hierarchical generation process.
Frequency Guidance. Similar to NFIG, VAR follows "coarse-to-fine to--eee approach but differs significantly in loss computation across resolutions. VAR computes loss between different resolutions and the raw image, causing disproportionately large loss values at lower resolutions. In contrast, NFIG utilizes frequency components to guide feature learning, providing more balanced loss values throughout generation. As Figure 5 shows, this leads to dramatic variations in vector quantization

---

Vector Quantization Loss Comparison
Table 5: Ablation study the improvement of NFIG. We evaluate rFID and gFID on the Ima- geNet validation set. "FR- "FRR-Quatiizer" is the quan- tizer of FR-VAE. "DINO-Dise" means "DINO Dis- criminator", which denotes the discriminator used in VAR's (I9) tokenizer.
FIG
VAR
2.75
2.47
Quanization Loss
2.15
1.8
1.50
1.5
Metric
1.0
0.79
Type
Mothod
Length
[HD
gRDD
0.5
0.61
AR
256
1.62
0.0
0.01
0.01
0.00
Image Jokenizer
FR FIR-Qunnizer
680
1.40
DINO-Dise
680
0.85
Image Scale Factor
Genenatio Trunsformer
4
+ AdaLN
680
0.85
9.7
Figure 5: Vector quantization loss comparison between NFIG and VAR across image scales.
+ Top_k
680
0.85
CFG
680
0,85
2.81
loss VASs~VR exhibits substantially higher values across all scale factors, while NFIG maintains considerably lower loss values across all resolutions.
5
Conclusion
This paper introduces FexttFreyuency Image Generation, novel autoregressive framework that decomposes image generation into frequency-guider stages. Our key insight leverages the hierarchical spectral distribution of natural images: low-frequency components encode global structures and long-range dependencies, while high-frequency components contain local details requiring greater information entropy. By progressively generating from low to high frequencies, the proposed method significantly outperforms existing models with comparable parameter counts, demonstrating superior quality metrics while maintaining computational efficiency. Experiments confirm that our frequency- guided approach represents an important advancement in autoregressive image synthesis.
6 Limitation and Future Work
Our frequency-guided autoregressive image generation approach shows promise, but has limitations. Improving frequency decomposition. A primary issue is the simplistic frequency band division by scale, which inadequately captures information in the first band. Implementing a more rigorous division based on statistical analysis and physical principles would likely enhance NFIG's perfor- mance. Recent advances in beneficial noise theory [45 46) suggest that properly designed noise can reduce task complexity, which could inform better frequency decomposition and data augmentation Extension to other modalities. Beyond 2D spatial frequency, future work could extend video generation by incorporating temporal frequency decomposition, or to 3D object generation where frequency analysis is vital for accurate light field representation. For multi-modal generation, techniques that enhance cross-modal alignment through learnable noise (49 may offer insights for frequency-basee fusion strategies. techniques (50) could be integrated with our framework to ensure privacy protection generated content. Due to computational constraints and time limitations, these promising directions remain unexplored and are left for future investigation.
strategies [47 48
pewwacy-pesservng generation. Adversarial noise
References
[1] Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1):1391144 2020.
[2] Tero Karras, Samuli Laine, and Timo Aila. style-based generator architecture for generative adversarial networks. In IEEE Conference Computer Vision and Patiern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 4401-4410. Computer Vision Foundation / IEEE, 2019.
10

---

[3] Axel Sauer, Katja Schwarz. and Andreas Geiger. Stylegan-xi xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[4] Jihyun Kim, Changjae Oh, Hoseok Do, Soohyun Kim, and Kwanghoon Sohn. Diffusion- driven gan inversion for multi-modal face image generation. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 10403-10412, 2024.
the JEEE/CVF
[5] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances neural information processing systems, 33:6840-6851, 2020.
[6] Yufan Zhou, Bingchen Liu, Yizhe Zhu, Xiao Yang, Changyou Chen, and Jinhui Xu. Shifted diffusion for text-to-image generation. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 10157- 10166. IEEE, 2023.
[7] Alexandros Graikos, Srikar Yellapragada, Minh-Quaa Le. Saarthak Kapse, Prateek Prasanna, Joel H. Saltz, and Dimitris Samaras. Learned representation--uiiee diffusion models for large- image generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 8532 8542. IEEE, 2024.
[8] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single trans- former to unify multimodal understanding and generation. arXiv preprint arXiv:2408. /2528, 2024.
[9] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, al. Con- ditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
[10] Menggi Huang, Zhendong Mao, Zhuowei Chen, and Yongdong Zhang. Towards accurate image coding:
Improved autoregressive image generation with dynamic vector quantization. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, Canada, June 17-24, 2023, pages
BC,
IEEE, 2023.
[11] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 24449-2644... IEEE, 2024.
[12] Haohan Weng, Danqing Huang, Yu Qiao, Zheng Hu, Cew Lin, Tong Zhang, and CL Chen. Desigen: A pipeline for controllable design template generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12721-12732. 2024.
[13] Runqi Wang, Caoyuan Ma, Guopeng Li, Hanrui Xu, Yuke Li, and Zheng Wang. You think, you act: The new task of arbitrary text motion generation, 2025. URL httts: org/ abs/2404. 14745
[14] Runqi Wang, Caoyuan Ma, Jian Zhao, Hanrui Xu, Dongfang Sun, Haoyang Chen, Lin Xiong, Zheng Wang, and Xuelong Li. Leader and follower: Interactive motion generation under trajectory constraints, 2025. URL
11563
[15] Rewon Child. Very deep generalize autoregressive models and can outperform them images. In International Conference on Learning Representations, 2021.
[16] Chenlin Meng, Lantao Yu, Yang Song. Jiaming Song, and Stefano Ermon. Autoregressive score matching. Advances in Neural Information Processing Systems, 33:6673-6683, 2020.
[17] Jiawei Shao and Xuelong Li. Ai flow at the network edge. IEEE Network, 2025.
[18] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.

---

[19] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive mod- eling: Scalable image generation via next-scale prediction. In Advances in Neural Information wwtp:iipps...pppceeee
Processing Systems 38, 2024, URL httphh mips. cc/ wwwppper.pppppppcpppppppeeee.... files/ fwwes/ppper///eeeeeeeeaaaaaaeeet pappp/ /024/ /
ha22:122222222111666664444666662 Confereeee .html
[20] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pages 12873-12883, 2021.
[21] Prajit Ramachandran, Tom Le Paine, Pooya Khorrami, Mohammad Babaeizadeh, Shiyu Chang, Yang Zhang, Mark Hasegawa-Johnse Roy Campbell, and Thomas Huang. Fast genera- tion for convolutional autoregressive models. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
[22] Yurui Qian, Qi Cai, Yingwei Pan, Yehao Li, Ting Yao, Qibin Sun, and Tao Mei. Boosting diffusion models with moving average sampling in frequency domain. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 891 1-8920, 2024.
[23] Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Zhe Lin, Rita Singh, and Bhiksha Raj. Controlvar: Exploring controllable visual autoregressive modeling. arXiv preprint 2Xii:24406009765 2024.
[24] Xinlong Wang, Xiaosong Zhang. Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tiejun Huang, and Zhongyuan Wang. Emu3: Next-token prediction is all you need. CoRR, Abs/2409.1886) 2024.
[25] Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint 1.1i::2006.11833, 2024.
[26] Hu Yu, Hao Luo, Hangjee Yuan, Yu Rong, and Feng Zhao. Frequency autoregressive image generation with continuous tokens. arxiv: 2503.05303 2025.
[27] Xiang Li. Kai Qiu, Hao Chen, Jason Kuen, Jiuxiang Gu. Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregressive image generation with folded tokens. CoRR, abs/2410.017552 2024.
[28] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[29] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelig images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[30] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: VQ-VAE made simple. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 1, 2024, 2024.
[31] Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523- 11523-11533 11532, 2022.
[32] Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Jiuxiang Gu, Jindong Wang, Zhe Lin, and Bhiksha Raj. Xq-gan: An open-source image tokenization framework for autoregressive generation, 2024. http:/ttt////xxx//a org/ abs/ /0tp.//42..0000 01762.
[33] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252, 2015.
[34] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse hige-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[35] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. International Conference on Learning Representations, 2019. URİ.
12

---

[36] Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10124-10134, 2023.
[37] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 2021.
[38] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[40]
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195 2023.
[41]
Alpha- VLLM. Large-dit-imagenet, 2024. URL
VLLM/
LLaMA2 http://ccacy.oooottoooooooooaass
Large Marge-DiT-ae D ImageNet, Accessed: 2, 8.
[42] Tianhong Li, Dina Katabi, and Kaiming He. Self-conditioned image generation via generating representations. CoRR, 2023.
[43] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523- 11532, 2022.
[44] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[45] Xuelong Li. Dositive-incentive noise. IEEE Transactions on Neural Networks and Learning Systems, 35(6))8708-877) -814,, 2024.
[46] Hongyuan Zhang, Sida Huang, Yubin Guo, and Xuelong Li. Variational positive-incentio noise: How noise benefits models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(9):8313-8320 2025.
[47] Hongyuan Zhang, Yanchen Xu, Sida Huang, and Xuelong Li. Data augmentation of contrastive learning is estimating positive-incentive noise. arXiv preprint arXiv:2408. 09929, 2024.
[48] Siqi Huang, Yanchen Xu, Hongyuan Zhang, and Xuelong Li. Learn beneficial noise as graph augmentation. In Forty- International Conference on Machine Learning, 2025,
[49] Sida Huang, Hongyuan Zhang, and Xuelong Li. Enhance vision-language alignment with noise. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 17449-17457, 2025.
[50] Junying Wang, Hongyuan Zhang, and Yuan Yuan. Adv-cpg: A customized portrait generation framework with facial adversarial attacks. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 21001-22 11001-2101,, 2025.
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: None
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: None
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
None
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: None
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [No]
Justification: Due to security review requirements for the code and data, the code will be open-sourced upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips.cc/pp cc/public/guu wttpggudddee/eeebbininnneeee
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: None
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification: Constrained by computational resources, we report results from a single run.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: None
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:p/pw....aeoooooo/ooo//o//aa
Answer: [Yes]
Justification: None
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The scope of this work is limited to academic research in computer science, with no immediate further societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Thefi scope of this work is limited to academic research in computer science, with no immediate further safeguards impact.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: None
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: None
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

Appendix A: Fourier Analysis in Natural Images
This appendix provides essential mathematical formulations and conceptual insights into the Fourier analysis of natural images, building upon the concepts discussed the main text. We focus on the Discrete Fourier Transform (DFT) and its implications for image representation and the characteristies of natural scenes.
A.1 2D Discrete Fourier Transform (2D DFT)
The 2D DFT transforms M x N digital image [x, y) from the spatial domain (where M 1} and y E {o....,P. N 1} are spatial coordinates) to the frequency domain, yielding an M x N representation F(u, (where E {O...., M 1} and v E {O....,NN -- are frequency coordinates). The formula is given by:
M-IN-1
F(u,
Σ Σ 2=0
v)e )2 52m
(8)
Here, is the imaginary unit = and the exponential term represents the basis functions (complex sinusoids) at different frequencies (u, v).
A.2 Inverse 2D Discrete Fourier Transform (2D IDFT)
The 2D IDFT allows us to reconstruct the original spatial domain image f(x v) from its frequency domain representation F F(u, v). The formula is:
M-1N-1
(x,
Σ
Σ
Flu,
MN
==0
V=0
Note the scaling factor MN and the positive sign in the exponent compared the forward transform.
A.3 Magnitude Spectrum and Power Spectrum
The frequency domain representation F Fu, v) obtained from the DFT is generally a complex number. Its magnitude, [u, v) is known as the Magnitude Spectrum, which quantifies the amplitude of each frequency component present in the image.
Flu, w) = v Re 8))² + Im( 0))²
Closely related is the Power Spectrum (or Power Spectral Density), defined as the square of the magnitude spectrum. It represents how the total signal energy is distributed across the different frequencies:
P(u, v) = (u,
key characteristic of natural images is that their power spectrum typically exhibits a rapid decay as frequency increases. Specifically, the power P(u, v) tends to fall off with increasing radial frequency fr Vu² + v², often approximated by 1/f" law, where is constant typically around 2. This 1/f property implies that low spatial frequencies (corresponding to coarse structures and overall variations) contain significantly more energy than high spatial frequencies (corresponding to fine details and sharp transitions). This fundamental statistical feature of natural images widely utilized and modeled in various image processing and computer vision tasks.
Appendix B: Addition Experiments
VAR sets different training epochs for models of different sizes: 200 epochs for 310M, 250 epochs for 600M, 300 epochs for IB, and 350 epochs for 2B. Limited by computational resources, we focus
21

---

training our 310M model. As the table above shows, NFIG outperforms VAR at matched epochs and demonstrates superior parameter efficiency. At 200 epochs, NFIG already outperforms VAR-d16 of the same size. With extended training, NFIG-310M achieves performance comparable to or better than VAR-d20-600M (twice the parameters) while using significantly fewer resources.
B.1 Details of Loss function
We will provide detailed mathematical formulations of our loss function components and their respective roles in the training process. The total loss function for the FR-VAE (image tokenizer for NFIG) is defined as:
L - 1= (1,5,()
(10)
Here, the first two terms represent the reconstruction loss for the image (1 vs ń) and and its frequency- guided quantized loss (f vs f), respectively, ensuring fidelity in both pixel and feature. Lp is LPIPS perceptual loss and Ly gan loss.
For the NFIG Transformer, which predicts the frequency tokens, we utilize a standard cross-entropy loss:
C(T,T)= = Σ 4 log((.)
(11)
This loss is computed between the predicted tokens 7 and PA- ground truth tokens T, ensuring accurate prediction of the quantized frequency representations across all scales.
B.2 Frequency Keep Ability
We are grateful for your encouragement to discuss both successes and challenges. Your suggestion for frequency analysis was particularly insightful. you requested, we conducted frequency- comparison between our model (NFIG) and VAR-16.
As requested, we provide frequency-domain comparisons between VAR-16 and NFIG: (1) Power Spectral Density (PSD): Overall frequency fidelity; (2) Frequency Keep Score (FKS): Weighted similarity across High/Mid/Low frequency bands (weights: 0.15, 0.28, 0.57, emphasizing structural low-frequency information).
Our analysis revealed that while both models effectively preserve low-freqyency information, and NFIG preserves middle and high frequency information with higher fidelity.
Model
PSD
FKS
Low
Middlet
Hight
VAR-16
0.87
79.5%
98.3%
57.6%
48.2%
NFIG(ours)
0.47
87.6%
98.9%
75.3%
66.7%
B.4 Diverse Image Types
To demonstrate broader applicability, we conducted preliminary reconstruction evaluations (FID) of TR-VAE across diverse image types.
Model
DTD
QRCODE
Diagrams
Chest-X
CelebA-HQ
COCO
LSUN-Bedroom
PR-VAE
6.86
11.01
21
0.74
3.51
7.51
6.12
B.5 Failure Case
Despite strong overall performance, NFIG occasionally produces visual artifacts. shown in Figure these include anatomical errors (extra bird leg), texture abnormalities (goldfish patterns), and fine
22

---

detail loss (bird claws). Red boxes highlight the anomalies. These failures reflect challenges in maintaining semantic consistency across frequency bands. The issues are particularly pronounced for complex structures and fine details. Such limitations are common to frequency-basee generation approaches and present opportunities for future improvement.
Figure 6: Failure case for NFIG.
23


Reviews:
[{"review_id": "xeMElC91Rh", "reviewer": "Reviewer_eDnV", "summary": "This paper focuses on the field of autoregressive image generation, proposing a Next-Frequency Image Generation (NFIG) method that leverages the inherent bias of image information decomposition to achieve a high-performance and efficient framework. The authors claim that NFIG runs 1.25× faster than VAR-d20 while outperforming larger-scale VAR models, achieving a superior FID of **2.81 on ImageNet-256**.", "questions": "Same as weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \nThis work is the first to utilize spectral information to guide and enhance autoregressive image generation through the Next-Frequency Image Generation (NFIG) approach. It also introduces a Frequency-guided Residual-quantized VAE as an image tokenizer. Experimental results demonstrate superior performance over the strong baseline VAR while maintaining comparable efficiency at the same model scale. \n\nWeaknesses: \n1.\tUnfair comparison with VAR: The publicly available VAR code shows that VAR-d16 (200 epochs) and VAR-d20 (250 epochs) were trained for fewer epochs than NFIG (350 epochs). This discrepancy raises concerns about whether the improvements stem from methodological innovation or simply longer training, casting doubt on the validity of the claimed advantages. \n\n2.\tLimited model scale exploration: The paper only evaluates a single model size (310M parameters), leaving uncertainty about the **scaling laws**—whether this method remains effective at larger scales is unknown. \n\n3.\tInsufficient methodological details: While the core idea is clearly presented, key implementation details are lacking. For example: \n1.\tThe exact composition of the total loss function and the role of each component are not clearly explained. \n2.\tHow positional information is incorporated or processed for frequency-based tokens is omitted. \n  Although the architecture is described as similar to VAR, critical differences are not thoroughly elaborated. \n\n4.\tEfficiency concerns: Since this work adopts the same generation framework as VAR but introduces additional operations (e.g., Fourier transforms and inverse transforms), **theoretical efficiency should be slightly worse**. The claim of equal efficiency at the same scale is questionable—how exactly was efficiency improved? Furthermore, these extra operations might impose greater disadvantages at higher resolutions. \n\nIn summary, this work introduces a novel perspective for image generation, contributing positively to the field. Thus, if the authors can address the four concerns above clearly, I can increase my score to 4. \n\nAdditional Questions: \n\n5.\tFundamental superiority over VAR?Some blogs, articles, and preliminary experiments suggest that VAR also generates coarse global information in early stages and gradually refines high-frequency details later. Can the authors clarify whether NFIG **truly surpasses VAR in essence**? I suspect that NFIG might simply apply Fourier transforms and inversions to intermediate results during generation, similar to VAR. Could the authors provide theoretical or empirical evidence of its genuine advantages? \n\n6.\tExtension to text-to-image generation. Since the codebook consists of frequency-based tokenized vectors, has the author considered how NFIG could be adapted to **text-to-image generation**? Would aligning these frequency tokens with text introduce additional challenges?", "was_revised": true, "final_justification": "Part of the problem has been solved, but there are still some issues that failed to convince me, please see my comment for more details. But I don't think a piece of work needs to be able to be perfect in all aspects, and there are some explorations that can expand the knowledge and benefit a lot of researchers are also enough to be rated as 4 points.", "rebuttal": "### Rebuttal by Author\n\n>**W1**: Unfair comparison with VAR: The publicly available VAR code shows that VAR-d16 (200 epochs) and VAR-d20 (250 epochs) were trained for fewer epochs than NFIG (350 epochs). This discrepancy raises concerns about whether the improvements stem from methodological innovation or simply longer training, casting doubt on the validity of the claimed advantages.\n\n**Reply**: VAR sets different training epochs for models of different sizes: 200 epochs for 310M, 250 epochs for 600M, 300 epochs for 1B, and 350 epochs for 2B. Limited by computational resources, we focus on training our 310M model. \n\nAs the table below shows, **NFIG outperforms VAR at matched epochs and demonstrates superior parameter efficiency**. At 200 epochs, NFIG already outperforms VAR-d16 of the same size. With extended training, NFIG-310M achieves performance comparable to or better than VAR-d20-600M (twice the parameters) while using significantly fewer resources.\n\n| Model     | Epochs | FID↓ | IS↑  | Pre↑ | Rec↑ | Params|\n|:----------|:------:|:----:|-----:|-----:|-----:|------:|\n| VAR-d16   |   200  | 3.55 |274.4 | 0.84 |0.51 | 310M |\n| VAR-d20   |   250  | 2.95 |302.6 | 0.83 |0.56 | 600M |\n| NFIG(ours) |   200  | 3.35 |309.2 | 0.79 |0.55 | 310M |\n| NFIG(ours) |   250  | 3.16 |311.9 | 0.78 |0.56 | 310M |\n| NFIG(ours) |   300  | 2.93 |325.9 | 0.79 |0.56 | 310M |\n| NFIG(ours) |   350  | 2.81 |332.4 | 0.77 |0.59 | 310M |\n\n\n>**W2**: Limited model scale exploration: The paper only evaluates a single model size (310M parameters), leaving uncertainty about the scaling laws—whether this method remains effective at larger scales is unknown.\n\n**Reply**: We appreciate you highlighting this limitation. While current computational constraints limit our evaluation to the 310M model, we are confident NFIG scales effectively. As NFIG retains VAR's core Transformer architecture—known for consistent scaling from 310M to 2B parameters—its fundamental scaling properties should be preserved.\n\n\n>**W3&W4**: Unclear explanation of total loss function components.\n\n**Reply**: We will provide detailed mathematical formulations of our loss function components and their respective roles in the training process. The total loss function for the FR-VAE(image tokenizer for NFIG) is defined as:\n\n$\\mathcal{L} = \\||I - \\tilde{I}\\||_2^2 + \\||f - \\tilde{f}\\||_2^2 + \\mathcal{L}_p(\\tilde{I}) + 0.5\\mathcal{L}_g(\\tilde{I})$.\n\nHere, the first two terms represent the reconstruction loss for the image ($I$ vs $\\tilde{I}$), and and its frequency-guided quantized loss ($f$ vs $\\tilde{f}$), respectively, ensuring fidelity in both pixel and feature. $\\mathcal{L}_p$ is LPIPS perceptual loss and $\\mathcal{L}_g$ is gan loss.\n\nFor the NFIG Transformer, which predicts the frequency tokens, we utilize a standard cross-entropy loss: \n\n$\\mathcal{L}_T = -\\sum_{i=1}^n t_i \\log(\\tilde{t}_i)$ \n\nThis loss is computed between the predicted tokens $\\tilde{T}$ and FR-VAE ground truth tokens $T$, ensuring accurate prediction of the quantized frequency representations across all scales.\n\n>**W3&W5**: How positional information is incorporated or processed for frequency-based tokens is omitted. Although the architecture is described as similar to VAR, critical differences are not thoroughly elaborated.\n\n**Reply**: **Our approach follows VAR's embedding strategy**, where tokens are embedded with positional information (2D spatial location), resolution hierarchy (coarse-to-fine levels), and class information. However, a critical difference lies in our Frequency Residual Quantization (FR-Quantization), which inherently imbues these tokens with multi-resolution frequency information. This distinct property, where tokens directly represent specific frequency bands, enables our progressive low-to-high frequency generation. We plan to leverage this inherent frequency information as an explicit additional embedding dimension in future work for further enhancement.\n\n>**W6**: Efficiency concerns: Since this work adopts the same generation framework as VAR but introduces additional operations (e.g., Fourier transforms and inverse transforms), theoretical efficiency should be slightly worse. The claim of equal efficiency at the same scale is questionable—how exactly was efficiency improved? Furthermore, these extra operations might impose greater disadvantages at higher resolutions.\n\n**Reply**: Our Fourier Transform is applied to the encoded features for decomposition; for a 256x256 input image, the feature map is 16x16 in size. The computational overhead of FFT operations is **negligible (100μs per sample, representing only 0.167% of the total 0.06s inference time)**. Our empirical tests show that both methods generate 40 images in approximately 4 seconds, with a difference of no more than 0.1 seconds. For higher-resolution images, the increase in neural network's computational complexity significantly outpaces that of FFT on features, thus it will not pose a significant problem.\n\n>**Q1**: Fundamental superiority over VAR? Some blogs, articles, and preliminary experiments suggest that VAR also generates coarse global information in early stages and gradually refines high-frequency details later. Can the authors clarify whether NFIG truly surpasses VAR in essence? I suspect that NFIG might simply apply Fourier transforms and inversions to intermediate results during generation, similar to VAR. Could the authors provide theoretical or empirical evidence of its genuine advantages?\n\n**Reply**: Thank you for this insightful question. **While VAR implicitly learns multi-scale features driven by data, NFIG fundamentally surpasses it by explicitly leveraging frequency information as a guiding mathematical prior**. This is a critical difference that goes beyond simply applying Fourier transforms, as our Frequency Residual Quantization (FR-Quantization) actively forces the model to learn and reconstruct specific frequency bands at each stage.\n\nOur empirical results in Figure 5 provide compelling evidence of this essential advantage: VAR's quantization loss increases from **0.41 to 3.18** across scales, whereas NFIG consistently maintains a significantly lower loss **0.001-0.14** across all scales. This 95%+ reduction in quantization loss validates that our explicit frequency guidance fundamentally improves multi-scale representation learning, which is a core limitation addressed by NFIG's architecture.\n\n>**Q2**: Extension to text-to-image generation. Since the codebook consists of frequency-based tokenized vectors, has the author considered how NFIG could be adapted to text-to-image generation? Would aligning these frequency tokens with text introduce additional challenges?\n\n**Reply**: **Limited by the computation resource, we failed to extend NFIG to text-to-image. Now, we are actively working on extending NFIG to text-to-image generation.** The main challenge lies in collecting hierarchical captions that align with our frequency-based approach - from coarse to fine-grained image descriptions. Our current strategy involves using large vision-language models to generate multi-level descriptive prompts that match our low-to-high frequency generation paradigm. This allows us to maintain consistency between text conditioning and our core frequency-guided image generation process. **While aligning frequency tokens with text inherently introduces complexities, we anticipate that our explicit frequency-guided structure will ultimately offer more controllable and interpretable text-to-image synthesis, which we are actively exploring.**\n\nI hope my response can address your concerns. If you have any further questions, please feel free to let me know. Thank you!"}, {"review_id": "dPkNwMMymv", "reviewer": "Reviewer_hBGv", "summary": "This paper introduces frequency order modeling into AR visual generation. With the observation of the low frequency encodes global structure while the high frequency local details, the authors propose the Next-Frequency Image Generation (NFIG) framework for frequency guided visual generation and a FR-VAE for visual reconstruction. Experiments demonstrate the effectiveness of NFIG.", "questions": "The overall motivation is good, and the design is feasible, but some weaknesses remain. I will raise my score if my concerns can be addressed.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1) The motivation is clear, and frequency-based AR modeling shows potential in addressing the problem of error accumulation in vanilla AR generation. Encoding the feature into different levels of frequency helps formulate a new generation paradigm and increase inference speed.\n2) The FR-VAE encodes the visual feature into the frequency-based domain, which further improves reconstruction performance.\n3) Next-Frequency Image Prediction adopts the \"Coarse-to-Fine\" generation process that is more suitable for visual generation instead of the vanilla one. The experiment shows promising results.\n\n**Weakness**\n1) It is unclear what the notation v_i signifies. It seems the f_i and v_i denote the same one. This part should be further clarified for a clear illustration.\n2) Can frequency-based AR modeling be leveraged into single-scale prediction, e.g., LlamaGen. Since it can be easily extended to a unified model framework. \n3) Why can the NFIG accelerate the generation process. The total predicted tokens are the same as VAR.\n4) I am worried about the scalability of the NFIG framework, since it is key to further application.\n5) In Table 3, the improvement of incorporating FR-Quantizer is not significant and most of the benefits are brought by the Dino-discriminator.", "was_revised": true, "final_justification": "Given the new proposed frequency modeling in AR generation, which encompasses high potential of further developing a more suitable generation paradigm instead of plain AR, I decide to raise my score to 4. However, some of my concerns should be further addressed.", "rebuttal": "### Rebuttal by Author\nWe are grateful for your thorough and insightful feedback. Our responses to each comment are detailed below:\n\n>**W1**: It is unclear what the notation v_i signifies. It seems the f_i and v_i denote the same one. This part should be further clarified for a clear illustration.\n\n**Reply**: Thank you for pointing out this ambiguity. We apologize for the notation's lack of clarity. **f_i denotes the unquantized feature, while v_i represents the quantized  feature for f_i.** We will clarify this distinction explicitly in the paper.\n\n>**W2**: Can frequency-based AR modeling be leveraged into single-scale prediction, e.g., LlamaGen. Since it can be easily extended to a unified model framework.\n\n**Reply**: Yes, our frequency-based AR modeling can certainly be leveraged into a single-scale prediction framework like LlamaGen. \n\n**The core idea would be to replace LlamaGen's image VQ-VAE with our FR-VAE**. Our FR-VAE is designed to provide frequency-aware visual tokens, which would naturally integrate into LlamaGen's autoregressive architecture.\n\nIt's worth noting that to truly maximize the benefits of NFIG's low-to-high frequency progression within such a unified model, we should collect both coarse-grained and fine-grained textual descriptions in the image prompts and captions. This would allow the model to leverage the different frequency tokens more effectively, aligning the textual input with the hierarchical visual information.\n\n>**W3**: Why can the NFIG accelerate the generation process. The total predicted tokens are the same as VAR.\n\n**Reply**: The total number of predicted tokens in NFIG is similar to VAR. However, the acceleration in the generation process doesn't come from reducing the token count, but from NFIG's superior efficiency and quality-to-model-size ratio.\n\nSpecifically, **while NFIG (a 310M parameter model) has comparable inference speed to VAR-16 (also a 310M model), NFIG significantly outperforms the much larger VAR-20 (a 600M parameter model) in generation quality**.\n\nTherefore, when comparing models that achieve a similar high level of generation quality, NFIG provides faster inference because it achieves that quality with a substantially smaller and more efficient architecture than larger VAR variants. Our speedup is realized by delivering better results with less computational overhead per generated image.\n\n>**W4**: I am worried about the scalability of the NFIG framework, since it is key to further application.\n\n**Reply**: That's a very important point, as scalability is indeed crucial for future applications. We will discuss it from following two points:\n\n- **Scaling Law**: **We're confident in the scalability of the NFIG framework because it inherits the architectural strengths of VAR**. Due to the Transformer in our model architecture inheriting from VAR, we expect to demonstrate comparable advantages in terms of scaling laws. This means that as we scale up NFIG's model size and training data, we anticipate similar performance gains and efficiency characteristics as observed with VAR.\n\n- **Easy to extend to more application**: NFIG can be utilized to text-to-image, visual question answer and unified multimodal model(such Show-o, MMaDA). For FR-VAE can replace their image tokenizer easily. Furthermore, NFIG offers an additional advantage: its flexible design for frequency band partitioning. This flexibility becomes particularly effective when explicit image frequency priors are available, allowing for more targeted and efficient scaling based on the nature of the data. \n\n>**W5**: In Table 3, the improvement of incorporating FR-Quantizer is not significant and most of the benefits are brought by the Dino-discriminator.\n\n**Reply**: Thank you for your observation regarding the contributions shown in Table 3. We'd like to clarify the role of the DINO-discriminator in our evaluation.\n\nIt's important to note that the DINO-discriminator is also utilized by our baseline, VAR, during its training process. We've confirmed this directly through examination of the original VAR codebase. **While VAR's paper might not detail its use of GAN loss, it's a well-established practice in the field that models based on the VQGAN architecture, as both VAR and NFIG are, benefit significantly from GAN training with a discriminator.** This use of adversarial loss is, in fact, a core contribution of the VQGAN framework over a plain VQVAE.\n\nTherefore, the improvements seen from the DINO-discriminator are present in both our method and the baseline. The key takeaway from Table 3 is the incremental gain provided by our FR-Quantizer and the subsequent frequency-guided generation strategy on top of this shared adversarial training foundation. **The significant overall performance jump from the initial AR model to the full NFIG demonstrates the combined power of our novel frequency-based components, which operate within the effective VQGAN training paradigm.**\n\nI hope my response can address your concerns. If you have any further questions, please feel free to let me know. Thank you!"}, {"review_id": "dNM6zzKm59", "reviewer": "Reviewer_mpvq", "summary": "This paper introduces Next-Frequency Image Generation (NFIG), a novel autoregressive framework that decomposes image generation into frequency-guided stages, progressing from low to high frequencies. The core idea is that leveraging the natural hierarchical frequency distribution of images, where low frequencies encode global structure and high frequencies contain local details and can improve both generation quality and computational efficiency compared to traditional spatial autoregressive methods.", "questions": "questions:\n1. How much of the performance gain comes from DINO pre-training vs. the frequency-ordering principle? Can you provide results without ImageNet-pretrained components?\n2. Why not use learnable frequency decomposition (smth like learned filterbanks) instead of fixed FFT-based masks?\n3. How does the method perform on images that violate natural image statistics (textures, QR codes, diagrams, medical images)?\n4. What is the computational overhead of FFT operations compared to the claimed efficiency gains?\nMinors:\n- Figure 2 caption: \"Ntok_i = h_iw_i\" formatting is inconsistent\n- Missing related work on wavelet-based image generation", "limitations": "limitations from above can be addressed with following experiments.\n1. **Frequency ordering ablation**: What happens with reversed (high→low) or random frequency ordering?\n2. **Cross-dataset evaluation**: Performance on COCO, LSUN, CelebA-HQ, etc.\n3. **Frequency-aware metrics**: PSD distance, frequency band preservation scores\n4. **Robustness tests**: Performance on images with unusual frequency characteristics\n\nThe work makes a solid contribution but needs major revisions to:\n\n1. Address the evaluation bias through cross-dataset experiments\n2. Add frequency-aware evaluation metrics\n3. Provide statistical significance testing or at least verify it on more than 1 correlated dataset\n4. improve baseline comparasion\n\nWith these improvements, this could be a clear accept. As it stands, it's a borderline paper that introduces interesting ideas but lacks through full validation.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- Novel frequency-based decomposition approach that aligns with natural image statistic and suggestion the hypothesis about the natural structure of images follows a hierarchical frequency distribution.\n- The empirical results are quite strong. Achieving FID 2.81 on ImageNet-256 represents meaningful progress, especially when combined with the high IS score of 332.42. What's particularly impressive is that NFIG achieves this with 310M parameters compared to VAR-d20's 600M, while also providing a 25% inference speedup.\n- The technical contribution is clear and well-executed. The FR-VAE design elegantly combines frequency decomposition with residual quantization in a way that makes intuitive sense. The progressive accumulation of frequency bands during generation is a clever way to ensure global coherence while adding fine details.\n- The ablation study in Table 3 is thorough and revealing. It clearly shows that both the frequency-guided tokenizer and the generation strategy components contribute meaningfully to the final performance. The progression from 18.65 to 2.81 gFID through systematic component addition tells a compelling story.\n- The paper is well-written with effective visualizations. Figure 2 clearly illustrates the frequency decomposition process, and Figure 4's spectrograms provide good intuition for how the progressive generation works in practice.\n\n### Weakness:\n\n- My biggest concern is the evaluation setup. The authors use DINO features, which were pre-trained on ImageNet, for both encoder initialization and as a discriminator, then evaluate on ImageNet. This creates a circular dependency that could significantly inflate the reported performance. It's unclear whether the improvements come from the frequency-ordering principle or simply from better leveraging ImageNet-specific features through DINO.\n- The evaluation is surprisingly limited for a paper claiming a fundamental advance in image generation. Testing only on ImageNet-256 leaves many questions unanswered about generalization. Additionally, relying solely on FID and IS metrics feels outdated when more perceptually-aligned metrics exist. Most critically, there are no frequency-domain evaluation metrics, which seems like a missed opportunity given the paper's focus.\n- The frequency band division strategy appears somewhat arbitrary. Equation 7 uses a simple scale-based partitioning that doesn't seem grounded in signal processing principles. Why these specific scaling factors? Why not use information-theoretic criteria or learn the decomposition? This feels like a key design choice that lacks proper justification.\n- The paper lacks theoretical depth in crucial areas. While the intuition about frequency ordering is appealing, there's no formal analysis of why this should be superior to spatial ordering. The relationship between frequency content and token allocation is stated but not rigorously justified. Some theoretical guarantees about information preservation through the decomposition would strengthen the claims.\n- The scope feels narrow given the ambitious claims. Without testing on diverse image types (synthetic patterns or perhaps medical images - you can start at least with LSUN), it's hard to know if this works beyond natural images. The comparison with VAR also seems somewhat unfair given the parameter count difference. Perhaps most importantly, FAR is mentioned as doing frequency-based generation but isn't compared against, which feels like a significant omission", "was_revised": true, "final_justification": "After addressed questions and some mentioned limitations that authors address during rebuttal, I want to raise the score. Primary reason why I can't set \"accept\" is because I don't understand would author add new results (super important from my point of view) to the final manuscript to make all the aspects clear. And because originally it was a lot of concerns there from my side, meanwhile I'm happy to accept paper, given all the disclosed results in author response and originally strong formulation of the method.", "rebuttal": "### Rebuttal by Author\nWe sincerely appreciate your detailed and constructive comments. Our point-by-point responses are provided here with:\n\n>**W1&Q1**: The concerns about the pretrained-model DINO, which is also trained on Imagenet. The gain can be from the DINO model rather than frequency-guided part. Can you provide results without ImageNet-pretrained components?\n\n**Reply**: We understand your concern regarding the potential influence of ImageNet-pretrained components on our results, specifically that the performance gains might stem from the DINO model rather than our frequency-guided approach. Due to time constraints, we are unable to retrain NFIG from scratch on 1.28 million images without ImageNet-pretrained components. However, it's important to note that our baselines, such as **VAR, also utilize the ImageNet-pretrained DINO model in their training process**. Therefore, when comparing NFIG to VAR, the observed **performance improvement is primarily attributable to our novel frequency-guided generation strategy**. This strategy consistently enhances performance, regardless of the underlying feature extractor.\n\n>**W2&E2&L3**: Evaluation on ImageNet-256 leaves questions unanswered about generalization. FID and IS are outdated. Besides, no frequency-domain evaluation metrics.\n\n**Reply**: We appreciate your valuable suggestions.\n\n**Dataset Generalization**: We acknowledge that evaluating on diverse datasets could strengthen our generalization claims. However, **ImageNet is highly representative for conditional image generation**, serving as the most common challenging dataset in this field[1]. Its 1000 distinct categories already offer a robust assessment of our model's generalization capabilities.\n\n**Evaluation Metrics: FID and IS remain standard benchmarks[1,2]** enabling direct comparison with established baselines like VAR. However, we acknowledge the value of frequency-domain metrics for showcasing our method's advantages.\n\n**Frequency-Domain Evaluation**: We conducted additional experiments using:\n\nPower Spectral Density (PSD) for overall frequency fidelity\nFrequency Keep Score (FKS): weighted similarity metric across High/Mid/Low frequency bands, with perceptually-motivated weights of 0.15, 0.28, and 0.57 respectively (emphasizing structural low-frequency information).\n\n| Model     |  PSD↓ | FKS↑ |Low↑  |Middle↑| High↑ |\n|:---|:---- |:-----:|:-----:|:-----:|:-----:|\n| VAR-16    | 0.87  |79.5% | 98.3%|57.6% | 48.2%| \n| NFIG(ours)| 0.47  |87.6% | 98.9%|75.3% | 66.7%| \n\n[1] Tian, Keyu, et al. \"Visual autoregressive modeling: Scalable image generation via next-scale prediction.\" NeuriPS, 2024.\n\n[2] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" CVPR 2022.\n\n>**W3**: The frequency band division strategy appears somewhat arbitrary. \n\n**Reply**: **Our frequency band division strategy is grounded in information theory principles**.  Following Shannon's First Theorem, different resolution feature maps have varying information capacities (fewer bits = less encodable information). These insights led to Equation 7's balanced partition strategy, which empirical results validate as effective. **Besides, we initially explored the 1/f decay in frequency band division**, but found that encoded features don't follow this decay pattern. As a result, most bands became uninformative (black/white outputs) except for the last three scales. \n\n>**W4**: The paper lacks theoretical depth in crucial areas. \n\n**Reply**: Thanks for your suggestions. Here are the theoretical analysis of NFIG:\n\n**Information-Theoretic Foundation**: According to Shannon's information theory, images at different resolutions carry different amounts of information. Based on information content calculations, low-frequency components contain less information while high-frequency components carry more. This provides the theoretical basis for our frequency ordering - we process components in order of increasing information complexity, which is theoretically optimal.\n\n**Error Propagation Analysis**: Spatial resolution methods like VAR force low-resolution images to directly learn complete image information, inevitably causing substantial initial errors. Residual learning cannot necessarily resolve such errors in subsequent propagation. As shown in Figure 5, VAR's quantization loss values are significantly higher than NFIG's, empirically validating this theoretical concern.\n\n>**W5&Q3&L4**: Testing NFIG on diverse image types (textures, QR codes, diagrams, medical images). The comparison with VAR also seems somewhat unfair given the parameter count difference. FAR is mentioned as doing frequency-based generation but isn't compared against. \n\n**Reply**: Thank you for raising these important concerns regarding the scope, comparisons, and missing baselines. We address them point-by-point below.\n\n**Scope and Diverse Image Types:** Our primary claim focuses on NFIG's effectiveness for natural images, with ImageNet as the most representative benchmark. **To demonstrate broader applicability, we conducted preliminary reconstruction evaluations of FR-VAE across diverse image types.**\n\n| Model     | DTD  | QRCODE | Diagrams| Chest-X |\n|:----------|:----:| :------:|:------:  |:------:  |\n| FR-VAE| 6.86 | 11.01  |   21    |  0.74   |\n\n-**Fairness of Comparison with VAR**: To clarify parameter fairness, the VAR performance (FID=2.95) refers to VAR-20 (600M parameters). **When compared to VAR-16 of similar size (310M), NFIG achieves FID=2.81 vs VAR-16's FID=3.55,** demonstrating clear superiority at comparable model scales.\n\n**Comparison with FAR**: NFIG and FAR are concurrent works with distinct approaches (explicit multi-scale FFT-based masks vs. continuous visual tokens). **Our direct comparison shows NFIG achieves superior efficiency.**\n\n| Model     | FID↓ | IS↑ | Pre↑ | Rec↑ | Params|\n|:----------|:----:|-----:|-----:|-----:|------:|\n| FAR-B     | 4.26 |248.9| 0.79  |0.51 | 208M |\n| FAR-L     | 3.45 |282.2| 0.80  |0.54 | 427M |\n| FAR-H     | 3.21 |300.6| 0.81  |0.55 | 812M |\n| NFIG(ours)| 2.81 |332.4| 0.77  |0.59 | 310M |\n\n>**Q2**: Why not use learnable frequency decomposition (smth like learned filterbanks) instead of fixed FFT-based masks?\n\n**Reply**: A key motivation of our work is to **add controllable mathematical priors into deep learning, enhance the interpretability of NFIG**. We specifically chose FFT-based masks over learnable filters because they provide explicit and interpretable frequency decomposition based on well-established signal processing principles. While learnable filters might achieve similar performance, they would sacrifice the interpretability and theoretical guarantees that our physics-based approach provides.\n\n>**Q4**: What is the computational overhead of FFT operations compared to the claimed efficiency gains? \n\n**Reply**: Thank you for this pertinent question. The computational overhead of FFT operations is **negligible (100μs per sample, representing only 0.167% of the total 0.06s inference time)**. This minimal cost is achieved because FFT operations are performed on downsampled feature maps (e.g., 16×16 for 256×256 input images) rather than full-resolution images.\n\n**More importantly, our primary efficiency gains stem from architectural effectiveness rather than computational tricks.** NFIG achieves superior performance with significantly fewer parameters (310M) compared to larger baselines (e.g., VAR-20 with 600M parameters). This fundamental efficiency translates to **faster inference speeds and reduced computational requirements during deployment.**\n\nIn essence, the FFT overhead is quickly amortized by the substantial gains from using a more compact and performant architecture, making the net computational impact highly favorable.\n\n\n>**L1**: Frequency ordering ablation: What happens with reversed (high→low) or random frequency ordering?\n\n**Reply**: Thank you for raising this insightful question regarding frequency ordering. \nOur preliminary experiments with reversed (high→low) and random frequency ordering consistently resulted in significantly degraded performance. Both approaches **failed to achieve reasonable reconstruction loss, leading to noticeably blurry image reconstructions**.\n\n>**L2&E1&E3**: Cross-dataset evaluation: Performance on COCO, LSUN, CelebA-HQ, etc. Provide statistical significance testing or at least verify it on more than 1 correlated dataset.\n\n**Reply**: Thank you for this suggestion. **While cross-dataset evaluation is valuable, it is uncommon in conditional image generation tasks**, where models are typically evaluated within the same domain[1]. ImageNet provides a comprehensive and diverse benchmark that effectively demonstrates NFIG's capabilities for conditional natural image generation.\n\nTo demonstrate the generalization capability of NFIG, we evaluate FR-VAE's reconstruction performance on these datasets, showcasing its zero-shot learning ability:\n\n| Model       |CelebA-HQ| COCO  |LSUN-Bedroom|\n|:----------  |:----:   |:-----: |:-----:|\n| FR-VAE      | 3.51    |7.51   | 6.12 |\n\nAs to the statistical significance testing, FID and others metrics are computed by 50,000 images, we have changed the random seed and repeat 5 times. FIDs are 2.810, 2.810, 2.811, 2.810, 2.811(others are similar). Thus, the conclusion is already significant.\n\n[1] Li, Xiang, et al. \"Xq-gan: An open-source image tokenization framework for autoregressive generation.\" arXiv preprint arXiv:2412.01762 (2024).\n\n>**E4**: improve baseline comparasion.\n\n**Reply**: We appreciate this feedback and will enhance our baseline comparisons by including additional baselines FAR and incorporating frequency-domain metrics for more comprehensive evaluation. \n\nI hope my response can address your concerns. If you have any further questions, please feel free to let me know. Thank you!"}, {"review_id": "xoeDbt1c82", "reviewer": "Reviewer_vKnk", "summary": "This paper proposes NFIG, a novel autoregressive framework that decomposes image generation into frequency-guided stages, first generating low-frequency components to capture global structure and then adding high-frequency details progressively. The approach uses a Frequency-guided Residual-quantized VAE to separate frequency components, enabling efficient representation and generation. Experiments on ImageNet-256 show NFIG achieves an FID of 2.81, outperforming VAR.", "questions": "See Weaknesses", "limitations": "See Weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. I like the idea of this paper, which is very interesting.\n2. The problem of efficient image generation is important.\n3. Image generation from the perspective of the frequency domain is natural.\n\nWeaknesses:\n\n1. While NFIG outperforms VAR-d20, the improvement in key metrics (e.g., FID from 2.95 to 2.81) is modest. Given that VAR already uses a multi-scale strategy, the incremental gain raises questions about whether the frequency-domain paradigm provides a transformative advance rather than incremental tuning.\n\n2. The paper claims to generate images by progressively adding frequency bands, but it does not sufficiently justify why each incremental frequency band is necessary. For instance, it is unclear whether intermediate frequency steps contribute meaningfully to the final quality or if they could be simplified without performance loss, weakening the case for the \"progressive frequency\" mechanism.\n\n3. Beyond marginal FID and speed improvements, the paper fails to articulate unique strengths of the frequency-domain approach. For example, it does not demonstrate whether NFIG better captures specific image characteristics (e.g., texture consistency, global structure coherence) compared to spatial or scale-based autoregressive models. Qualitative analyses (e.g., failure cases) are absent, limiting understanding of when and why frequency guidance matters.\n\n4. The frequency bands are divided based on token scale, which the authors acknowledge as a limitation. This heuristic approach may not optimally capture the statistical properties of natural image spectra (e.g., 1/f decay). A more principled division strategy (e.g., based on energy distribution) is needed to validate the framework’s robustness.", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns.", "rebuttal": "### Rebuttal by Author\nWe deeply appreciate the detailed and valuable feedback provided. Our responses addressing each point are outlined as follows:\n\n>**W1**: While NFIG outperforms VAR-d20, the improvement in key metrics (e.g., FID from 2.95 to 2.81) is modest. Given that VAR already uses a multi-scale strategy, the incremental gain raises questions about whether the frequency-domain paradigm provides a transformative advance rather than incremental tuning.\n\n**Reply**: Thank you for this important point. The comparison cited is against VAR-d20 (600M parameters, FID 2.95).  **Our NFIG (310M parameters) achieves FID 2.81, significantly outperforming the comparable-sized VAR-d16 (310M parameters, FID 3.55).**\n\nThis demonstrates that NFIG provides substantial improvements in parameter efficiency - achieving better performance with half the model size indicates a transformative architectural advance rather than incremental tuning.\n\n| Model     | FID↓ | IS↑  | Pre↑ | Rec↑ | Params|\n|:----------|:----:|-----:|-----:|-----:|------:|\n| VAR-d16   | 3.55 |274.4| 0.84 |0.51 | 310M |\n| VAR-d20   | 2.95 |302.6| 0.83 |0.56 | 600M |\n| NFIG(ours)| 2.81 |332.4| 0.77 |0.59 | 310M |\n\n>**W2**: The paper claims to generate images by progressively adding frequency bands, but it does not sufficiently justify why each incremental frequency band is necessary. For instance, it is unclear whether intermediate frequency steps contribute meaningfully to the final quality or if they could be simplified without performance loss, weakening the case for the \"progressive frequency\" mechanism.\n\n **Reply**: Thank you for this important question about the necessity of each frequency band in our progressive generation mechanism. Our approach is built on the understanding that **each frequency band contributes distinct and indispensable information to image synthesis**:\n\n - **Low frequencies**: These bands contain the **fundamental structural information** and dominant color compositions of an image. Without part of them, the generated image's overall structure would collapse easily, lacking any coherent form or global consistency.\n\n - **Mid frequencies**: Encode texture information and intermediate details. the **loss of mid-frequencies leads to noticeable visual quality degradation**, producing overly smooth, unrealistic images.\n\n - **High frequencies**: Determine fine details and sharpness. They are **essential for high-resolution generation** and realistic texture rendering.\n\n Unlike methods that progressively increase spatial resolution, which might allow for skipping intermediate steps, **frequency bands are complementary and interdependent**. Each band captures orthogonal information that cannot be fully compensated for by others. The structural layout (low), texture patterns (mid), and fine details (high) represent fundamentally different aspects of visual content, all of which are crucial for high-quality image generation.\n\n>**W3**: Beyond marginal FID and speed improvements, the paper fails to articulate unique strengths of the frequency-domain approach. For example, it does not demonstrate whether NFIG better captures specific image characteristics (e.g., texture consistency, global structure coherence) compared to spatial or scale-based autoregressive models. Qualitative analyses (e.g., failure cases) are absent, limiting understanding of when and why frequency guidance matters.\n\n**Reply**: Thank you for your valuable feedback. We agree that it's crucial to demonstrate how NFIG specifically improves image characteristics like texture consistency and global structure coherence, and we acknowledge the current absence of qualitative analyses, including failure cases.  While rebuttal page limits prevent us from providing extensive visual demonstrations or failure cases here, **we are committed to including these in the main paper or the appendix of a future version**. This will allow us to clearly illustrate when and why frequency guidance matters and provide a deeper understanding of NFIG's impact on image quality.\n\n**Quantifying Frequency Fidelity and Preservation**\n\nTo directly address your point about NFIG's unique capabilities, we've focused on quantitatively demonstrating its superior \"frequency keeping ability\" compared to VAR-16. Our evaluation uses two key metrics:\n\n- **Power Spectral Density (PSD)**: This metric assesses the overall fidelity of the frequency distribution in generated images. \n- **Frequency Keep Score (FKS)**: A weighted similarity metric (High: 0.15, Middle: 0.28, Low: 0.57) showing frequency preservation across bands, emphasizing low frequencies for structure and middle for textures.\n\nHere's how NFIG compares to VAR-16:\n\n| Model     |  PSD↓ | FKS↑ |Low↑  |Middle↑| High↑ |Param|\n|:---:|:---- |:-----:|:-----:|:-----:|:-----:|:-----:|\n| VAR-16    | 0.87  |79.5% | 98.3%|57.6% | 48.2%| 310M|\n| NFIG(ours)| 0.47  |87.6% | 98.9%|75.3% | 66.7%| 310M|\n\nAs the results clearly show, NFIG significantly outperforms VAR-16 in maintaining frequency fidelity:\n- **Low Frequency (98.9% vs. 98.3%)**: NFIG maintains excellent preservation of low-frequency components, which are crucial for global structural coherence.\n- **Middle Frequency (75.3% vs. 57.6%)**: NFIG shows a substantial improvement in the middle-frequency band, which directly translates to better texture consistency and richer intermediate details.\n- **High Frequency (66.7% vs. 48.2%)**: NFIG is significantly better at preserving high frequencies, which are essential for fine details and sharpness, elements often lost during encoding and quantization in other models.\n\nThese results quantitatively confirm that NFIG's frequency-guided approach effectively improves the model's ability to preserve essential frequency information, leading to better global structure coherence and texture consistency in generated images.\n\n\n>**W4**: The frequency bands are divided based on token scale, which the authors acknowledge as a limitation. This heuristic approach may not optimally capture the statistical properties of natural image spectra (e.g., 1/f decay). A more principled division strategy (e.g., based on energy distribution) is needed to validate the framework’s robustness.\n\n**Reply**: Thank you for this excellent point. We acknowledge that our token-scale-based division is heuristic and agree that principled frequency division is important for framework robustness. \n\nWe actually explored 1/f decay-based division in preliminary experiments but encountered a fundamental challenge:  **applying strict 1/f principles to encoded feature maps (rather than raw images) creates severely imbalanced bandwidth allocation**. Specifically, high-frequency bands would encompass ~87.5% of total bandwidth, while the initial four scales contained predominantly low-frequency information that became nearly monochrome and uninformative for training.\n\nWe recognize this as an important area for future work. Developing a principled frequency division strategy that accounts for the unique properties of encoded feature maps, rather than just raw image statistics, is a fascinating challenge we're keen to explore further.\n\nI hope my response can address your concerns. If you have any further questions, please feel free to let me know. Thank you!"}]

Author rebuttal:

Meta-review: AC has three main concerns about this paper after reading the rebuttal and the reviewers' thoughts:

1. Epoch to epoch, NFIG does not seem to be much better than VAR.

2. Model scalability issue seems to be unresolved even after the rebuttal. Authors explained that they lack the time and compute to verify that, it still leaves a bit of doubt in the approach.

3. It is unclear about the contributions of DiNO. Authors explained that they don't have the time and compute to verify that NFIG is not just benefitting from the contribution of DiNO, but note that VAR also uses DiNO features. That is reasonable but actually proving that the frequency addition is the main contributor goes a long way.

4. NFIG has only been tested on ImageNet. However, AC knew that this is quite common among image generation paper.

On the other hand, AC likes the idea of generation based on frequency which captures global and local features. In fact, wan2.2, even though if for video generation, has good performance following a similar approach.

Reviewers also pointed out the main strengths of the paper as follow: (1) This work is the first to utilize spectral information to enhance autoregressive image generation through NFIG. It also introduces a Frequency-guided Residual-quantized VAE as an image tokenizer; (2) Experimental results demonstrate superior performance over the strong baseline VAR while maintaining comparable efficiency at the same model scale

In view of the above, AC weighs the pros and cons and decides to recommend the paper for publication at neurips.
Decision: Accept (poster)