Title: NFIG: Multi-Scale Autoregressive Image Generation via Frequency Ordering
Venue: NeurIPS 2025
Accepted: yes
Keywords: Image Generation

Auto-regressive model

Frequency analysis
Abstract: Autoregressive models have achieved significant success in image generation. However, unlike the inherent hierarchical structure of image information in the spectral domain, standard autoregressive methods typically generate pixels sequentially in a fixed spatial order. To better leverage this spectral hierarchy, we introduce Next-Frequency Image Generation (NFIG). NFIG is a novel framework that decomposes the image generation process into multiple frequency-guided stages. NFIG aligns the generation process with the natural image structure. It does this by first generating low-frequency components, which efficiently capture global structure with significantly fewer tokens, and then progressively adding higher-frequency details. This frequency-aware paradigm offers substantial advantages: it not only improves the quality of generated images but crucially reduces inference cost by efficiently establishing global structure early on. Extensive experiments on the ImageNet-256 benchmark validate NFIG's effectiveness, demonstrating superior performance (FID: 2.81) and a notable 1.25x speedup compared to the strong baseline VAR-d20.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes NFIG, a novel autoregressive framework that decomposes image generation into frequency-guided stages, first generating low-frequency components to capture global structure and then adding high-frequency details progressively. The approach uses a Frequency-guided Residual-quantized VAE to separate frequency components, enabling efficient representation and generation. Experiments on ImageNet-256 show NFIG achieves an FID of 2.81, outperforming VAR.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. I like the idea of this paper, which is very interesting.\n2. The problem of efficient image generation is important.\n3. Image generation from the perspective of the frequency domain is natural.\n\nWeaknesses:\n\n1. While NFIG outperforms VAR-d20, the improvement in key metrics (e.g., FID from 2.95 to 2.81) is modest. Given that VAR already uses a multi-scale strategy, the incremental gain raises questions about whether the frequency-domain paradigm provides a transformative advance rather than incremental tuning.\n\n2. The paper claims to generate images by progressively adding frequency bands, but it does not sufficiently justify why each incremental frequency band is necessary. For instance, it is unclear whether intermediate frequency steps contribute meaningfully to the final quality or if they could be simplified without performance loss, weakening the case for the \"progressive frequency\" mechanism.\n\n3. Beyond marginal FID and speed improvements, the paper fails to articulate unique strengths of the frequency-domain approach. For example, it does not demonstrate whether NFIG better captures specific image characteristics (e.g., texture consistency, global structure coherence) compared to spatial or scale-based autoregressive models. Qualitative analyses (e.g., failure cases) are absent, limiting understanding of when and why frequency guidance matters.\n\n4. The frequency bands are divided based on token scale, which the authors acknowledge as a limitation. This heuristic approach may not optimally capture the statistical properties of natural image spectra (e.g., 1/f decay). A more principled division strategy (e.g., based on energy distribution) is needed to validate the framework’s robustness.\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Next-Frequency Image Generation (NFIG), a novel autoregressive framework that decomposes image generation into frequency-guided stages, progressing from low to high frequencies. The core idea is that leveraging the natural hierarchical frequency distribution of images, where low frequencies encode global structure and high frequencies contain local details and can improve both generation quality and computational efficiency compared to traditional spatial autoregressive methods.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- Novel frequency-based decomposition approach that aligns with natural image statistic and suggestion the hypothesis about the natural structure of images follows a hierarchical frequency distribution.\n- The empirical results are quite strong. Achieving FID 2.81 on ImageNet-256 represents meaningful progress, especially when combined with the high IS score of 332.42. What's particularly impressive is that NFIG achieves this with 310M parameters compared to VAR-d20's 600M, while also providing a 25% inference speedup.\n- The technical contribution is clear and well-executed. The FR-VAE design elegantly combines frequency decomposition with residual quantization in a way that makes intuitive sense. The progressive accumulation of frequency bands during generation is a clever way to ensure global coherence while adding fine details.\n- The ablation study in Table 3 is thorough and revealing. It clearly shows that both the frequency-guided tokenizer and the generation strategy components contribute meaningfully to the final performance. The progression from 18.65 to 2.81 gFID through systematic component addition tells a compelling story.\n- The paper is well-written with effective visualizations. Figure 2 clearly illustrates the frequency decomposition process, and Figure 4's spectrograms provide good intuition for how the progressive generation works in practice.\n\n### Weakness:\n\n- My biggest concern is the evaluation setup. The authors use DINO features, which were pre-trained on ImageNet, for both encoder initialization and as a discriminator, then evaluate on ImageNet. This creates a circular dependency that could significantly inflate the reported performance. It's unclear whether the improvements come from the frequency-ordering principle or simply from better leveraging ImageNet-specific features through DINO.\n- The evaluation is surprisingly limited for a paper claiming a fundamental advance in image generation. Testing only on ImageNet-256 leaves many questions unanswered about generalization. Additionally, relying solely on FID and IS metrics feels outdated when more perceptually-aligned metrics exist. Most critically, there are no frequency-domain evaluation metrics, which seems like a missed opportunity given the paper's focus.\n- The frequency band division strategy appears somewhat arbitrary. Equation 7 uses a simple scale-based partitioning that doesn't seem grounded in signal processing principles. Why these specific scaling factors? Why not use information-theoretic criteria or learn the decomposition? This feels like a key design choice that lacks proper justification.\n- The paper lacks theoretical depth in crucial areas. While the intuition about frequency ordering is appealing, there's no formal analysis of why this should be superior to spatial ordering. The relationship between frequency content and token allocation is stated but not rigorously justified. Some theoretical guarantees about information preservation through the decomposition would strengthen the claims.\n- The scope feels narrow given the ambitious claims. Without testing on diverse image types (synthetic patterns or perhaps medical images - you can start at least with LSUN), it's hard to know if this works beyond natural images. The comparison with VAR also seems somewhat unfair given the parameter count difference. Perhaps most importantly, FAR is mentioned as doing frequency-based generation but isn't compared against, which feels like a significant omission\n\nQUESTIONS:\nquestions:\n1. How much of the performance gain comes from DINO pre-training vs. the frequency-ordering principle? Can you provide results without ImageNet-pretrained components?\n2. Why not use learnable frequency decomposition (smth like learned filterbanks) instead of fixed FFT-based masks?\n3. How does the method perform on images that violate natural image statistics (textures, QR codes, diagrams, medical images)?\n4. What is the computational overhead of FFT operations compared to the claimed efficiency gains?\nMinors:\n- Figure 2 caption: \"Ntok_i = h_iw_i\" formatting is inconsistent\n- Missing related work on wavelet-based image generation\n\nLIMITATIONS:\nlimitations from above can be addressed with following experiments.\n1. **Frequency ordering ablation**: What happens with reversed (high→low) or random frequency ordering?\n2. **Cross-dataset evaluation**: Performance on COCO, LSUN, CelebA-HQ, etc.\n3. **Frequency-aware metrics**: PSD distance, frequency band preservation scores\n4. **Robustness tests**: Performance on images with unusual frequency characteristics\n\nThe work makes a solid contribution but needs major revisions to:\n\n1. Address the evaluation bias through cross-dataset experiments\n2. Add frequency-aware evaluation metrics\n3. Provide statistical significance testing or at least verify it on more than 1 correlated dataset\n4. improve baseline comparasion\n\nWith these improvements, this could be a clear accept. As it stands, it's a borderline paper that introduces interesting ideas but lacks through full validation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces frequency order modeling into AR visual generation. With the observation of the low frequency encodes global structure while the high frequency local details, the authors propose the Next-Frequency Image Generation (NFIG) framework for frequency guided visual generation and a FR-VAE for visual reconstruction. Experiments demonstrate the effectiveness of NFIG.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1) The motivation is clear, and frequency-based AR modeling shows potential in addressing the problem of error accumulation in vanilla AR generation. Encoding the feature into different levels of frequency helps formulate a new generation paradigm and increase inference speed.\n2) The FR-VAE encodes the visual feature into the frequency-based domain, which further improves reconstruction performance.\n3) Next-Frequency Image Prediction adopts the \"Coarse-to-Fine\" generation process that is more suitable for visual generation instead of the vanilla one. The experiment shows promising results.\n\n**Weakness**\n1) It is unclear what the notation v_i signifies. It seems the f_i and v_i denote the same one. This part should be further clarified for a clear illustration.\n2) Can frequency-based AR modeling be leveraged into single-scale prediction, e.g., LlamaGen. Since it can be easily extended to a unified model framework. \n3) Why can the NFIG accelerate the generation process. The total predicted tokens are the same as VAR.\n4) I am worried about the scalability of the NFIG framework, since it is key to further application.\n5) In Table 3, the improvement of incorporating FR-Quantizer is not significant and most of the benefits are brought by the Dino-discriminator.\n\nQUESTIONS:\nThe overall motivation is good, and the design is feasible, but some weaknesses remain. I will raise my score if my concerns can be addressed.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the field of autoregressive image generation, proposing a Next-Frequency Image Generation (NFIG) method that leverages the inherent bias of image information decomposition to achieve a high-performance and efficient framework. The authors claim that NFIG runs 1.25× faster than VAR-d20 while outperforming larger-scale VAR models, achieving a superior FID of **2.81 on ImageNet-256**.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \nThis work is the first to utilize spectral information to guide and enhance autoregressive image generation through the Next-Frequency Image Generation (NFIG) approach. It also introduces a Frequency-guided Residual-quantized VAE as an image tokenizer. Experimental results demonstrate superior performance over the strong baseline VAR while maintaining comparable efficiency at the same model scale. \n\nWeaknesses: \n1.\tUnfair comparison with VAR: The publicly available VAR code shows that VAR-d16 (200 epochs) and VAR-d20 (250 epochs) were trained for fewer epochs than NFIG (350 epochs). This discrepancy raises concerns about whether the improvements stem from methodological innovation or simply longer training, casting doubt on the validity of the claimed advantages. \n\n2.\tLimited model scale exploration: The paper only evaluates a single model size (310M parameters), leaving uncertainty about the **scaling laws**—whether this method remains effective at larger scales is unknown. \n\n3.\tInsufficient methodological details: While the core idea is clearly presented, key implementation details are lacking. For example: \n1.\tThe exact composition of the total loss function and the role of each component are not clearly explained. \n2.\tHow positional information is incorporated or processed for frequency-based tokens is omitted. \n  Although the architecture is described as similar to VAR, critical differences are not thoroughly elaborated. \n\n4.\tEfficiency concerns: Since this work adopts the same generation framework as VAR but introduces additional operations (e.g., Fourier transforms and inverse transforms), **theoretical efficiency should be slightly worse**. The claim of equal efficiency at the same scale is questionable—how exactly was efficiency improved? Furthermore, these extra operations might impose greater disadvantages at higher resolutions. \n\nIn summary, this work introduces a novel perspective for image generation, contributing positively to the field. Thus, if the authors can address the four concerns above clearly, I can increase my score to 4. \n\nAdditional Questions: \n\n5.\tFundamental superiority over VAR?Some blogs, articles, and preliminary experiments suggest that VAR also generates coarse global information in early stages and gradually refines high-frequency details later. Can the authors clarify whether NFIG **truly surpasses VAR in essence**? I suspect that NFIG might simply apply Fourier transforms and inversions to intermediate results during generation, similar to VAR. Could the authors provide theoretical or empirical evidence of its genuine advantages? \n\n6.\tExtension to text-to-image generation. Since the codebook consists of frequency-based tokenized vectors, has the author considered how NFIG could be adapted to **text-to-image generation**? Would aligning these frequency tokens with text introduce additional challenges?\n\nQUESTIONS:\nSame as weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}