Title: ScaleDiff: Higher-Resolution Image Synthesis via Efficient and Model-Agnostic Diffusion
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion

High Resolution Image Synthesis
Abstract: Text-to-image diffusion models often exhibit degraded performance when generating images beyond their training resolution.
Recent training-free methods can mitigate this limitation, but they often require substantial computation or are incompatible with recent Diffusion Transformer models.
In this paper, we propose ScaleDiff, a model-agnostic and highly efficient framework for extending the resolution of pretrained diffusion models without any additional training.
A core component of our framework is Neighborhood Patch Attention (NPA), an efficient mechanism that reduces computational redundancy in the self-attention layer with non-overlapping patches.
We integrate NPA into an SDEdit pipeline and introduce Latent Frequency Mixing (LFM) to better generate fine details.
Furthermore, we apply Structure Guidance to enhance global structure during the denoising process.
Experimental results demonstrate that ScaleDiff achieves state-of-the-art performance among training-free methods in terms of both image quality and inference speed on both U-Net and Diffusion Transformer architectures.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces **ScaleDiff**, a training-free, model-agnostic framework for extending the resolution of pretrained diffusion models beyond their original training size. \n\nThe key component, **Neighborhood Patch Attention (NPA)**, efficiently reduces redundancy in self-attention by using non-overlapping patches, making it suitable for both U-Net and Diffusion Transformer architectures.\n\n Combined with **Latent Frequency Mixing (LFM)** for fine-detail generation and **Structure Guidance** for improving global structure, ScaleDiff achieves state-of-the-art image quality and inference speed among training-free upsampling methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**  \n- The paper is complete and generally clear for a broad readership.  \n- The method does not require costly retraining, making it more practical.  \n- Experimental results indicate that the approach is promising.\n\n**Weaknesses**  \n- The claim of state-of-the-art inference speed and performance is not fully supported. In Table 2, performance improvements over competing methods are marginal or absent, and in some cases, ScaleDiff performs worse than FLUX+GAN. Additionally, the reported inference speed is significantly slower than FLUX+GAN, which contradicts the stated contributions.  \n- The notation for `Z_rgb` is confusing. It is unclear whether this denotes a latent representation or pixel-level RGB data. Since the main text consistently treats `Z_` variables as latents, this inconsistency reduces clarity.  \n- The motivation behind constructing `Z_ref` by adding `Z_high` and `Z_low` is unclear. Wouldn’t using the actual RGB image as a reference latent provide a clearer supervisory signal than this manipulated combination?  \n- There is a large discrepancy in the `tau` parameter between the FLUX backbone (`tau=0.6`) and SDXL (`tau=400`). Since timestep values are typically integers, the reason for such a wide gap should be explained.  \n- The generated images in Figure 4 appear random, making it difficult to visually assess the superiority of the proposed method.  \n- Minor issue: there is a typo “SclaeDiff” in line 92.\n\nQUESTIONS:\n### Questions and Suggestions for the Authors\n\n**Q1: Clarify and support the claims on inference speed and performance.**  \n- Table 2 shows only marginal performance gains or even worse results compared to FLUX+GAN, and the inference speed is notably slower.  \n- Could the authors provide a more detailed and transparent comparison, including runtime benchmarks under consistent conditions?  \n- Please clarify or temper the claims regarding state-of-the-art speed and performance if the evidence does not fully support them.\n\n**Q2: Explain the notation and role of `Z_rgb` clearly.**  \n- The notation `Z_rgb` is ambiguous—is it a latent representation or raw RGB pixel data?  \n- Given that other `Z_` variables denote latent features, please clarify this inconsistency to avoid reader confusion.  \n- Consider adding a dedicated explanation or diagram showing how `Z_rgb` differs from or relates to `Z_latent`.\n\n**Q3: Justify the construction of the reference latent `Z_ref` as the sum of `Z_high` and `Z_low`.**  \n- What is the motivation behind this design choice?  \n- Would using the actual RGB image as the reference latent provide a clearer or stronger supervisory signal?  \n- Please discuss alternatives and provide ablation studies or analyses demonstrating why this approach is preferable.\n\n**Q4: Explain the large difference in timestep parameter `tau` between backbones.**  \n- Why is `tau=0.6` used for FLUX while `tau=400` is used for SDXL?  \n- Since timesteps are typically integers, please clarify what these values represent and why such a large gap is necessary.  \n- Providing more details on how `tau` is chosen and tuned for different backbones would improve reproducibility.\n\n**Q5: Improve qualitative results and visual assessments.**  \n- The generated images in Figure 4 appear random, making it difficult to visually assess improvements.  \n- Could the authors provide additional visual comparisons with controlled conditions or include quantitative perceptual metrics to better demonstrate quality differences?\n\nLIMITATIONS:\nNo, they didn't discuss.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on extending existing diffusion models for higher-resolution image generation in a training-free manner. Specifically, the authors propose ScaleDiff, a novel approach capable of scaling up the generation resolution of diffusion models based on both U-Net and DiT. The framework incorporates two key components: Neighborhood Patch Attention (NPA) and Latent Frequency Mixing (LFM). Experimental results demonstrate that the proposed ScaleDiff efficiently achieves higher-resolution image generation while exhibiting better performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. **Writing and Presentation**: The paper is clear, well-written, and easy to follow.\n2. **Concise and Effective Design**: The proposed ScaleDiff is a training-free approach that can be broadly applied to existing diffusion models based on U-Net and DiT.\n3. **Outstanding Performance**: Experimental results demonstrate that ScaleDiff can efficiently generate higher-resolution images of high quality, illustrating its effectiveness and efficiency.\n4. **Necessary Ablation Studies**: The paper includes essential ablation studies to validate the effectiveness of the proposed key modules, making the results more convincing.\n5. **Comprehensive Visualization**: The authors also provide additional qualitative results at various resolution scales in the supplementary material, further confirming the method's strong scalability and promising application potential.\n\n**Weaknesses**:\n\n1. **Limited Evaluation Scope**: The current quantitative evaluation is conducted on 1,000 samples from LAION, which is somewhat limited and may introduce performance variance. Thus, a larger-scale evaluation testbed is expected to make the results more robust and convincing.\n2. **More Comprehensive Evaluation and Metrics**: While the authors provide synthesis results at other scales, quantitative results (similar to ScaleCrafter) at extreme aspect ratios (e.g., 4:1 and 1:4) could further highlight ScaleDiff’s universality. Moreover, beyond FID and patch-based FID, metrics like KID and patch-based KID are also widely adopted for evaluating higher-resolution image generation performance.\n3. **Figures and Explanation**: I recommend refining Figure 3 to reduce blank space and improve layout efficiency. Additionally, it would be helpful to explicitly illustrate how low-frequency and high-frequency components are derived in the diagram (readers might intuitively assume bandpass filtering, while the method actually uses downsampling+upsampling). Furthermore, the hypothesis in Line 189 is confusing: why do imperfect images generated via latent-space upsampling provide high-quality high-frequency components? A deeper explanation, either theoretical or experimental, would help readers better understand this subsection.\n4. **More Key Baselines**: The paper would benefit from including comparisons with more baseline approaches. For instance, replacing BSRGAN with StableSR [1*] or ESRGAN [2*], as well as conventional approaches like Bilinear/Bicubic interpolation, is expected to better demonstrate the advantages of the proposed ScalDiff.\n5. **More Related Work to be discussed**: The paper would benefit from a more thorough discussion and comparison with recent advances in high-resolution image/video generation, such as URAE [3*], MegaFusion [4*], BeyondScene [5*], ResAdapter [6*], FreeScale [7*], and LinFusion [8*]. (especially those post-processing and tuning-free ones)\n\n[1*] StableSR: Exploiting Diffusion Prior for Real-World Image Super-Resolution, in IJCV 2024.\n\n[2*] ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks, in ECCV workshop 2018.\n\n[3*] Ultra-Resolution Adaptation with Ease, in ICML 2025.\n\n[4*] MegaFusion: Extend Diffusion Models towards Higher-resolution Image Generation without Further Tuning, in WACV 2025.\n\n[5*] BeyondScene: Higher-Resolution Human-Centric Scene Generation With Pretrained Diffusion, in ECCV 2024.\n\n[6*] ResAdapter: Domain Consistent Resolution Adapter for Diffusion Models, in AAAI 2025.\n\n[7*] FreeScale: Unleashing the Resolution of Diffusion Models via Tuning-Free Scale Fusion, in arXiv 2024.\n\n[8*] LinFusion: 1 GPU, 1 Minute, 16K Image, in arXiv 2024.\n\nQUESTIONS:\nPlease refer to the 'Weaknesses' section for details.\n\nMy primary unresolved concerns focus on the following aspects:\n1. **More comprehensive and convincing evaluations**.\n2. **Further polish on figures and presentations**.\n3. **Some essential baseline methods**.\n4. **More related literature expected to be discussed and compared**.\n\nSince the overall quality of the paper is relatively impressive, I think I will be glad to raise my rating if the authors can address my concerns.\n\nLIMITATIONS:\nYes, the authors make an attempt to solve the existing limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

comments: SUMMARY:
This paper presents ScaleDiff, a training-free, resolution-agnostic framework for extending the generation resolution of pretrained text-to-image diffusion models, including both U-Net and Diffusion Transformer architectures. The method introduces Neighborhood Patch Attention (NPA) to reduce redundancy in self-attention over non-overlapping patches, significantly improving efficiency over standard patch-based approaches. Additionally, Latent Frequency Mixing (LFM) enhances fine detail synthesis, while Structure Guidance improves global coherence during denoising. Integrated into an upsample-diffuse-denoise pipeline, ScaleDiff achieves state-of-the-art performance in image quality and inference speed compared to other training-free high-resolution generation methods.

STRENGTHS AND WEAKNESSES:
**Strength**

+ Works with both U-Net and Diffusion Transformer backbones, broadening applicability beyond methods tailored only for U-Nets.

+ The proposed Neighborhood Patch Attention (NPA) effectively reduces redundant computations while preserving receptive field coverage, improving speed and scalability.

+ Significant improvements in FID and CLIP scores on multiple datasets and backbones. Visuals show noticeably better high-frequency details.

**Weaknesses**

+ The porposed techniques (NPA, LFM, SG), while efficient in theory, may require non-trivial engineering to integrate into existing pipelines as they looks quite complex to implement, especially NPA.

+ Lack of discussion about failure cases.

QUESTIONS:
+ Include some analysis or visual examples of where ScaleDiff fails or degrades (e.g., extreme compositions, long prompts, or cluttered scenes). How does the system behave under such inputs?

LIMITATIONS:
No other limitations
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo address the degradation in image quality when existing open-source text-to-image models generate outputs beyond their training resolution, this paper proposes ScaleDiff, a training-free framework composed of three components: NPA, LFM, and SG. Specifically, NPA eliminates the additional computational overhead introduced by previous patch-based methods, LFM mitigates detail loss caused by upsampling in pixel space, and SG enhances global consistency in the generated images. Qualitative and quantitative experiments validate the effectiveness of the method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-written with a clear motivation, making it easy to understand.\n2. The proposed NPA is interesting; it is a training-free and model-agnostic attention mechanism that allows effective aggregation of contextual information for each token without introducing additional computational overhead.\n3. The proposed pipeline demonstrates its effectiveness by outperforming several previous approaches when applied to the pretrained SDXL model.\n\nWeaknesses:\n1. The paper claims that previous methods are not applicable to diffusion models based on the DiT architecture, but it does not provide adequate qualitative or quantitative evidence to support this claim. I noticed that Figure 1 displays a qualitative comparison of MultiDiffusion applied to DiT, yet MultiDiffusion is not a recent state-of-the-art method.\n2. While the paper includes comparisons with earlier methods, it would benefit from evaluations against more recent models such as AccDiffusion v2, FreeScale, AP-LDM, and MegaFusion.\n3. Although the proposed method is effective, it lacks in-depth explanations of certain techniques. For instance, why does interpolating high-frequency information in the latent space with LFM help preserve details?\n4. Line 46 of the paper mentions that upsampling in RGB space may lead to over-smoothing, which is indeed a valid concern. However, to the best of my knowledge, most existing methods that perform RGB-space upsampling typically apply a diffusion model for refinement afterward. Therefore, I am concerned that the issue raised by the paper may not be particularly significant.\n5. I noticed that using SG alone may have a negative impact on the model’s output, both in qualitative and quantitative experiments. This raises concerns that the generalizability of SG might be limited.\n6. According to previous studies [1], directly changing the sequence length of the attention inputs in a training-free setting can lead to degraded attention performance. How does NPA address the risk of such degradation?\n\n[1] Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis. NIPS2023.\n\nQUESTIONS:\nI will consider increasing the score once all concerns are addressed.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for extending the resolution of the output images from pretrained diffusion models without additional training. For this, the authors introduce Neighborhood Patch Attention. This helps in reducing computational redundancy in the self-attention layer with non-overlapping patches. They also introduce Latent Frequency Mixing for generating fine-grained details in the output images. Additionally, global consistency is enforced through Structure Guidance, applied in the latent space.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. The ideas of avoiding patch-based processing for non-self-attention layers and extracting queries from non-overlapping patches to reduce computation, and latent frequency mixing for image quality enhancement are interesting. \n2. The proposed method can be integrated with pretrained diffusion models without additional training.\n3. The proposed method shows promising results in terms of processing time and image quality.\n\n\nWeakness\n1. In the context of upscaling pipeline, the process of computing the high-frequency components is ambiguous. How is the latent downsampled and upsampled in this context?\n2.  The shifted crop sampling strategy is not clearly explained. The authors have also not discussed whether there is a possibility of information loss due to shifted crop sampling. If so, how does the proposed method deal with this potential problem?\n3. The authors hypothesize that upscaling the latent in the latent space can mitigate oversmoothing. However, the basis of this hypothesis is not described in the paper. \n4. The process of creating $Z_{latent}$ is not clear to me. Does it consist of components from both the RGB space and the latent space? If so, how are these two components blended? What is the motivation behind it?\n\nQUESTIONS:\n1. In the context of upscaling pipeline, how is the latent downsampled and upsampled? Please provide a detailed description of how to compute the high-frequency components.\n2.  The shifted crop sampling strategy should be clearly explained. \n3. The authors need to discuss whether there is a possibility of information loss due to shifted crop sampling. If so, how does the proposed method deal with this?\n3. The authors hypothesize that upscaling the latent in the latent space can mitigate oversmoothing. The basis of this hypothesis needs to be explained. \n4. The process of creating $Z_{latent}$ and the motivation of using it should be explained.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}