Title: RepLDM: Reprogramming Pretrained Latent Diffusion Models for High-Quality, High-Efficiency, High-Resolution Image Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Model

High-Resolution Image

Attention Guidance

progressive pixel space upsampling
Abstract: While latent diffusion models (LDMs), such as Stable Diffusion, are designed for high-resolution image generation, they often struggle with significant structural distortions when generating images at resolutions higher than their training one.

Instead of relying on extensive retraining, a more resource-efficient approach is to reprogram the pretrained model for high-resolution (HR) image generation; however, existing methods often result in poor image quality and long inference time.

We introduce RepLDM, a novel reprogramming framework for pretrained LDMs that enables high-quality, high-efficiency, high-resolution image generation; see Fig. 1. RepLDM consists of two stages: (i) an attention guidance stage, which generates a latent representation of a higher-quality training-resolution image using a novel parameter-free self-attention mechanism to enhance the structural consistency; and (ii) a progressive upsampling stage, which progressively performs upsampling in pixel space to mitigate the severe artifacts caused by latent space upsampling. The effective initialization from the first stage allows for denoising at higher resolutions with significantly fewer steps, improving the efficiency.

Extensive experimental results demonstrate that RepLDM significantly outperforms state-of-the-art methods in both quality and efficiency for HR image generation, underscoring its advantages for real-world applications.
Codes: https://github.com/kmittle/RepLDM.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a training-free image generation framework called RepLDM, which aims to generate high-resolution images while preserving global structural consistency. To achieve this goal, the authors introduce a parameter-free self-attention mechanism (PFSA) and a pixel-space-based Progressive Upsampling Stage. Both quantitative and qualitative comparisons, as well as ablation studies, demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n(1) The paper proposes a plug-and-play, parameter-free self-attention mechanism (PFSA), which elegantly improves the global structural consistency of generated images at the cost of only a small increase in computational overhead. This effectively addresses a long-standing issue in Stable Diffusion when generating ultra-high-resolution images. If the method proves to be effective, it has the potential to become a general paradigm for high-resolution image generation tasks.\n\n(2) The proposed RepLDM framework requires no additional training and features a simple, intuitive pipeline, making it easy to reproduce.\n\nWeaknesses:\n\n(1) As mentioned in Strengths (1), PFSA could be a promising approach if it truly works. However, due to its highly counter-intuitive nature, I find it difficult to judge whether it is genuinely effective. Why would a simple post-processing self-attention operation significantly improve so-called global structural consistency? Although the authors provide some supporting references (e.g., [44, 52]) and ablation studies (Section 5.1), the evidence remains unconvincing. A more detailed analysis of PFSA is needed—such as numerical evaluation or visualizations—to help readers better understand its mechanism. Moreover, the concept of global structural consistency should be more clearly defined.\n\n(2) While the paper emphasizes high-efficiency in its title, the experimental section only compares inference time, which seems rather one-sided. A more comprehensive comparison including parameters, FLOPs, and NFE would provide a fuller picture of the method’s efficiency.\n\nQUESTIONS:\nThe concerns are as described in the Weaknesses section. Due to the uncertainty regarding the effectiveness of PFSA, I am assigning a borderline score for now out of caution. If the authors can adequately address my concerns during the rebuttal phase, I would consider raising my score.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a framework named RepLDM to reprogram pretrained latent diffusion models (LDMs) for generating high-resolution images without requiring additional training. The approach integrates a parameter-free self-attention mechanism to enhance the structural consistency of latent representations at the training resolution and progressive upsampling in pixel space to mitigate artifacts caused by latent space upsampling, improving both image quality and generation efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n1.\tThe idea of reprogramming pretrained LDMs for high-resolution image generation is novel and resource-efficient.\n2.\tUpsampling in pixel space alleviates artifacts and allows for efficient denoising at higher resolutions, significantly reducing inference time.\n3.\tThe paper conducts comprehensive quantitative and qualitative experiments to demonstrate the effectiveness and efficiency of RepLDM compared to state-of-the-art baselines.\nWeaknesses:\n1.\tWhile RepLDM improves efficiency compared to other methods, generating ultra-high resolution images still requires more sub-stages in the progressive upsampling stage, potentially increasing inference time.\n2.\tThe paper does not report error bars or statistical significance tests for the experimental results, making it difficult to assess the robustness of the claims.\n\nQUESTIONS:\n1. PFSA's efficacy appears critically to be dependent on input latent quality.\n(a) How does performance degrade under structured noise?\n(b) Provide robustness analysis using metrics like SSIM drop ≥ 0.2 relative to baseline.\n\n2. The complexity of PFSA's self-attention operation poses significant bottlenecks at high resolutions. Please give the computational load analysis for this issue. \n\n3. For the progressive denoising with pixel space upsampling scheme, \n(a)Does it rely on fixed-size convolutional kernels (e.g., 3×3/5×5) for pixel-space operations? Will it introduce the resolution issues when using different kernels? \n(b) Did you explore integrating blind deblurring modules within the denoising steps?\n\nLIMITATIONS:\nPlease see questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for sampling high-resolution images using pretrained diffusion models. Their method consists of progressively making the image larger as they perform the denoising process. Additionally, to better capture the image details they make use of a parameter free self attention operator that shows promise in improving sample quality.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper provides a clear explanation of the method\n- The paper provides lots of experiments and comparisons \n\nWeaknesses\n\n- Could the authors define HR and TR. I believe I know there meaning but it would be good for the paper\n- The method requires decoding and encoding multiple times, which is not computationally ideal\n\nQUESTIONS:\nMy biggest question is trying to understand what attention guidance is doing. Although the authors provided some discussion on the appendix, it is still not clear to me. For ease of this discussion lets consider a black and white image, therefore we only have one channel. Although I understand that this operation is done in the latent space the concept should be the same.\n\nWhen using attention guidance $$AG(v) = \\sigma(vv^T)v$$ the values of v become close to each other and not the other way around. Therefore when we perform the operation $$v  = \\gamma AG(v) + (1-\\gamma) v = v + \\gamma (AG(v) - v)) $$ the pixels with higher values (more black) become smaller and white values become smaller. Therefore the whole image is converging to a more smoothened gray image. \n\nAnalogously this operation in the latent space pulls the \"latent pixels\" closer to each other, which should reduce the details. Therefore that would suggest that there is something particular about the latent space that AG is using to make this work. Could you elaborate on this? \n\nAdditionally, it would be nice to provide a discussion on why do the numerical results are so similar while the qualitative samples look so different\n\nMy final question is regarding the experiments. It seems that you have limited compute and perhaps this could be a cause of issue, but FID and other metrics like IS have very high variance when using a small amount of samples like $1000$. Additionally it seems like the reference statistics are from the SAM dataset, which are not related to the aesthetics generation that SDXL and these models are trained on, therefore the results are a little hard to interpret. Although the qualitative examples do show lots of promise, it would be good to have this be a more robust metric\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RepLDM, a novel training-free framework for adapting pretrained latent diffusion models (LDMs) for high-resolution image generation. The method tackles the common problem of structural distortion when generating images at resolutions beyond the model's training data. RepLDM works in two stages: (1) an Attention Guidance stage that uses a novel parameter-free self-attention (PFSA) mechanism to generate a structurally-consistent latent at the base training resolution, and (2) a Progressive Upsampling stage that iteratively increases the image resolution in pixel space to avoid artifacts common to latent-space upsampling. By starting the upsampling process with a high-quality latent, the method requires significantly fewer denoising steps at high resolutions, leading to major gains in both image quality and inference speed.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Novelty and Elegance of Core Theory (PFSA): The paper's central contribution, the \\textbf{Parameter-Free Self-Attention (PFSA)} mechanism, is exceptionally elegant. The formulation in \\textbf{Eq. 1} ($PFSA(z) = f^{-1}(\\text{Softmax}(\\frac{f(z)f(z)^{T}}{\\lambda})f(z))$) cleverly strips the standard self-attention mechanism of its learnable projection matrices ($W_{Q}, W_{K}, W_{V}$). This design is based on the strong and insightful premise that the global spatial modeling capability of attention is inherent to its structure, not its learned parameters. This simplification provides a powerful, first-principles approach to enhancing structural consistency in a training-free manner.\n\n- The decomposition of the problem into an \"Attention Guidance\" stage and a \"Progressive Upsampling\" stage is logical and effective. This design smartly separates the task of enforcing global structure (at the native resolution) from the task of adding details at higher resolutions. The framework is clean and addresses two distinct failure modes of naive high-resolution generation.\n\n- The authors' identification of latent-space upsampling as a key source of artifacts is a critical insight. Their pilot study (Fig. 4) provides clear empirical evidence for this hypothesis. The decision to perform upsampling in the pixel domain, as formalized in Equation 4, $\\hat{z}_{0} = \\mathcal{E} \\circ \\mathcal{U} \\circ \\mathcal{D}(z_{0})$, is a simple but highly effective solution that directly mitigates this issue, demonstrating a deep understanding of the VAE's properties and limitations.\n\n- The method delivers on its promises, significantly outperforming strong recent baselines like DemoFusion and AccDiffusion across multiple metrics (FID, IS, CLIP Score) and resolutions. The ~5x speedup in inference time is a massive practical advantage that makes high-resolution generation far more accessible. The user study further corroborates the superior quality of the generated images.\n\nWeaknesses\n- Nuance in \"Parameter-Free\" Claim- While the core PFSA block is indeed parameter-free, the overall attention guidance mechanism introduces several key hyperparameters that require careful tuning. The linear interpolation in Equation 2 depends on the guidance scale $\\gamma$, and the adaptive schedule in Equation 3 introduces a decay factor $\\beta$ and a delay rate $\\eta_{1}$. While these are not learned weights, they represent a set of sensitive parameters that are crucial to the method's success, making the \"parameter-free\" description of the overall guidance slightly imprecise.\n\n- As the authors rightly point out in Section 6, the method is a \"reprogramming\" framework, not a fine-tuning one. As such, it cannot fix fundamental flaws of the base LDM, such as the difficulty with generating coherent text. This is an inherent limitation of the approach.\n\nQUESTIONS:\n- Your formulation for PFSA effectively sets the Query, Key, and Value representations equal to the same feature representation $f(z)$. This is a fascinating simplification. Have you considered or experimented with asymmetric variants within the parameter-free constraint? For example, could one use the raw feature $f(z)$ for the Query, but a structurally-simplified version (e.g., from a blurred or pooled feature map) for the Key and Value? My hypothesis is that this might allow the model to more explicitly attend *from* fine details *to* coarse structures.\n\n- I recently came across a work titled \"Swift parameter-free attention network for efficient super-resolution\" which also explores parameter-free attention mechanisms before. Could you clarify how your Parameter-Free Self-Attention (PFSA) mechanism relates to or differs from the approach in that work? This would help to better position your contribution within the existing literature.\n\n- You note that directly applying PFSA can lead to unstable denoising, which motivates the linear interpolation with the original latent $z$ in \\textbf{Eq. 2}. Could you offer more insight into this instability? For instance, does $PFSA(z)$ produce a latent with a significantly different statistical distribution (e.g., higher variance) compared to $z$? Is the primary role of the $(1-\\gamma)z$ term to act as a statistical anchor to keep the guided latent on-manifold?\n\n- The use of a cosine annealing schedule for $\\gamma_{t}$ is a well-motivated choice, aligning guidance strength with the diffusion model's tendency to generate structure first. How sensitive is the final image quality to the specific functional form of this schedule? Did you experiment with simpler schedules, like linear or step-wise decay, and did they yield comparable results?\n\n- The experiments are thoroughly performed using SDXL as the base model. How well do the RepLDM framework and its associated hyperparameters ($\\gamma$, $\\eta_{1}$, $\\eta_{2}$) generalize to other popular pretrained LDMs, such as the Stable Diffusion v1.5/v2.1 series or various community fine-tuned models?\n\nLIMITATIONS:\nYes. The authors provide a clear and well-written \"Limitations and Future Work\" section (Section 6). They responsibly acknowledge the method's inability to fix inherent flaws in the base model (like text generation) and discuss the computational scaling challenges when targeting ultra-high resolutions. This transparent reporting strengthens the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}