RepLDM: Reprogramming Pretrained Latent Diffusion Models for High-Quality, High-Efficiency, High-Resolution Image Generation
Authors: Boyuan Cao
Jiaxin Ye
Yujie Wei
Hongming Shan
Venue: NeurIPS 2025 spotlight ()
Keywords: Diffusion Model
High-Resolution Image
Attention Guidance
progressive pixel space upsampling
Primary area: deep_learning

Abstract: While latent diffusion models (LDMs), such as Stable Diffusion, are designed for high-resolution image generation, they often struggle with significant structural distortions when generating images at resolutions higher than their training one.

Instead of relying on extensive retraining, a more resource-efficient approach is to reprogram the pretrained model for high-resolution (HR) image generation; however, existing methods often result in poor image quality and long inference time.

We introduce RepLDM, a novel reprogramming framework for pretrained LDMs that enables high-quality, high-efficiency, high-resolution image generation; see Fig. 1. RepLDM consists of two stages: (i) an attention guidance stage, which generates a latent representation of a higher-quality training-resolution image using a novel parameter-free self-attention mechanism to enhance the structural consistency; and (ii) a progressive upsampling stage, which progressively performs upsampling in pixel space to mitigate the severe artifacts caused by latent space upsampling. The effective initialization from the first stage allows for denoising at higher resolutions with significantly fewer steps, improving the efficiency.

Extensive experimental results demonstrate that RepLDM significantly outperforms state-of-the-art methods in both quality and efficiency for HR image generation, underscoring its advantages for real-world applications.
Codes: https://github.com/kmittle/RepLDM.

RepLDM: Reprogramming Pretrained Latent Diffusion Models for High-Quality, High-Efficiency, High-Resolution Image Generation
Boyuan Cao!
Jiaxin Ye!
Yujie Wei
Hongming Shan
TTXiv:2410.06055V2 [cs.CV] 4 4 Dec 2025
Institute of Science and Technology for Brain-Inspire Intelligence & MOE Key Laboratory of Computational Neuroscience and Brain-Inspire Intelligence & MOE Frontiers Center for Brain Science, Fudan University
{caoby23, jxye22, yj wew22] @m. fudan. edu. en, hmshan@fudan edu. cn
1609640086
(4036x7280)
Figure 1: High-resolution images generated by RepLDM using single consumer-grade 3090 GPU. The corresponding thumbnails are generated by SDXL [36] at their training resolution.
Abstract
While latent diffusion models (LDMs), such as Stable Diffusion, are designed for high-resolution (HR) image generation, they often struggle with significant struc- tural distortions when generating images at resolutions higher than their training one. Instead of relying on extensive retraining, a more resource-efficient approach is to reprogram the pretrained model for HR image generation; however, existing methods often result in poor image quality and long inference time. We introduce RepLDM, novel reprogramming framework for pretrained LDMs that enables high-quality, high-efficiency, high-resolution image generation; see Fig. 1. Re- pLDM consists of two stages: (i) attention guidance stage, which generates latent representation of a higher-qualis traiiingrresolttion image using a novel training-fres leff-attention mechanism to enhance the structural consistency; and (ii) progressive upsampling stage, which progressively performs upsampling in pixel space mitigate the severe artifacts caused by latent space upsampling. The effective initialization from the first stage allows for denoising at higher resolutions with significantly fewer steps, improving the efficiency. Extensive experimental re- sults demonstrate that RepLDM significantly outperforms state-of state-of-the-arr methods in both quality and efficiency for HR image generation, underscoring its advantages for real-world applications. Codes: https: //gp::://ite thub. com/ cwm/kmitttttcccoeeocmmmcmm...cca kmittle el e/RepLDM.
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(008
(a) SDXL
(b) MultDD iffusiif
(c) Upsample Guidance
(d) DemoF usion
(el RepLDM (Ours)
Figure 2: Comparison our RepLDM with prior work in generating x image. The prompt is Neon lights illuminate the bustling cityscape at night, casting colorful reflections on the wet streets. Zoom-in for a better view.
Introduction
Diffusion models (DMs) have demonstrated impressive performance in visual generation tasks, particularly in text-to-image generation 17, 8, 16, 33, 34, 36, 46, 49-52, 55J. One notable variant of DMs is the latent diffusion model (LDD)). which performs diffusion modeling in latent space to reduce training and inference costs, enabling HR generation up to 1024 x 1024. While it is plausible to modify the input size for higher-reoolution generation, this often results in severe structural distortions, as illustrated in Fig. 2(a). Therefore, a recent research focus is on adapting trained LDMs for HR image generation without the need for additional training or fine-tuning (i.e. training-free manner), which can inherit the strong generation capacities of existing LDMs, especially open spen-sourced versions like Stable Diffusion.
Existing training-free approaches for HR image generation can be roughly categorized into three types: sliding window-based, parameter rectification-basse Sliding window-based methods first divide the HR image into several overlapping patches and use sliding window strategies perform denoising [1, 12, 25]. However, these methods could result in repeated structures and contents due to the lack of communication between windows; see Fig. 2(b). Parameter sectification-base methods attempt to correct models' parameters for better structural consistency through the entropy of attention maps, signal-to-noise ratio, and dilation rates of the convolution layers [14, 20-22, 56]. Though efficient, they often lead the degradation of texture details; see Fig. 2(c). Unlike the two types mentioned above, progressive upscaling- -baend methods are iteratively upsample the image resolution, which maintains better structural consistency and shows state-of -fet (SOTA) performance [6, 27, 28, 37J. Unfortunately, these methods require fully repeating the denoising process multiple times, leading to an unaffordable computational burden; e.g- AccDiffusion [28] takes 26 minutes to generate 4096 4096 image. In addition, their upsampling operation the latent space may introduce artifacts; see Fig. 2(d). To sum up, existing methods fail to ensure the fast, high-quality HR image generation.
and progressive upsampling-base -haeed.
In this paper, we propose RepLDM, novel reprogramming framework for pretrained LDMs that is capable of generating high-quality, high-resolution images while keeping high-efficiency; see Fig. 2(e). Specifically, RepLDM decomposes the denoising process of LDMs into two stages: (i) an attention guidance stage, and (ii) progressive upsampling stage. The first stage aims to generate a latent representation of high-quality image at the training resolution through the proposed attention guidance, which is implemented via a novel training-fres self-attention mechanism (TFSA) to improve structural consistency". The second stage aims progressively upsample the resolution in the pixel space rather than latent space, which can alleviate the severe artifacts caused by the latent space upsampling. By leveraging the effective initialization from the first stage, RepLDM can perform denoising in the second stage with significantly fewer steps, enhancing the overall efficiency with 5x speedup. Extensive experimental results demonstrate the effectiveness and efficiency of RepLDM in generating HR images over the SOTA baselines.
The contributions of this work are summarized as follows. (i) We propose RepLDM, a novel framework for high-quality, high-efficiency, high-resolution image generation through reprogramming pretrained LDMs. (ii) We propose attention guidance, which can utilize a novel training- free self-
IIn this paper, structural consistency refers to the plausibility the overall scene layout and the realism of object structures within an image. Specifically, a reasonable layout should follow logical spatial relation- ships for example, the sky should appear above the ground while realistic object structures should conform common sense, such cat having four legs rather than five.

---

attention to improve the structural consistency of the latent representation towards high-quality images at the training resolution. (iii) We propose progressively upsampling the resolution of latent representation in the pixel space, which can alleviate the artifacts caused by the latent space upsampling. (iv) Extensive experimental results demonstrate that the proposed Repl.DM significantly outperforms the SOTA models in terms of image quality and inference time, emphasizing its great potential for real-world applications.
2 Related Work
HR image generation with super-resolution. An intuitive approach to generating HR images is to first use pre-trained LDM to generate training-resslutionn
resolution (TR) images and then apply super- resolution model to perform upsampling [26, 31, 47, 48, 54J. Although one can obtain structurally consistent HR images in this way, super-resolution models are primarily focused on enlarging the image, and shown to be unable to produce the details that users expect in HR images [6, 27, 28].
HR image generation with additional training. Existing additional training methods either fine-tune existing LDMs with HR images [10, 19, 57] or train cascaded diffusion models to gradually synthesize righer-resolution images [17. 44]. Though effective, these methods require expensive training resources that are unaffordable for regular users.
HR image generation in training-free manner. Current training- free methods can be roughly classified into three categories: sliding window-based parameter rectification based, and progressive upsampling- ppssepling-based based methods. Sliding window-basee based methods consider spatially splitting HR image generation [1, 12, 25]. Specifically, they partition an HR image into several patches with overlap, and then denoise each patch. However, due to the lack of communication between windows, these methods result in structural disarray and content duplication. While enlarging the overlaps of the windows mitigates this issue, it result in unbearable computational costs. For the parameter rectification-batt methods, some researchers discovered that the collapse of HR image generation due the mismatches between higher resolutions and the model's parameters [14, 20-22, 56). These methods attempt to eliminate the mismatches by rectifying the parameters such as the dilation rates of some convolutional layers. While mitigating the structural inconsistency. they often lead to the degradation of image details. Different from the aforementioned two types, the progressive upsampling-bast based methods show SOTA performance in some recent studies [6. 27, 28, 37J. Though promising, they require fully repeating the denoising process multiple times, which incurs unbearable computational overhead. Additionally, these methods perform upsampling in the latent space, which may introduce artifacts.
Although their remarkable results, these methods fail to improve the quality of HR images and computational efficiency at the same time. In contrast, RepLDM aims to generate HR images with high quality and high efficiency, towards practical applications.
3 Method
3.1 Overview of RepLDM
Fig. presents the overview of RepLDM, which reprograms a pre-trained LDM generate HR images without further training. Formally, a pre-trained LDM utilizes a denoising U-Net model F to iteratively denoise the latent representation of size h x w x c, which is then converted back to the pixel space for final image generation through the decoder D of a variational autoencoder ( (VAE), We note that the initial latent representation is sampled from Gaussian distribution NO0 I), and for inference the encoder & of VAE is not involved.
Our RepLDM extends pre-trained LDMs for higher-resolution image generation in training free manner; i.e., E, D and F are fixed. RepLDM achieves this by decomposing the standard denoising process in the latent space into stages: (i) attention guidance stage, and (ii) progressive upsampling stage. In the first stage, RepLDM aims to generate a latent representation of a higher-quality TR
IIn this paper, training resolution refers to the resolution used during model training, while high resolution denotes resolution that substantially exceeds the training resolution beyond the level at which the model can directly produce satisfactory results.
3

---

Stage I: Attention Guidance Denoising
Stage 2: Progressive Upsompling Denoising
Upsarople Renutin
Reratin To
D
2v.
Atlention
Upcample
X
3
Attention Guidonce
Upsomple & Diffuse
X Mitrix Mullglication
Earmeni-wis Addition
W.
Pixel Space Upsampling
F Deneising U-Net
&
WAE Encoder
D
WiE Decoder
Figure 3: Overview RepLDM. RepL.MM divides the denoising process of a pre-trained L.DM into two stages. The first stage leverages the introduced attention guidance to enhance the structural consistency by utilizing a novel training-fres self eef-attention mechanism (TFSA). The second stage iteratively upsamples the latent representation in pixel space to eliminate artifacts.
image through the proposed attention guidance. The attention guidance is implemented as linearly combining the novel training-free self-attention mechanism (TFSA) and original latent representation to improve the structural consistency. In the second stage, RepLDM uses the latent representation provided by the first stage a better initialization, and iteratively obtains righheo-rsolution images via the pixel space upsampling and inffusion-denoisiie refinement.
We detail the attention guidance stage in $3.2, followed by the progressive upsampling stage in $3.3.
3.2 Attention Guidance Stage
Enhancing the structural consistency helps improve image quality [43]. However, it is challenging to do this in a training-free manner. We observe that the self-attention mechanism presents powerful global spatial modeling capability [5, 13, 29, 45], and this capability is parameter- agnostic. It is determined by the paradigm of global similarity calculation inherent to the self-attention mechanism [45, 58]. These insights motivate us consider designing novel training-free self- attention mechanism to elegantly enhance the global structural consistency of the latent representation.
Motivation.
Denoising with attention guidance. To improve the structural consistency of the latent repre- sentation at the training resolution z E R we propose a simple yet effective training-fres self-attention mechanism for attention guidance, termed TFSA, formulated
TFSA(2) = f-1 - [Softmax
[(=)((z)
I(z)).
where the operation reshapes the latent representation into shape (hw) x and f-1 reshapes it back; À is the scaling factor, with a default value of X = Vc.
However, we empirically observe that directly using the TFSA in Eq. (1) to improve the structural consistency of the latent representation could lead to unstable denoising. Therefore, we propose linearly combining the outputs of TFSA and the original latent representation as attention guidance, which is formulated as:
2 TFSA (=) + (1 (1-7)=
(2)
where & the structurally enhanced latent representation and y the guidance scale. Appendix A. we demonstrate that TFSA functions by modulating the distribution of latent representations. The term (1 (1 in Eq. (2) serves as a statistical anchor, helping to keep the guided latent representations on the data manifold and ensuring smooth transitions in their distribution.
As shown Fig. 3, we append the attention guidance in Eq. (2) denoising U-Net model and repeat the denoising process for a total of To times for the first stage. We note that the denoising
process starts from step To to 1, and the final output of the first stage denoted z (0)
daptive guidance seale. Considering that the latent representation mostly non-semantic noise in the first few steps of denoising, we delay i steps in introducing attention guidance. Moreover,

---

Med bease Up
033
00.0
0.47
3.0
Sampling flbee
(b) Processed only by WAE (c) upeempling poae space (d) upeempling in Istent space
PSNR and SSIM
Comparison between upsampling in pixel space and latent space. (a) RepLDM with latent space upsampling leads to severe artifacts. (b)-(e): Qualitative and quantitative comparisons of different upsampling methods.
Figure 4:
during the denoising process, the image structure is generated first, followed by local details [32, 44, 53J. Therefore, we primarily employ attention guidance in the early to mid-steps of denoising to focus on enhancing the structural consistency of the latent representation. Specifically, we introduce the adaptive guidance scale "Y by applying a decay to given guidance scale y, formulated as:
To To-k-t x) 1]]
2 cos
if i otherwise,
where B is the decay factor. In practice, considering that depends To for different resolutions, we use delay rate 771-
& control the number of steps for delaying attention guidance.
3.3 Progressive Upsampling Stage
Motivation. Fig. 2(a) shows that pre-trained LDMs still retain some ability generate high- frequency information when directly used to synthesize HR images, although they exhibit structural disarray. Therefore, intuitivelyy we can utilize the latent representation produced by the first stage as structural initialization, and generate the HR images through the caeaamlleddffus-deeee iteration in the latent space. However, this pipeline leads to severe artifacts, as shown Fig. 4(a). We speculate that this is due to the upsampling of latent representations in the latent space.
Pilot study. To examine this hypothesis, we conduct the following experiments. Specifically, we randomly select 10k images from ImageNet [4] to create an image set P. For each image P. we perform the following operations to obtain three additional image sets; (i) which use VAE to obtain the reconstructed image set Piet (ii) à up oD o E o down(x), which performs upsampling pixel space obtain the image set Ppxxi and (iii) which performs upsampling in latent space to obtain the image set Pua- Both upsampling up and downsampling down are performed using bicubic interpolation. Fig. 4(e) reports the quantitative results, where r represents the upsampling or downsampling rate. We calculate the PSNR and SSIM for pixel space upsampling set Ppix and latent space upsampling set Piat with respective the reference set Pier. It can be clearly observed that the latent space upsampling leads to significant performance decline compared to pixel space upsampling. Fig. 4(b-d) shows upsampling in the pixel space produces images close to the reference while upsampling in latent space leads to severe artifacts and detail loss.
D (xx),
D o up of down(x).
Progressive denoising with pixel space upsampling. Based on the above conclusion, we propose performing upsampling the pixel space rather than latent space and utilize diffusion and denoising to refine the upsampled higher-resolution image. Specifically, the second stage consists of sub- stages progressively upsample the training resolution to target resolution, each corresponding to one upsampling operation. For i-th sub-stage,
1,..., we prepend an upsample and diffuse operation before the denoising process, which can be defined as:
(i
o D(z)
0
2 (i-1() + /1 Vr-āT,
where l/ represents upsampling operation, ãT; is the noise schedule hyper-parameter of the Ti-th diffusion time step, and = (.(+1) the output of the 1)-th sub-stage; we use 2 denote the output from the first stage. Then, F is used to iteratively denoise 27, from time step T; obtain =
5

---

After completing all sub-stages, we obtain (n), which is then decoded to produce the final output xx()) = -D(2)").
We empirically found that generating higher- regher-resolution images requires more sub-stages. Addition- ally, when refining images using diffusion and denoising. higher resolutions demand larger time steps [44]. In practice, for flexibility, RepLDM allows users to customize the number of sub-stages n, and the diffusion time steps T, for each sub-stage by a pre-specified variable-length progressive
scheduler 7/2 normalized by To-
T1 T2 T5 T5 .......
The elements of 7/2 represent the denoising steps of each sub-stage,
4 Experiments
4.1
Implementation Details
Experimental settings. We use SDXL [36] as the pre-traines LDM and conduct inference using a single NVIDIA 4090 GPU. To ensure consistency when testing inference speed, we use single 3090 GPU, aligning with other methods. We randomly sample 33k images from the segment anything model (SAM) [24] dataset as the benchmark. Following the released code from DemoFusion [6], we use the FulerrDisereeeScchedlerr [23] setting To 50 and the classifier-free guidance [18] scale to 7.5.
Pixel space upsampling is performed using bicubic interpolation, and the decay factor 33 is fixed at 3.
Evaluation metries. The widely recognized metries Frechet Inception distance (FID) [15], Inception score (IS) [40], and contrastive language-image pre-training (CLIP) score [38] are used to evaluate model performance. Additionally, since calculating FID and IS requires resizing images to 299 x 299, which may not be suitable for evaluating HR images, we follow the experimental settings of [6, 28] perform ten 1024 1024 window crops on each image to calculate FID, and IS,. Since FID is known to be sensitive to small implementation details [35]. we employ a widely recognized implementation from a publicly available repository [42].
4.2 Quantitative Results
We compare RepLDM with the following models: (1) SDXL [36]; (2) MultiDiffusion [1]; (3)
ScaleCrafter [14]: (4) DemoFusion [6]; (5) Upsample Guidance (UG) [21]; (6) AccDiffusion [28];
and (7) HiDiffusion [56]. For fair comparisons, we disabled the FreeU trick [43] in all experiments.
Table Quantitative comparison results. The best results are marked in bold, and the second best results are marked by underline.
2016
2058
2018
4006
FID:
CLIP
FID:
FID:
FID
15
FID
15:
SDXL [36]
114.3
15.3
24.6
14.4
21.1
SaalCrrleer
98.2
151.9
175.1
9.7
164.5
170.1
7.5
UG
81.2
65.8
155.7
82
165.0
185.
6.8
20.5
1873
7M
197.6
21.8
Himt (56)
81.0
64.1
130.7
12.2
91.0
13.6
24.1
175.
12.8
11.3
25.1
148.1
14:00
7.4
21.2
DenoFusion
72.3
21.6
33.5
19.1
25.2
96.3
17.7
62.3
15.0
25.0
99.6
16.4
61.9
14.7
24.4
101.
20.7
63.5
13.5
24.7
Acchiiff [00)
71.6
21.0
52.1
13.0
25.1
98.8
16.4
67.9
11.1
24.5
100.2
15.2
61.4
11.5
24.2
105.
36.1
65.9
13..
34.
RM
06.0
21.0
47.4
17.5
25.1
$0.0
20.3
56.0
19.0
25.0
93.2
10.5
56.9
16.5
24.9
906
21.1
58.0
148
34.
report the performance of all methods on four different resolutions (Height x Width): 4096 x 4096,
4096 x 2048, 2048 x 4096, and 2048 x 2048. Considering that the generation time for HR images far exceeds that for low-resolution images, we used 2k prompts at the resolution of 2048 x 2048, and 1k prompts for resolutions greater 2048 x 2048. For resolutions, we set 7 = 0.004, 3 and 7/1 = 0.06 for Repl.DM. Given that the 4096 x 4096 resolution is significantly larger than other resolutions, we set 7/2 = [0.1, 0.2] (i.e., To = 50, Ti = 5, T- = 10) for 4096 x 4096, and 772 = [0.2] (i.e., To 50 and Ti 10) for other resolutions. When generating images with aspect ratio of r' we reshape the initially sampled Gaussian noise in the first stage to match r'. This process keeps the number of tokens in unchanged, preventing drastic fluctuations in the entropy of the attention maps in the transformer [22] leading to higher-quality images.
Table manifests that RepLDM significantly outperforms previous SOTA models, AceDiffusion and DemoFusion. This indicates that RepLDM generates images with higher quality. For more comprehensive analyses, we repeat the experiments of Table I with different random seeds to perform error analyses and conduct a further comparison of the models on the LAION-5B benchmark [41]; see Appendix G.

---

Table 2 indicates that RepLDM demonstrates remarkable advantage in inference speed compared to the SOTA models. On a single 3090 GPU, RepLDM requires only about one-fifth of the inference time needed by SOTA models such as DemoFusion and AccDiffusion.
Table 2: Model inference time.
The best results are marked in bold. Unit of Time: minute.
Resolutions
SDXL_
MultiDit: (1]
SealeCrafter [14]
UG
DemoFusion
AcdDiff (28]
Hibirt
Repl.DM
2048 x 2048
1.
3.0
1.0
1.8
3.0
3.0
0.2
0.6
2048 4096
6.0
6.0
12.7
4.3
Qualitative Results
In Fig. 5, RepLDM is qualitatively compared with AccDiffusion, DemoFusion, and MultiDiffusion. MultiDiffusion fails to maintain global semantic consistency. As indicated by the red boxes, Demo- Fusion and AccDiffusion tend to result in chaotic content repetition and severe artifacts, which speculate are caused by upsampling in the latent space (as analyzed in $3.3). In contrast, RepLDM not only preserves excellent global structural consistency but also synthesizes images with more details. More qualitative comparison results can found in Appendix B.
(a) SDXL
(b) MultiDifuusion
(c) DemoFusion
(d) AccDiffusion
(e) RepLDM (Ours)
beautiful swan gliding on pond.
A medieual kniggt armoo
A historic castle hill.
2048
Warm light spills from the cottage.
(086612008))
dog running through field of wilde wildflowers
Figure 5: Qualitative comparison with other baselines. The prompts used to generate the images are presented the white boxes. MultiDiffusion fails to maintain global semantic consistency. DemoFusion and AccDiffusion exhibit severe artifacts and content repetition. The red boxes indicate some synthesis errors. Zoom-in for a better view.

---

4.4 User Study
We invite 16 volunteers to partici- pate in double-bling buible-blind experiment to further evaluate the performance of the models. Each volunteer is re- quired to answer 35 questions. In each question, three images gener-
Table 3: Results of the user study.
Structural Consistency
Color Abundance
Detail Richness
Method
score
score"
score
score"
score
score"
AceDitt. [28]
6,28
0.88
6.78
0.60
6.18
0.53
DemoFusion [6]
5.99
0.59
6.69
0.51
6.18
0.53
Repl.DM
7.42
2.02
7.64
1.45
7.41
1.76
ated by AccDiffusion, DemoFusion, and RepLDM are presented. The volunteer needs to rate each image from to 10 in terms of structural consistency, color abundance, and detail richness. We calculate the average of their scores. Moreover, to eliminate bias in each volunteer's ratings for subtract the minimum value among the three scores given by each
each metric in each question, volunteer for each metric in each question. The rectified score is denoted as score Table shows that RepLDM surpasses previous SOTA models across all metrics.
5
Ablation Study
5.1 Attention Guidance
In this section, we first conduct ablation experiments on attention guidance, followed by ablation experiments on the hyper-parameters of attention guidance. In Appendix A, we provide detailed analysis of how attention guidance improves latent structural consistency and image quality.
Ablation on attention guidance. We keep unchanged and analyze the effect of attention guidance through qualitative and quantitative experiments. Table shows that attention guidance leads to improvements across various metrics, indicating that using attention guidance to enhance the consistency of latent encoding results in higher-qualit, images. The qualitative experiments in Fig. 6 demonstrate that using attention guidance eliminates image blurriness and enriches the image details. Note that FID and IS quantify the statistical differences between two distributions [2. 15, 40]. Since attention guidance mainly enhances visual quality by modifying the mid- and high-frequency components while preserving the low-frequency structure of the image, it has limited impact on the overall distributional statistics. Although attention guidance may not yield significant improvements in quantitative metrics, it provides a noticeable enhancement in human visual perception; see Table 3. Please refer to Appendix C.2 for additional qualitative ablation results.
Table 4: Ablation attention guidance (AG). The best results are marked in bold.
Method
2018
2018
2018 x 4006
1006
2018
4005
F10
PID,
15,
CLIP
F1D
15
F13)
15.
CL1
F1D
15
FID.
13.
CLI
R1D
13
F12.
15.
CLIP
w/o AG
60.8
21.6
47.5
17.4
25.3
91.5
20.3
58.0
14.5
25.0
95.3
19.9
58.4
14.5
24.9
92.0
21.6
59.8
13.6
24.5
w/
AG
66.0
21.0
47.4
17.5
25.1
89.0
20.3
56.0
19.0
25.00
93.2
19.5
56.9
16.5
24.9
90.6
21.1
59.0
14.8
24.6
[2048x2048]
[409 4098
(a) w/ attention guidance (b) w/o attention guidance (c) w/ attention guidance (d) w/o attention guidance
Figure 6: Ablation attention guidance. Zoom-in for better view.
Ablation on attention guidance with ControlNet. To further demonstrate the generalization ability of attention guidance, in this section, we perform qualitative ablation study of attention guidance with ControlNet [55]. Specifically, we conducted comparative experiments using two types of conditional guidance (canny and depth) across two resolution scales: 4096 x 4096 and 2048 x 2048. As shown Fig. the integration of attention guidance with ControlNet substantially enhances chromatic fidelity and structural granularity in synthesized images.
Ablation on guidance scale 7. We fix ? 0.06, 7/2 = [0.2] and then explore the effect of the guidance scale y through both quantitative and qualitative experiments. For the quantitative

---

Condition image:
wi Attenttionn Guidance
wil Attenton Guidance
Cendition image: depth.
W/Atention Guidance
wro Attention Guidance
Figure 7: Ablation on attention guidance with ControlNet.
experiments, we find that y = 0.004 performs better. Interestingly, when a larger y is used, the visual quality of the images be further enhanced. As shown in Fig. 8, using larger guidance scale results in richer image details. This allows users to generate images according to their preferences for detail richness and color contrast by adjusting the guidance scale. The setup and results of the quantitative experiments are detailed in Appendix C.2.
9448/20081
(a) y=0.000 (w/o AG)
(b) 0=0.004
(c) 0=0.006
(d) V=0.009
Figure 8: Ablation on guidance scale. Zoom in for better view.
Ablation on delay rate Th- We fix
y 0.004, 1/2 [0.2] and then in- vestigate the impact of the delay rate
? through both quantitative and qual- itative experiments. The quantitative analysis results indicate that better generation results can be achieved when 171 0.06. indicating that ap- propriately delaying the effect of at- fur-
(a) 19s Figure Ablation on delay rate. Errors indicated by
(b)
(c) 0.00
(d)
red boxes can be eliminated by delaying attention guidance. Zoom-in for a better view.
tention guidance contributes ther improving the quality of the images. We conjecture that this is because, at the very beginning of the denoising process, the structural information in the latent encoding has not yet emerged, and thus attention guidance cannot effectively enhance structural consistency. As shown in Fig. 9, delaying the effect of attention guidance eliminates some generation errors, further improving image quality. The setup and results of the quantitative experiments are detailed in Appendix C.2.
A Ablation the time steps of atten- tion guidance. To explain why at- tention guidance needs to be applied during the early to middle steps of denoising, we apply attention guid- ance during different denoising steps of the first stage: (a) 47 33, (b) 32 to 17, and (c) 16 to 1. Fig. 10 shows that when attention guidance ap- plied during the early to middle steps of denoising, the image becomes clearer and more detailed; however, when attention guidance applied during the later steps of denoising, has negligible effect on the generated image. We speculate that this is because diffusion models tend to synthesize
qu
2-13
9-1
Figure 10: Applying attention guidance at different denois-
ing steps. Zoom-in for better view.

---

structural information first [32, 44, 53], and once the structural information is generated, attention guidance may have limited impact on structural consistency.
5.2 Progressive High-Resolution Denoising
In this section, we conduct abla- tion experiments on the progressive scheduler 7/2 the second stage of RepLDM. Specifically, 0, 0. 0 and then explore the ef- feet of the progressive scheduler 7/2 through both quantitative and quali- tative experiments. Quantitative perimental results indicate that an ex- cessively large progressive scheduler value may result in decline in image quality. This
fixed y
(a) Mi [0.9] Figure 11: Generated images using different 7/2- (a): When the value of progressive scheduler is too large, the structural repetition issue may reappear. (b) (d): The visual effects are similar. Therefore, we can use a smaller progressive scheduler value to accelerate inference.
(b) n. [0.7]
(c) [0.5]
(d) n2 = [0.3]
also be observed in Fig. 11. It is evident that a too large progressive scheduler value may lead to structural misalign- ment and repetition issues observed in pre-trained SDXL. When the progressive scheduler value is sufficiently small, changing it yields similar visual effects. Therefore, we can choose a smaller progressive scheduler value (e.g., 0.2) to accelerate inference. The setup and quantitative results are detailed in Appendix C.2.
Limitations And Future Work
RepLDM exhibits limitations in the following aspects: Effectively con- trolling text in images is challeng- ing, demonstrated by examples in Fig. 12. This may be due to the inher- limitations of SDXL in generat- ing textual symbols. Text, due to its more regular structure compared to other image content, is difficult to re- store by directly enhancing the struc- tural consistency of the latent repre- sentation. We speculate that the most reliable approach would be to fine-tune the model specifically images containing text. (ii) When generating ultra-high resolution images, such as 12800 12800, the second stage of RepLDM inevitably needs to be decomposed into more sub-stages, which increases the model's inference time.
(d) RepLDM Figure 12: Limitations of RepLDM. The generation results of SDXL at its training resolution and those of RepLDM at higher resolutions are provided. As indicated by the white boxes, RepLDM fails to address the text structure errors in- herited from SDXL.
SDXL
(b) RepLDM
(c) SDXL
Developing a low-cost and effective fine-tuning method correct text generation errors may be promising topic. Moreover, adapting attention guidance to other tasks, such as video generation can be an interesting issue.
7 Conclusion
In this paper, we reprogram pretrained LDMs, unlock their potentials, and propose RepLDM for high-quality, high-efficiency, high-resolution image generation. RepLDM divides the denoising process of an LDM into two stages: (i) attention guidance stage, and (ii) progressive upsampling stage. The first stage generates structurally enhanced latent representations through the proposed attention guidance, employing novel parameter-fi free self-attention mechanism. The second stage iteratively performs upsampling in the pixel stage, thus eliminating the artifacts caused by latent space upsampling. Extensive experiments show that our proposed Repl.DM significantly outperforms SOTA models while achieving 5x speedup in HR image generation.
10

---

Acknowledgement
This work was supported by the National Natural Science Foundation of China (No. 62471148 148) and the computations in this research were supported by the CFFF platform of Fudan University.
References
[1] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. ICML, 2023.
[2] Shane Barratt and Rishi Sharma. A note on the inception score. arXiv preprint arXiv:1801.01975 2018.
[3] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. NIPS, 15.16344-163355 2022.
[4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255. Icee, 2009.
[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins arXiv:2010. /929, 2020.
[6] Ruoyi Du, Dongliang Chang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. Demofusion: Democratising high-resolution image generation with no $$S. In CVPR, pages 6159-6168, 2024.
[7] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In ICML, 2024.
[8] Zhida Feng, Zhenyu Zhang, Xintong Yu, Yewei Fang, Lanxin Li, Xuyi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shikun Feng, et al. Ernie-ilg 2.0: Improving text-to-image diffusion model with knowledge-enhanee mixture-of-denoiiing--oo 10135-10145, 2023.
In CVPR, pages
[9] Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Mare Sun, and Benjamin Bossan. Accelerate: Training and inference scale made simple, efficient and adaptable. https: :/gi thh www.huggiigaaeeeeeoooooooooooeoo acwee/eaceeeeeeeeeeeeeeeaeaaeeee 2022.
[10] Lanqing Guo, Yingqing He, Haoxin Chen, Menghan Xia, Xiaodong Cun, Yufei Wang, Siyu Huang, Yong Zhang, Xintao Wang, Qifeng Chen, et al. Make a cheap scaling: self-cascade diffusion model for higher-resolution adaptation. arXiv preprint 2rXiv:2402.10491, 2024.
[11] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Ani- matediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.0472.. 2023.
[12) Moayed Haji-Ali, Guha Balakrishnan, and Vicente Ordonez. Flasticdiffusion: Training-free arbitrary size image generation. arXiv preprint arXiv:23/1. 18822, 2023.
[13] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. NIPS,
2021.
[14] Yingqing He, Shaoshu Yang, Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang. Ran He. Qifeng Chen, and Ying Shan. Scalecrafter: Tuning-free free higher- resouutinoo visual generation with diffusion models. In ICLR, 2023.
[15] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. NIPS, 30, 2017.

---

[16] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NIPS, 33:6840- 35:8840-6851, 2020.
[17] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
[18] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. M2Kv:2207.11599. 2022.
arXiv preprint
[19] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In ICML, pages
PMLR, 2023.
[20] Linjiang Huang, Rongyao Fang, Aiping Zhang, Guanglu Song, Si Liu, Yu Liu, and Hongsheng Li. Fouriscale: A frequency perspective on training-free high-resolution image synthesis. In ECCV, pages 196-212. Springer, 2024.
[21] Juno Hwang. Yong- Mayg- Park, and Junghyo Jo. Upsample guidance: Scale up diffusion models without training. arXiv preprint arXiv:2404.0177 2024.
[22] Zhiyu Jin, Xuli Shen, Bin Li, and Xiangyang Xue. Training-free diffusion model adaptation for variable-sized text-to-image synthesis. NIPS, 36, 2024.
[23] Tero Karras, Mika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basee generative models. NIPS, 35:26565-2666 2022.
[24] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. In ICCV, pages 4015-4026 2023.
[25] Yuseung Lee, Kunho Kim, Hyunjin Kim, and Minhyuk Sung. Syncdiffusion: Coherent montage via synchronized joint diffusions. NIPS, 36:50648 0..50668--0660,, 2023.
[26] Jingyun Liang. Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In ICCV, pages 1833-1844, 2021.
[27] Zhihang Lin, Minghao Lin, Wengyi Zhan, and Rongrong Ji. Accdiffusion v2: Towards more accurate higher-resolution diffusion extrapolation. arXiv preprint arXiv:4412.0009, 2024.
[28] Zhihang Lin, Mingbao Lin, Meng Zhao, and Rongrong Ji. Accdiffusson: An accurate method for higher-resolution image generation. arXiv preprint arXiv:2407.10738, 2024.
[29] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. ICCV, pages 10012-10022, 2021.
[30] Mario Lucic, Karol Kurach, Marcin Michalski. Sylvain Gelly, and Olivier Bousquet. Are gans created equal? large-scale study. NIPS, 31, 2018.
[31] Xiaotong Luo, Zekun Ai, Qiuyuan Liang, Ding Liu, Yuan Xie, Yanyun Qu, and Yun Fu. Adaformer: Efficient transformer with adaptive token sparsification for image super-resolution. In AAAL, volume 38, pages 4009-4016, 2024.
[32] Yang Luo, Yiheng Zhang, Zhaofan Qiu, Ting Yao, Zhineng Chen, Yu-Gang Jiang, and Tao Mei. Freeenhance: Tuning-free image enhancement via content-consistent noising-and-deeoiisngg process. In ACMMM, pages 7075-7084, 2024.
[33] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In AAAI, volume 38, pages 4296-4304, 2024.
[34] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, pages 8162-8177 8171. PMLR, 2021.
12

---

[35] Gaurav Parmar, Richard Zhang, and Jun- Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In CVPR. pages 11410-11444 11400, 2022.
[36] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint wwXiv2230..019522 2023.
[37] Haonan Qiu, Shiwei Zhang. Yujie Wei, Ruihang Chu, Hangjie Yuan, Xiang Wang, Yingya Zhang, and Ziwei Liu. Freescale: Unleashing the resolution of diffusion models via tuning-free scale fusion. arXiv preprint arXiv:24122096600 2024.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 7748-87665 PMLR, 2021.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[40] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. NIPS, 29, 2016.
[41] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278-25294 2022.
[42] Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. https:/ https:///ithubb. wtt.thhh..mm/mmmmemee...... pytorch- fid, August 2020. Version 0.3.0.
[43] Chenyang Si, Ziqi Huang. Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. In CVPR, pages 4733 4743, 2024.
[44] Jayyan Teng, Wendi Zheng, Ming Ding, Wenyi Hong, Jianqiao Wangni, Zhuoyi Yang, and Jie Tang. Relay diffusion: Unilying diffusion process across resolutions for image synthesis. arXiv preprint arXiv:2309.03330 2023.
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS, 30, 2017
[46] Chenhui Wang, Tao Chen, Zhihao Chen, Zhizhong Huang, Taoran Jiang, Qi Wang, and Hong- ming Shan. Fidm-vton: Faithful latent diffusion model for virtual try-on. arXiv preprint 24X.:2244.14462, 2024.
[47] Jianyi Wang. Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Ex- ploiting diffusion prior for real-world image super-resolution. arXiv preprint 200.00530 2023.
[48] Yang Wang and Tao Zhang. Osffnet: Omni-stage feature fusion network for lightweight image 2024.
super-resolution. In AAAI, volume 38, pages
[49] Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jin- gren Zhou, and Hongming Shan. Dreamvideo: Composing your dream videos with customized subject and motion. In CVPR, pages 6537-6549, 2024.
[50] Yujie Wei, Shiwei Zhang, Hangjie Yuan, Biao Gong, Longxiang Tang, Xiang Wang, Haonan Qiu, Hengjia Li, Shuai Tan, Yingya Zhang, et al. Dreamrelation: Relation-centric video customization. arXiv preprint (tl.:25533076002. 2025.
[51] Yujie Wei, Shiwei Zhang, Hangjie Yuan, Xiang Wang, Haonan Qiu, Rui Zhao, Yutong Feng. Feng Liu, Zhizhong Huang, Jiaxin Ye, et al. Dreamvideo-2: Zero-shot subject-driven video customization with precise motion control. arXiv preprint XrXiv:2410.333 13830, 2024.
[52] Jiaxin Ye, Boyuan Cao, and Hongming Shan. Emotional face-to-speech. arXiv preprint 2000v2250.01144, 2025.
13

---

[53] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-free free energy-guidedn conditional diffusion model. In ICCV, pages 28144-23184 2023.
[54] Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte. Designing practical degradation model for deep blind image super-resolution. In ICCV, pages 4791-4800, 2021.
[55] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pages 3836-3847, 2023.
[56] Shen Zhang. Zhaowei Chen, Zhenyu Zhao, Yuhao Chen, Yao Tang, and Jiajun Liang. Hidiffu- sion:
Unlocking higher-resolution creativity and efficiency in pretrained diffusion models. In ECCV, pages 145-161. Springer, 2025.
[57] Qingping Zheng, Yuanfan Guo, Jiankang Deng, Jianhua Han. Ying Li. Songcen Xu, and Hang Xu. Any-size-diffusion Toward efficient text-driven synthesis for any-size hd images. In AAAI, volume 38, pages 7571-7578, 2024.
[58] Yupeng Zhou, Daquan Zhou, Ming-Ming Miigg Cheng, Jiashi Feng, and Qibin Hou. Storydiffu- sion: Consistent self-attention for long-range image and video generation. arXiv preprint 4rXiv:2405.01434, 2024.
14

---

Appendix
How Does TFSA/Attention Guidance Work?
15
A.1 TFSA Clusters Semantically Related Tokens
15
A.2 TFSA Adjusts the Amplitude of High- and Low- frequency Components
16
A.3
Visualization of Attention Maps in TFSA
17
B Supplementary Qualitative Comparison of $4.3
18
Supplementary Ablation Experiments of
18
C.1 Further Qualitative Analysis Attention Guidance
C.2 Ablation on the hyper-parameters of Attention Guidance
18
C,3 Ablation on Progressive Scheduler Value
18
D Ablation on the Attention Guidance Components
20
D.I Ablation on the Guidance Scale Decay Strategy
20
D.2 Ablation on the Attention Calculation Paradigm
E Further Model Efficiency Analysis
21
RepLDM Algorithm
23
G Robustness Analysis
24
H Comparative and Ablation Analysis Based on StableDiffusion 2.1
25
H.1 Comparison Experiments
25
H.2 Ablation Study Attention Guidance
25
I Attention Guidance Also Works in Other Generation Frameworks
26
L1 Quantitative Ablation
26
1.2 Qualitative Ablation
26
J Super-Resolved Images Tend to Lack High- Righ-Ressluttin Details
27
K Memory Usage Analysis
28
A How Does TFSA/Attention Guidance Work?
In this section, we further elaborate on the working mechanism of attention guidance. Our attention guidance enhances the structural consistency of the latent representation by integrating the output of TFSA. Therefore. be described in two aspects: (i) clustering the related tokens in the latent representations; (ii) adjusting the amplitude of the high-frequency and tow-frequency components in the latent representations.
conduct detailed analysis of TFSA. Specifically, the functionality of TFSA can
A.1 TFSA Clusters Semantically Related Tokens
Visualization of the clustering effect of TFSA. TFSA reorganizes tokens based on their simi- larities. Intuitively, this enables TFSA to perform token clustering, which enhances the structural
15

---

consistency of latent representations. To demonstrate the clustering effect of TFSA, we calculated the deviation of the tokens' mean (DTM) of the latent representations & and z5. Concretely, assuming
Z: E Rhxxxx and Z = Flatten(z;) (z+) = yyu ..v] E RRNx xc, where N = h x w, we calculate
DTM as:
(5) To provide an intuitive illustration of the clustering effect of TFSA, we visualize the DTM based on token indices (i.e,' 1,..., N) when is relatively large. As shown in columns (A) and (B) of Fig. 13, compared to the DTM of ,, (blue points), the DTM of &, (red points) becomes more dispersed and exhibits distinct stripe patterns, indicating that TFSA indeed clusters the tokens of the latent representations. This clustering effect can be more directly demonstrated when is smaller. As shown the heatmaps in columns (C) and (D) of Fig. 13, it is evident that TFSA clusters semantically related tokens.
DTM = (y - mcan Zi) for i = =I,..., I....,NN
(A)
(B)
(C)
(D)
(E)
16
(a) 8
06
-10
-20
-20
15
DIN (-10
(b)=
1024x10224
Figure 13: The clustering effect of TESA. Columns (A), (B), (C), and (D) show the DTM of latent representations, while column (E) presents the corresponding generated RGB images.
The clustering effect of TFSA leads to accelerated structural denoising. Fig. 13 shows that the clustering effect of TFSA A clarifies the semantic structures of objects, enabling the model to complete the denoising of low-frequency structures carlier. This early revelation of the overall image layout provides a stronger prior for subsequent fine-detail generation. To illustrate this. Fig. 14 presents the denoising process for the ablation of attention guidance. Note the regions highlighted by red boxes. With the incorporation of attention guidance, these areas exhibit clearer structures, which facilitates the generation of more affluent details and more vivid colors in subsequent steps.
To quantitatively demonstrate that TFSA ac- celerates structural emergence, we calculate the SSIM between z and Zo. where 1 E 1, 1.2...,T 1, and T 50. As shown in Fig. 15, compared to the naive denoising pro- cess, attention guidance consistently drives the latent representations closer to their final states each step, indicating the structural foresee- ability of TFSA.
1.0
0.9
SSIS
0.8
0.7
w/o AG
w/ AG
A.2 TFSA Adjusts the Amplitude of High- and Low-frequency Components
50
40
30
20
Denoising Step
Figure 15: Quantitatively analysis on the clus-
tering effect of TFSA. We calculate the SSIM between noised latents 2; (1 < < 49) and their corresponding clean latent 20.
The aim of this experiment is to explain: (i) why appropriately delaying attention guidance can resolve structural deformation issues (as shown in Fig. 9): (ii) why attention guidance enhances the details and colors of the image (as shown in Fig. 6 and 8); and (iii) why applying attention guidance in the later stages of denoising does not enhance the image details and colors (as shown in Fig, 10).
16

---

1=20
t=15 t=10 1=5
t=0
w/o
AG
w/
AG
Figure 14: Denoising visualization for the ablation of attention guidance. As indicated by the red boxes, the clustering effect of TFSA prompts earlier structural emergence, delivering better prior for subsequent fine-detail generation. Resolution: 1024 X 1024.
The Fourier transform of the latent representation and the mean of the standard deviations across all channels. 2, is represented in blue, while ž is represented red; the Fourier transforms are shown as solid lines, and the standard deviations are shown as dashed lines. The results are based on the generation process of 5k images.
Figure 16:
To explain the aforementioned three points, as shown in Fig. 16, we calculate the Fourier transforms of z, (blue solid line) and 2, (red solid line), along with the mean of the standard deviations for all their channels (dashed line). It can be observed that TFSA significantly alters the relative amplitudes of the high- and low- freeuencqccy components the latent representations during the initial denoising steps (from / = 49 to 47), particularly affecting the low-frequency components, which results in structural deformation. During the early and middle stages of denoising (from t = 44 to 29), TFSA increases the amplitudes of high- high-frequency components in the latent representations, which explains why attention guidance leads to richer details and colors. the later stages denoising (from 28 to 0), TFSA slightly suppresses the gigh-freuuecc components of the latent representations while almost leaving the fowefreqyyncy components unchanged. This explains why applying attention guidance in the later stages of denoising cannot enrich details and colors of the generated images.
Additionally, Fig. 16 shows that TFSA increases the standard deviation of ž, during the early and middle stages of denoising, while decreasing the later stages. The trend the standard deviation changes is closely consistent with the variation in the amplitude of the high-frequency components. We conjecture that this is because the amount of information in the latent representations is positively correlated with the standard deviation, where larger standard deviation corresponds to more image details and larger frehurryy components.
A.3 Visualization of Attention Maps in TFSA
To further demonstrate the clustering effect of TFSA on related tokens, visualize its attention maps. As shown in Fig. 17, without using projection matrices, the correlations between tokens are determined jointly by their represented colors and semantics. For example, in Fig. 17(a), the key
17

---

(a)
(b)
Figure 17: Visualization of attention maps in TFSA. The query tokens are highlighted with red boxes, and the heatmap color ranges from blue to red, indicating increasing correlation strength between the key tokens and the query tokens. Resolution: 1024 x 1024. Zoom-in for better view.
tokens correlated with the query token at the selected car location are related not only to the car itself (i.e., the concept of the car) but also to its color. TFSA leverages such correlations to fuse token information, thereby accelerating the formation of the overall image layout.
B Supplementary Qualitative Comparison of $4.3
Fig. 18 presents additional qualitative comparison results. MultiDiffusion continues to struggle with maintaining global consistency; indicated by the red boxes, DemoFusion tends to produce repetitive content, problem somewhat alleviated AccDiffusion but not fully resolved. As highlighted by the black boxes, another issue with AceDiffusion is the presence of noticeable streak artifacts in the images.
C
Supplementary Ablation Experiments of $5
C.1 Further Qualitative Analysis of Attention Guidance
Fig. 19 provides additional qualitative ablation results on attention guidance. Individual preferences for contrast, color vividness, and detail richness may vary. attention guidance allows users to adjust parameters such as the guidance scale to synthesize images according to their preferences.
C.2 Ablation on the hyper-parameters of Attention Guidance
Quantitative analysis of guidance scale. We sampled Ik prompts, fixed m = 0.06, = [0.2] and performed ablation studies for guidance scale 7. The quantitative results are shown in Table Considering all metrics, we find that 7 0.004 achieved better quantitative results.
Quantitative analysis of delay rate. We sampled 1k prompts, fixed 0.004, 7/2 [0.2] and performed ablation studies for delay rate 7/1 Table 6 presents the experimental results, indicating that better results can be achieved when 77 = 0.06. This means that appropriately delaying the effect of attention guidance can further enhance the quality of the generated images.
C.3 blation on Progressive Scheduler Value
This section presents the results of quantitative ablation analysis on the progressive scheduler 7a the second stage RepLDM. We fixed : = 0, mn = 0, sampled 500 prompts, and generated Ik images to investigate the optimal value of the progressive scheduler. Table 7 presents the quantitative results, indicating that using an excessively large progressive scheduler may lead decline in image quality.
18

---

(a) SDXL
(b) MultiDiffusion
(c) DemoF usion
(d) AccDiffusion
(e) RepLDM (Ours)
hustornccappp
(2046x
chagenlly Novers
2040
(4096)2048]
(4096x44096)
hahnrren casta hi Iie eno the petientty
(0066
Figure 18:
Qualitative comparison with other baselines. Zoom-in for a better view.
19

---

w/ AG
w/o AG
w/ AG
w/o AG
w/ AG
w/o AG
Figure 19: Further qualitative analysis of attention guidance (AG). Using attention guidance significantly enhances image quality. The details were enriched, for example: the clouds in the sky, ripples the water, reflections on the lake, and even the expressions in person's 's eyes. Resolution: x048x2048, Best viewed ZOOMED-IN.
Table 5: Quantitative ablation experiments on the guidance scale The best results are marked in bold, and the second best results are marked by underline.
Mechod
1024
1024
1600
1600
2045 x 2048
FED
15
FID
15c
CLIP
FID
FID, 1
15c
CLIP
FID
FID:
CLIP
0.000
90.35
59.15
21.21
17.69
25.05
$0.91
54.74
21.45
15,43
24.95
91.78
59.06
21.57
12.36
24.86
0.001
90.50
53.04
21.34
16.75
03
91.17
54.31
21.19
15.47
24.93
91.40
36.75
21.87
15.85
24.35
0.003
90.10
57.08
20:80
16.61
25
50.56
53.95
21.35
15:46
24.58
90.57
55.40
21.47
17.00
24.22
0.004
89.40
36.44
20.95
16.55
25.00
$9.91
54.23
20.91
15.54
25.01
98.11
38.11
21.88
16.78
24.04A
0.006
89.79
34.18
20.33
15.95
25.16
90.36
56.71
20.33
14.59
25.06
90.32
59.86
20.37
16.12
25,00
0.007
90.42
60.29
20.07
16.20
25
90.91
59.35
20.36
14.16
25.12
93.56
61.81
20.14
15.70
25.06
0.009
94.29
07.87
1911
13,00
23
94.38
70.21
19.00
12.13
35.16
94.39
08.54
19.22
13.03
25.33
Table 6: Quantitative ablation experiments on the delay rate 771 The best results are marked in bold, and the second best results are marked by underline.
1024
1600
F1D
15.1
111.,
15,
C1IP
F1D)
15:1
FID,
18,
CLIPP
FID
15.7
FID,
15.
C7.IP
R1 0.00
16:45
59.41
2481
0.02
$0.96
57.67
31.99
16.87
25.05
99.76
54.77
21.18
15:35
24.55
91.75
59.08
21.57
18.16
24.56
16.63
R 0.08
$9.01
56.64
16.55
29.91
54.23
25.01
21.18
in 0.08
95
50.97
21.05
16.76
25.05
39387
54.10
31.22
15655
24.5%
90.74
55.15
20.99
17:06
24.32
m 8938
16.55
53.85
21.24
2480
81 0.14
57.13
30.1
16:40
25.00
90.06
13.33
31.33
1560
34,99
93.69
15.25
31.17
76
24.11
$4.74
01
57.29
53.79
24.97
2480
D Ablation on the Attention Guidance Components
D.1 Ablation on the Guidance Scale Decay Strategy
To investigate the impact of different guidance scale decay strategies, we conduct ablation studies using two additional schemes linemmes-inearr linear decay and exponential decay- and analyze their quantitative and qualitative performance. For quantitative ablation, we generate 2k samples at a resolution of 2048 x 2048 using each strategy and calculate the criterions on the SAM benchmark. Table 8 shows that different strategies yield similar results, indicating that Repl.DM is not sensitive to specific decay strategy. Fig. 20 illustrates the qualitative results. Qualitatively, these decay strategies also produce similar visual experience.
20

---

Table 7: Quantitative ablation study the progressive scheduler Value. The best results are
marked in bold, and the second best results are marked by underline.
FID
18.
FiD,
15c
CLIP
F1D
15:
FID,
18.
CLIP
SDXI.
101.56
25.78
73.67
21:23
36.87
112.54
15:44
79.03
20.61
26.55
[0.55]
09.13
28.50
63.67
24.53
36.09
93.93
26.73
65.84
23.27
26.77
[0.7]
92.85
24.97
27.07
25.17
57.34
122 [0.9]
92.94
30.79
64.57
24.29
27.11
91.86
30.45
55.38
24.95
26,98
[0.5]
90.33
24.26
27.15
31.18
36.32
122 [0.4]
93.04
30.96
03.33
24.77
27.14
91.71
32.47
53.72
25.15
27.00
[0.3]
24.54
27.15
90.72
53.32
2643
12 [0.2]
09.09
61.23
23.71
12.17
92.71
53.19
[0.1]
93.44
30.69
63.75
25.18
27.22
92.94
30,00
53.77
24.71
27.18
Table 8: Ablation on the guidance scale decay strategies, The best results are marked in bold, and the second best results are marked by underline.
Strategles
18, 1
HID,
15.
CLIP
21.5
42.2
Esposearial
66.8
21.8
47.0
16.3
25.3
Cosine (defoal))
66.0
21.0
47.4
17.5
25.1
Linear
Exp
Cosine (Default)
Figure 20: Qualitative ablation on guidance scale decay ctrategies.
D.2 Ablation on the Attention Calculation Paradigm
For TFSA, objective is remove the learnable parameters from the Self-Attention mechanism, while maintaining its computational paradigm as unchanged as possible. In TFSA, Q. K, and V identical. Therefore, TFSA is totally symmetric formula. As analyzed before, this paradigm encourages the clustering of semantically related tokens, and finally leads to finer details and richer colors. An interesting question arises: if spatially downsample Q, K, V before applying TFSA and informulate/a it into an asymmetric paradigm (denoted as TFSA-A), would TFSA-A encourage the model attend more explicitly from fine details to coarse structures?
To answer this question, we design an asymmetric variants, TFSA-A. Specifically, TFSA-A performs 2x2 pooling operation downsample the K and V matrices before the attention calculation operation, ensuring that the output of Softmax ( //dVV V remains the of shape (hw) Table shows that TESA-A produces comparable quantitative results. In Fig. 21, we observe that although TFSA-A achieves quantitative results comparable those of TFSA, its visual quality is significantly inferior. In fact, TFSA-A tends to reduce image details. This aligns with our hypothesis: the 2 2 pooling acts low-pass filter, causing the loss of fine-grained information in the latent representations and leading the model to focus more on low-frequency structures.
E Further Model Efficiency Analysis
Computational complexity analysis of TFSA. Note that attention guidance is only applied during the first stage of generation. Assume we have HR image zo with resolution of H. x W C. we
21

---

Table 9: Ablation on the attention calculation paradigm. The best results are marked in bold, and the second best results are marked by underline.
Paradigm
FID,
18c
who guidance
66.8
21.6
47.5
17.4
25.3
ITSA
66.0
21.0
47.4
17.5
25.1
w/o AG
w/ TFSA-A
w/ TFSA
Figure 21: Ablation on the attention calculation paradigm. Resolution: 2048 x 2048.
encode the image To into latent space and obtain latent representation &0 R Before feeding zo into TFSA, we reshape to a (hw) X c matrice. The computation TFSA follows a formulation similar to that of self-attentio: Softmax = /c)z.. Thus, the computational complexity of TFSA ((hw))2) Taking SDXL as an example, the training resolution H 1024, W 1024. After VAE encoding,
W/8 128. For each denoising step, the FLOPS of TFSA is approximately 2 x (h x w) x c, which is around 2.15 GFLOPs negligible compared to the FLOPs of the denoising network (several TFLOPs per step).
4, h H/8- 128,
To answer this question, we analyze the time consumption of each component in DemoFusion and RepL.M when generating images at the resolution of 4096 x 4096.
How does pixel space upsampling accelerate generation?
Table 10: The time consumption of DemoFusion when generating 4096 x 4096 resolution images.
Metric
Denoise 1024
Denoise 2048
Denoise 3072
Denoise 8096
Decode 4096
Total
number of steps
50
50
50
50
200
Time
185
Table The time consumption RepLDM when generating 4096 x 4096 resolution images. The intermediate encoding/decoding operations are highlighted in underline.
Metric
Dencise
Decode 1024
Encede
Dencise 3304
Decode 3300
Encoci 4066
Dencise 4066
Uncode 4036
Total
number stepe
Table 10 shows that denoising at high resolutions is a time-consuming process. DemoFusion requires substantial generation time because it performs the full denoising process high resolutions. Note that, compared with the cost of the denoising process at high resolutions, the costs of encoding and decoding are negligible. Table 11 shows that RepLDM significantly accelerates generation by substantially reducing the number of denoising steps at high resolutions. This is because RepLDM performs pixel space upsampling through multiple rounds of encoding and decoding, producing high-quality low-resolution images that serve better initialization. As a result, RepLDM can significantly reduce the number sampling steps required for HR generation, thereby accelerating the process. Moreover, Table 11 shows that the additional overhead from multiple intermediate encoding and decoding operations is also relatively minor compared to the total generation cost.
Further efficiency comparison across different models. To provide a more comprehensive assessment of model efficiency, we further report the NFE and FLOPs of different models when generating single image at resolutions of 2048 2048 and 4096 4096. Tables 12 and 13 show
22

---

that RepLDM significantly reduces the NFE and FLOPs required for inference by decreasing the number of denoising steps high resolutions, thereby substantially reducing the time needed to generate HR images.
Table 12: Inference cost of generating 2048 2048 Image for different models.
Model
SDXL [36]
MuliDDni
ScaleCrafter
HiDiT, [56]
UG [2]]
DenoFusion
AcdDil [28]
RMpLDM
TFLOPs
3010
5420
2437
1857
3608
9015
8597
1140
Time
3,0
1.8
3.0
0.6
Table 13: Inference cost of generating a 4096 x 4096 Image for different models.
Model
SDXL [36]
Mulnixif: [i]
ScaleCrafter [14]
HiDiat. [56]
UG [2]]
DemoFusion
AccDidt. [28]
Repl.DM
NFF
50
30
50
50
200
200
63
TFLOPss
12026
29566
9759
5211
12624
73167
74225
7140
5.7
Qualitative analysis on the progressive upsampling stage. To clearly illustrate the progressive upsampling process of RepLDM, we set 1/2 = [0.2, 0.2, 0.2] to generate 4096 x 4096 images. As shown in Fig. 22, the images generated at different sub-stages RepLDM exhibit high degree of consistency, with only minor differences in details. Since our task focuses on generating HR images rather than traditional image super-resolution, these differences details are reasonable. As discussed in Table 10 and Table 11, for each denoising step, the time required for HR images is several times that for low-resolution images. Consequently, repeating full denoising process high resolution is extremely consuminn [6, 28]. Considering that HR and low-resolution images should share the same low-freque frequeccyncy structure, and that DMs naturally generate low-frequency structures first. during denoising [44, 53], Repl.DD leverages the prior knowledge of low-frequency structures in low-resolution images, thereby effectively accelerating the generation process.
Prompt-
now dloan
(yae card rock thick
senceed on svartan
thaie enamaai you
bocigraan buvtt lighning
dorpartion by tigh-
Aderrinn Saage 1228-124
PrOgeEt LORRYEIKE SESeROE
Trgpede S.D4Rge
Progestte Uppwrs Sis-enge
1500-1122
23.8 12
10 Jostas r92
Figure 22: Illustration of the progressive upsampling generation process. The inference speed is evaluated on single 3090 GPU.
F RepLDM Algorithm
The implementation details of Repl.DM can found in Algorithm 1, and further information available in our code repository.
23

---

Algorithm RepLDM Inference Pipeline
Require:
The number of inference time steps of the first stage To; progressive scheduler qu; attention guidance
scale 7y; attention guidance delay rate nn the decay factor B: target image size tuple (H', W"): ): the denoising
model denoising model's s training resolution tuple (H, W); VAE encoder E: VAE decoder noise scheduler's slyper-paramete list dr1:Tp-
I: Initialization:
2: STo = (0, I) (Sampling from Standard Gaussian Distribution)
(0)
3: nanges = (ength(rra) + [Get the total number of denoising stages)
4: = w (Keep the aspect ratio and number of pixels unchanged)
H''
5: H( = ceil(v x W x
6: Wx(0 ceill V
HxW
7: H (()) H'
8: W(()) W
9: areaja = linspace (T)( (7)
W()), HH(*) x w manges) (Upsampling according to number of pixels)
10: HAS [ceil( x)))( for i in areaus) (Get the height and width of each stage)
11: WISE = (ccil( (ir') for i in oreaus]
12: Kacnaising = [To] (Get the number of denoising steps for each stage)
13: Kaeeoising..
:ammiiingeeteend extend(
17])
= To x 171 (Obtain the number delay steps)
eoae T- +1+ y
15: Mist (
for i= 1, &] (Obtain the guidance scale for each step)
16: Denoising:
17: for O,..., Rstages do
18: Phanps Raznaisin [s]
if then
20:
n upsample(x Hss[s), Wisx(s) (Upsampling pixel space)
21:
(x)(t))
22:
N( ā Panpps 20 (1 ā (meeps))
23:
end
if
24:
for
Thaqps 1, 0 do
25:
+ ((z(1))
(x)
1) [Deooising]
26:
if s
== and l I
1: then
27:
(s) Ma (FSA((2)))) (1 [+) (Attention Guidance
28:
end if
29:
end for
30: (+) + -22)") (Ohtain the pixel space image)
31: end for
G Robustness Analysis
In this section, we conduct a robustness analysis to complement the experiments in $4.2, providing a more comprehensive evaluation of the models' performance. Our robustness analysis is conducted from two perspectives: (i) we vary the random seeds and repeat each experiment three times to compute the mean and standard deviation of all results; (ii) we randomly sample 20k HR images from the HR subset of LAION-5B dataset [41] to construct a new benchmark for evaluating the models' generalization performance. Since HR generation requires substantial computational resources, analyze the four rest-performing models from Table 1, i.e., HiDiffusion, DemoFusion, AccDiffusion, and RepLDM.
Analysis on the SAM benchmark. We maintain the exact experimental settings in $4.2 and conduct the analysis at resolutions of 2048 x 2048 and 4096 x 4096. Table 14 shows that Repl.NM .D continues to exhibit superior performance across the repeated experiments.
Table 14: Robustness analysis on the SAM benchmark. The best results are marked bold.
Method
2018
2048
4008
1006
FID
FID,
18.
CLIP
FID
15
RID, L
15,
CLIPT
IRDxi. [56]
80.291057
17.1810.10
63.55.1033
15.2610.66
24.95.0.04
164:24.0.84
12.7110.14
H6.6210.32
7.48.10.28
21.1810005
DemoRl
713991060
22.1010.37
53.5810:22
19.2110277
25.21:0011
101.83:0.49
20.5110.11
63,6010.66
149221124
24.55:0000
AedDell (18)
71.37.1048
21.2110.32
53.0610.33
1924111.22
25.13.10011
102.41.1..00
19.881.024
65.86.0.17
1273.1071
24.65.0022
RD
6688-002
22,13:0.74
47.31:0.11
20.38+2.08
25.30-0.12
0146:0061
21,63:0466
58,93-0.00
1502:0016
24.04+0.00
24

---

Analysis on the LAION-5B benchmark. Considering that only 1K samples were used for the 4096 4096 resolution in $4.2, which may lead unstable metric evaluations, we double the number of samples to 2k for this resolution in the current experiment. Regarding evaluation metrics, since IS may lead to high variances beyond ImageNet, we follow some recent studies and adopt Kernel Inception distance (KID) for more accurate evaluation [20, 37]. Table 15 shows that on the LAION benchmark, RepLDM still demonstrates superior performance, surpassing previous SOTA models across all metrics.
Table 15: Robustness analysis on the LAION-5B benchmark. The best results are marked in bold. Since the magnitude of KID is relatively small, we multiply its mean and standard deviation by 10
2048 2048
4006
4090
Method
FID,
KID,
CLIPT
KIDJ
HID,
KID,
CLIPT
48.17=0.41
8.0660.0
36.26:11.37
105340111
23.1620.05
9281=0.75
35.36410.00
12002:10
103.5=00.27
18:5560066
Demol: [6]
34.15_0.31
4,5010.05
21.3811.117
6,0110066
25.48.1002
37.0320.27
5.7110.14
30.77.0.336
16.12.0.222
25.1210.04
DcdDiff (35)
34.49_0.11
4,9210.08
22.7110.117
85710.11
249000002
38.5620.25
7.211.0200
38.85:10229
20.8710:200
26.46.0.01
RMpLDM
04.08.0:25
4.1810.04
20.3010.300
4.8710.13
25.78.10.3
34.0110.25
4.13100.88
23.08.0..66
12.08.00133
25.88.0..44
H Comparative and Ablation Analysis Based on StableDiffusion 2.1
H.1 Comparison Experiments
To validate the generalization capability of RepLDM, we conducted extensive quantitative and qualitative analyses using StableDiffusion 2.1 (SD2.1) as the pretrained base model.
Qualitative comparison. Fig. 23 presents the results of the qualitative comparison. It can be observed that, when generating high-resolution images, SD2.1 also faces issues with repeti- tive object structures. ScaleCrafter often hibits structural collapse during denoising with SD2.1, resulting in suboptimal performance. In contrast, RepL.DM consistently produces high- quality results across all resolutions, highlight- ing the generalizability of the RepLDM gener- ation framework.
(a) SD2.1
(b) ScaleCrafter
(c) RepLDM
1535x1536: A fierce lion roaring the savannah.
1024x2048: A majestic elephant the wild.
Quantitative comparison. Since the code for using SD2.1 the pretrained model in AceDif- fusion and DemoFusion is not publicly available, compare RepLDM with ScaleCrafter. We compared the model performance at four resolu- tions: 1536 1536, 1024 2048, 2048 1024, and 2048 x 2048, Considering that SD2.1's gen- cration capabilities are weaker than SDXL, we set = [0.2, 0.2, 0.3] for the experiments in this section, while keeping other settings consis- tent with $4.
Table 16 presents the results of the quantitative comparison, showing that RepLDM maintains strong performance when using SD2.1 the pre-trained model. In contrast, ScaleCrafter per- forms suboptimally, as it tends to produce struc- tural collapse in the generated images, a phe- nomenon that more apparent in the qualitative analysis.
2048:1024; A sports car speeding down an empty highway,
2048x2048: The sun rises, costing worm glow the land.
Figure 23: Qualitative comparison using SD2.1 as the pretrained model.
H.2 Ablation Study on Attention Guidance
Quantitative ablation. Table 17 shows the results of the quantitative ablation on attention guidance using SD2.1 as the pretrained model. can be observed that attention guidance leads improvements in metrics. These improvements more evident in the qualitative ablation analysis.
25

---

Table 16: Quantitative comparison results based on SD2.1. The best results marked in bold.
1036
1535
1024 x 2045
2018
1924
2018 x 2045
F1D
FID.
15.
F10
FID.
FID, 15,
SoleCn/tr
Repl.DM0
60.3
21.
916
184
914
Table 17: Quantitative ablation results based on SD2.1. The best results are marked in bold.
x
15,
HID:
F1D.
FID.
w/o
61.2
20.9
30.2
25.3
24.9
17.0
61.1
21.2
182
25.
NG
69.3
21.0
30.5
18.3
25.4
61.1
19,9
54.1
18.4
25.0
63.7
19.2
50.4
18.2
24.7
60.5
21.5
48.8
172
25.3
Qualitative ablation. Fig. 24 presents the ablation analysis of attention guidance based on SD2.1. From the figure, it can be observed that attention guidance also enhances detail richness and color vibrancy when using SD2.1, further demonstrating its generalization capability.
majestic eagle
casticc
recks pent'y on surfoce
over
caln
posceful village
ciffs.
(6)
(b)
(d)
(e)
Figure 24: Ablation study of attention guidance using SD22. the pre-trained model. Resolution: 2048 x 2048.
I Attention Guidance Also Works in Other Generation Frameworks
In this section, we apply attention guidance to other generative frameworks to demonstrate its generalization capability. Specifically, apply attention guidance the generative frameworks of HiDiffusion and DemoFusion, and perform both quantitative and qualitative ablation studies.
I.1 Quantitative Ablation
In this section, considering the long inference time of Demoousion, we perform quantitative ablation studies attention guidance using the HiDiffusion generation frameworks at resolution of 2048 x 2048, All experimental settings are consistent with those $4.
Table 18 presents the quantitative ablation results using the HiDiffusion framework. It is evident that incorporat- ing attention guidance improves HiDiffusion across all metries. This is further corroborated by the qualitative analysis in Fig. 25, which demonstrates that attention guidance alleviates some of the structural collapses observed in HiDiffusion.
Table 18: Quantitative ablation of atten- tion guidance using HiDiffusion frame- works. The best results are marked in bold.
Method
FID
CLIPP
HiDiftusion [56]
81.0
16.8
64.1
14.2
24.9
HiDitTosion-AG
I.2 Qualitative Ablation
HiDiffusion--ateeniin guidance. We incorporate attention guidance into the generative framework of HiDiffusion. Fig. 25 (a)-(c) demonstrate that using attention guidance effectively mitigates issue of structural collapse in synthesized images. Fig. 25 (d)-(f) further show that attention guidance can also address the structural deformation inherent to HiDiffusion, enhance image details, and improve overall image quality.
26

---

The rivrr flws carthy
Feeera boon in abrant
Birds chipp norrin
vitrann underwater sooe.
tine flls 62 the
peacera valae the
HiDiffusion
HiDiffusion+ Attention Guidance
(b)
(d)
(f)
Qualitative ablation of attention guidance in the HiDiffusion Framework. All images have a resolution of 2048 x 2048. Figures (a)-(c) demonstrate that attention guidance can mitigate the issue of structural collapse in generated images, while Figures (d)-(f) show that attention guidance resolves structural deformation issues and enhances image details.
Figure 25:
incorporate attention guidance into the generative frame- work of DemoFusion. As shown in Fig. 26 (a)-(c), attention guidance effectively mitigates the issue of repetitive structures in DemoFusion. Fig. 26 (f)-(f) further illustrate role of attention guidance in enriching image details and enhancing overall image quality.
RemoFusion+atteniie guidance.
meon casts
roboit through
shepperd
Agentll brecae
ciyscope right
flass gest
Light
DemoFusiog
DemoFusion+ Attention Guidance
(c)
(d)
(e)
Figure 26: Qualitative ablation of attention guidance in the DemoFusion Framework. All
images have a resolution of 2048 x 2048. Figures (a)-(c) demonstrate that attention guidance effectively mitigates the issue of repetitive structures in images, while Figures (d)-(f) showcase attention guidance's ability to enrich image details.
J Ruser-Re Images Tend to Lack High-Resolution Details
To explain why using super-resolution models to obtain HR images is sub-optimal, in this section, we conduct both qualitative and quantitative comparisons between RepLDM and the super-resolution results. Specifically, we use BSRGAN [54] to upsample the generated results of SDXL [36] at its training resolution.
As shown in table. 19, the susortiesolution model (SDXL BSRGAN) demonstrate comparable performance in quantitative experiments, a phenomenon also observed in DemoFusion's experiments. This is because super-resolution models can least preserve low-frequency structures of images without significant errors. However, quantitative metrics such as FID and IS, are widely recognized insufficient for comprehensively evaluating the performance of model's generation. As result, user studies are commonly employed to provide human-level
Quantitative results.
27

---

sports car speeding down on enppy highway.
beautiful mwon glickng
serene desert lontecops with sand cuncs.
gentle nustles
peoceful Zen garden.
a pond.
leoves
S BSRGAN
RepLDM
(a)
(b)
(c)
(d)
(e)
Figure 27: Qualitative comparison with SDXL+BSRGAN, The prompts for the generated images are provided above the figures. The resolution of (a) (c) are 2048 x 2048, and the resolution of (d) and (e) are 4096 4096.
evaluation with more intuition [11, 14, 22, 30, 36, 39, 55]. For example, in ScaleCrafter [14], they conducted both quantitative and user study analyses comparison with the SD+SR approach. Their results show that, although ScaleCrafter performs worse than SD+SR quantitative metrics, users significantly prefer the textures and details generated by ScaleCrafter. One important reason that the goal of the SR model is to produce images consistent with the input, which limits its performance in high- righuureoottion generation needing more detail for true high-resolutinn visuals beyond simple smoothing [6, 10, 14, 22, 27, 28, 57].
Table 19: Quantitative comparison results between RepLDM and SDXL The best results marked in bold.
Method
Fm
2005
FR2.
2018
15.
2015
FIT.
0006
FMII
15
10W
2015
C1P
FMI1
(UWN 1006
FITS,
15.
UTTP
SOXLABSRGAN
66.2
21.1
47.5
166
257
$0.7
19.8
50.3
12.3
25.1
92.7
17.6
57.9
12.1
34.9
90.8
20,9
56.8
13.8
25.3
ReyLDM
66.0
21.0
474
175
25.1
39.0
20.3
56.0
1900
25.0
93.2
19.5
56.9
16.5
34.9
90.6
21.1
59.0
143
24,5
Qualitative results. As shown Fig. 27, compared to RepLDM, SDXI+BSRGAN, while maintaining decent image structure, fails to generate the level of detail expected from HR images. The absence of these details sometimes leads to the model's inability to simulate realistic scenes. For example, in Fig. 27 (c), SDXL+BSRGAN fails to generate realistic shadows.
K Memory Usage Analysis
We compare the GPU memory usage required by the models. Specifically, we test the minimum GPU memory requirements during model inference based the model's open-source code. Table 20 shows the resource consumption of different models when generating images at various resolutions.
Table 20: Model Memory Usage (GB). The best results are marked bold, and the second best results marked by underline.
Resolutions
4096
4096
SDXL [36]
15.9
16.1
16.6
MuliDiff.
22.0
16.8
16.8
ScaleCrafter [14]
17,4
17.6
19.1
UG [2]]
DemoFusion [6]
15.2
18.4
16.8
DieDiff [28]
22.1
22.1
HiDiIT [56]
23.9
16.2
16.2
RepLDM
16.0
21.1
23.8
is worth noting that for HR image generation tasks, memory bottleneck Ties in the encoding and decoding of the VAE rather than interpolating the image in pixel space. To address the challenges of
28

---

encoding and decoding HR images, researchers typically employ tiled encoders and tiled decoders. In this work, we also utilize a tiled-encoder and decoder when generating hitra-high-resolution images. allowing us to generate images with resolutions up to 4096 x 7280 or higher on a 24GB VRAM NVIDIA 3090 GPU (as shown in Fig. 1).
It is important to note that different models have undergone varying degrees of additional optimization in their official open-source implementations. Specifically, some open-source codes utilize existing optimization tools, such as accelerate [9] or Flash Attention [3], which provide additional advantages in terms of inference speed and memory usage performance. To ensure a fair comparison, in Table 20,
we did not use such additional optimizations in the implementation of RepLDM.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations in $6, analyze their underlying causes, and discuss potential directions for future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
30

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We provide detailed description of the methodology and its underlying ideas in $3. addition, we present the full algorithmic pipeline using pseudocode in Appendix We also commit to open-sourcing the code of our method.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
31

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: In Appendix F, we detail the method using pseudocode. Additionally, upon acceptance. we will clean and release our code base and share it on GitHub. All data used in this paper belongs to existing open source datasets and have been correctly cited to ensure reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips. ctt.ps.ccppublcc/// baticc/giiiee guwdee/ide//// CCoeeSumeeeeaeeeeeeee woabbbmmsssion ssssnnolll.. onoolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The proposed method requires no training. We provide a detailed explanation the inference and evaluation settings in $4.1. We determine the hyperparameters through ablation studies, with details provided in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We repeated the experiments in Table in Appendix G, computing the mean and standard deviation, and also conducted additional replication experiments on the LAION dataset [41].
Guidelines:
The answer NA means that the paper does not include experiments.
32

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the resources needed to reproduce the experiments in $4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / http//euurrrreeeeeeeeee httpp////ppppcttcceeeeeeiiieeeee
Answer: [Yes]
Justification:
We conform to the NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: the proposed method builds upon pretrained generative model to produce righernne images in training-free manner, and thus does introduce any addi- tional specific societal impacts.
33

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our method does not rely any specific publicly released model and requires no specialized fine-tuning, and therefore does not necessitate additional safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-us, should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all used resources such implementations of baselines and data. We release our work with CC-By 4.0 license.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
34

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
This study does involve the release of any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: In the user study conducted in this work, we recruited volunteers to evaluate the quality of the generated images. Detailed instructions were provided to the participants. As the participants were volunteers, no compensation was involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: This study only required volunteers to evaluate the quality of generated images, and therefore poses no particular potential risks. Furthermore, protect the privacy of participants' preferences, all responses were anonymized and randomized, and no personal information collected.
35

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https: pettps://neeet /tt///eerrs..//e/ee theuurpps.cc/Cooereeeeee eeee/ooeeeeecce/2225//////////// for what should or should not be described.
36


Reviews:
[{"review_id": "nQhLWsxRmG", "reviewer": "Reviewer_46id", "summary": "This paper introduces RepLDM, a novel training-free framework for adapting pretrained latent diffusion models (LDMs) for high-resolution image generation. The method tackles the common problem of structural distortion when generating images at resolutions beyond the model's training data. RepLDM works in two stages: (1) an Attention Guidance stage that uses a novel parameter-free self-attention (PFSA) mechanism to generate a structurally-consistent latent at the base training resolution, and (2) a Progressive Upsampling stage that iteratively increases the image resolution in pixel space to avoid artifacts common to latent-space upsampling. By starting the upsampling process with a high-quality latent, the method requires significantly fewer denoising steps at high resolutions, leading to major gains in both image quality and inference speed.", "questions": "- Your formulation for PFSA effectively sets the Query, Key, and Value representations equal to the same feature representation $f(z)$. This is a fascinating simplification. Have you considered or experimented with asymmetric variants within the parameter-free constraint? For example, could one use the raw feature $f(z)$ for the Query, but a structurally-simplified version (e.g., from a blurred or pooled feature map) for the Key and Value? My hypothesis is that this might allow the model to more explicitly attend *from* fine details *to* coarse structures.\n\n- I recently came across a work titled \"Swift parameter-free attention network for efficient super-resolution\" which also explores parameter-free attention mechanisms before. Could you clarify how your Parameter-Free Self-Attention (PFSA) mechanism relates to or differs from the approach in that work? This would help to better position your contribution within the existing literature.\n\n- You note that directly applying PFSA can lead to unstable denoising, which motivates the linear interpolation with the original latent $z$ in \\textbf{Eq. 2}. Could you offer more insight into this instability? For instance, does $PFSA(z)$ produce a latent with a significantly different statistical distribution (e.g., higher variance) compared to $z$? Is the primary role of the $(1-\\gamma)z$ term to act as a statistical anchor to keep the guided latent on-manifold?\n\n- The use of a cosine annealing schedule for $\\gamma_{t}$ is a well-motivated choice, aligning guidance strength with the diffusion model's tendency to generate structure first. How sensitive is the final image quality to the specific functional form of this schedule? Did you experiment with simpler schedules, like linear or step-wise decay, and did they yield comparable results?\n\n- The experiments are thoroughly performed using SDXL as the base model. How well do the RepLDM framework and its associated hyperparameters ($\\gamma$, $\\eta_{1}$, $\\eta_{2}$) generalize to other popular pretrained LDMs, such as the Stable Diffusion v1.5/v2.1 series or various community fine-tuned models?", "limitations": "Yes. The authors provide a clear and well-written \"Limitations and Future Work\" section (Section 6). They responsibly acknowledge the method's inability to fix inherent flaws in the base model (like text generation) and discuss the computational scaling challenges when targeting ultra-high resolutions. This transparent reporting strengthens the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Novelty and Elegance of Core Theory (PFSA): The paper's central contribution, the \\textbf{Parameter-Free Self-Attention (PFSA)} mechanism, is exceptionally elegant. The formulation in \\textbf{Eq. 1} ($PFSA(z) = f^{-1}(\\text{Softmax}(\\frac{f(z)f(z)^{T}}{\\lambda})f(z))$) cleverly strips the standard self-attention mechanism of its learnable projection matrices ($W_{Q}, W_{K}, W_{V}$). This design is based on the strong and insightful premise that the global spatial modeling capability of attention is inherent to its structure, not its learned parameters. This simplification provides a powerful, first-principles approach to enhancing structural consistency in a training-free manner.\n\n- The decomposition of the problem into an \"Attention Guidance\" stage and a \"Progressive Upsampling\" stage is logical and effective. This design smartly separates the task of enforcing global structure (at the native resolution) from the task of adding details at higher resolutions. The framework is clean and addresses two distinct failure modes of naive high-resolution generation.\n\n- The authors' identification of latent-space upsampling as a key source of artifacts is a critical insight. Their pilot study (Fig. 4) provides clear empirical evidence for this hypothesis. The decision to perform upsampling in the pixel domain, as formalized in Equation 4, $\\hat{z}_{0} = \\mathcal{E} \\circ \\mathcal{U} \\circ \\mathcal{D}(z_{0})$, is a simple but highly effective solution that directly mitigates this issue, demonstrating a deep understanding of the VAE's properties and limitations.\n\n- The method delivers on its promises, significantly outperforming strong recent baselines like DemoFusion and AccDiffusion across multiple metrics (FID, IS, CLIP Score) and resolutions. The ~5x speedup in inference time is a massive practical advantage that makes high-resolution generation far more accessible. The user study further corroborates the superior quality of the generated images.\n\nWeaknesses\n- Nuance in \"Parameter-Free\" Claim- While the core PFSA block is indeed parameter-free, the overall attention guidance mechanism introduces several key hyperparameters that require careful tuning. The linear interpolation in Equation 2 depends on the guidance scale $\\gamma$, and the adaptive schedule in Equation 3 introduces a decay factor $\\beta$ and a delay rate $\\eta_{1}$. While these are not learned weights, they represent a set of sensitive parameters that are crucial to the method's success, making the \"parameter-free\" description of the overall guidance slightly imprecise.\n\n- As the authors rightly point out in Section 6, the method is a \"reprogramming\" framework, not a fine-tuning one. As such, it cannot fix fundamental flaws of the base LDM, such as the difficulty with generating coherent text. This is an inherent limitation of the approach.", "was_revised": true, "final_justification": "Clarifications on the “parameter-free” terminology, additional experiments exploring asymmetric PFSA, and the new annealing-schedule and stability analyses directly address my main questions. I appreciate the side-by-side comparisons (e.g., Table A-sym PFSA) and the evidence of hyper-parameter robustness across SD v2.1, ControlNet, DemoFusion, and HiDiffusion. All remaining issues are minor (terminology and small wording edits). I therefore maintain my overall positive assessment and still recommend acceptance. I look forward to seeing the updated manuscript incorporating the promised revisions and additional results.", "rebuttal": "We are grateful for your positive summary and constructive suggestions. We have tried our best to address your concerns as follows.\n\n > **W1:  On the definition of “parameter-free”**\n\nGood catch! In our paper, the term “parameter-free” in PFSA was intended to emphasize the absence of learnable parameters, in contrast to vanilla self-attention with learnable parameters. We appreciate your suggestion and will adopt a more accurate terminology in the next version by replacing *Parameter-Free Self-Attention (PFSA)* with *Training-Free Self-Attention (TFSA)*.\n\n>**W2: On the fundamental flaws of the base LDM**\n\nWe would like to clarify this point from three perspectives.\n- We would like to note that training-free frameworks are inherently limited by the capabilities of their underlying base models, which is a widely observed issue. To the best of our knowledge, existing training-free generation frameworks have not effectively addressed this limitation. Some efforts to address the limitations of base models, such as improving fine-grained text generation, rely on fine-tuning the base model with specific datasets [Ref1,Ref2,Ref3].\n- Although fine-tuning the base model can alleviate some of its inherent limitations, it may come at the cost of degrading its generative quality. In comparison, the reprogramming framework better retains the base model’s original generation capacity.\n- Our proposed RepLDM is a general framework for high-resolution generation. In Section C of the supplementary material, we demonstrate that RepLDM is also compatible with different generation conditions. This suggests that RepLDM can be applied to alternative, improved base models to circumvent certain limitations of the original base model.\n\n[Ref1] AnyText: Multilingual Visual Text Generation And Editing. ICLR2024.\\\n[Ref2] TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering. ECCV2024.\\\n[Ref3] TextDiffuser: Diffusion Models as Text Painters. NeurIPS 2023.\n\n> **Q1: On the different PFSA calculation paradigm**\n\nFor PFSA, our objective is to remove the learnable parameters from the Self-Attention mechanism, while maintaining its computational paradigm as unchanged as possible. Therefore, we opted to use symmetric $Q$ and $K$ matrices for computing PFSA. We appreciate your suggestion of using asymmetric $Q$ and $K$ matrices for PFSA computation, which we find to be a very interesting idea. To verify its effectiveness, we conducted additional experiments accordingly. Specifically, we apply a 2 $\\times$ 2 pooling operation to downsample the $K$ and $V$ matrices. This ensures that the output of $\\text{Softmax}(QK^T/\\sqrt{d})V$ remains of shape $(hw) \\times c$. We then use this new output tensor for guidance, while keeping all other hyperparameters unchanged. The results at 2048 $\\times$ 2048 resolution are shown in the table below (the best results are marked in **bold**):\n\n| Method | $\\text{FID}$ | $\\text{IS}$ | $\\text{FID}_c$ | $\\text{IS}_c$ | $\\text{CLIP}$ |\n| --- | --- | --- | --- | --- | --- |\n| w/o PFSA | 66.8 | 21.6 | 47.5 | 17.4 | **25.3** |\n| Asymmetric PFSA | 67.4 | **22.6** | 47.9 | **20.4** | **25.3** |\n| PFSA | **66.0** | 21.0 | **47.4** | 17.5 | 25.1 |\n\nQuantitatively, all metrics, except for FID, show improvement. Qualitatively, compared to PFSA, asymmetric PFSA results in fewer details but demonstrates more advantages in structural adjustments. Due to conference policies, we are unable to provide access to these qualitative results at this time; however, we will include these analyses and experiments in the next version of the paper.\n\nBy the way, we have considered alternative attention paradigms for guidance, such as parameter-free linear attention mechanisms. However, our experiments showed that using a linear attention mechanism for guidance directly leads to meaningless noise or entirely black images.\n\n> **Q2: On the difference between our PFSA and PFAM proposed by Wan et al.**\n\nWe appreciate your recommendation and have read the paper. In the research conducted by Wan et al., they proposed a parameter-free attention mechanism (PFAM) to replace self-attention for more efficient image super-resolution.\nPFAM differs from the conventional Self-Attention in that it does not compute the similarity between tokens. Instead, the attention map in PFAM is obtained by applying the proposed symmetric activation function to the output of the convolutional layer. The output of PFAM is the result of an element-wise multiplication between the attention map and the features.\n\nOverall, our PFSA differs from their attention mechanism (PFAM) in the following aspects:\n\n- Task objective: PFSA is designed for generative tasks, aiming to improve the quality of high-resolution outputs. In contrast, PFAM focuses on super-resolution, where maintaining consistency between high- and low-resolution images is the primary goal.\n- Modeling approach: PFSA retains the computational paradigm of self-attention and can be considered a specialized form of self-attention. In contrast, PFAM does not compute token similarity; instead, it focuses on the magnitude of tensor elements. Conceptually, its attention mechanism differs quite significantly from that of Self-Attention.\n- Training strategy: PFSA operates without any training; although PFAM is parameter-free, it still requires joint training with convolutional layers to be effective.\n\n> **Q3: On the stability analysis of PFSA and the role of $(1 - \\gamma)z$**\n\nVery insightful comment. First, we would like to draw your attention to Section A of the supplementary material, where we provided a stability analysis of the PFSA.\nSpecifically: (**i**) We compute the deviation of token means and observe that PFSA exhibits a clustering effect, bringing semantically related tokens closer together (Fig. 12).\n(**ii**) We compute the standard deviation of latent representations and their Fourier amplitude spectra, and find that PFSA is capable of modulating both the amplitudes across different frequency bands and the standard deviation of latent representations (Fig. 13).\n\nThese analyses suggest that PFSA may operate by modulating the distribution of latent representations. To make this effect more gradual and controlled, we introduce a guidance mechanism to adjust the representations progressively.\n\nYour observation is highly insightful — the use of a linear combination with $(1 - \\gamma)z$ serves to anchor the guided latent representations, helping to ensure that the latent representations remain on the data manifold. We will incorporate the explanation from the manifold perspective you mentioned in Section 3.2 of the next version of the paper.\n\n> **Q4: On the sensibility analysis of annealing strategy**\n\nGreat suggestion. Previous studies have shown that diffusion models first generate low-frequency structures and then generate fine details [Ref4,Ref5,Ref6]. Based on this observation, we introduce the annealing strategy to gradually decrease the structural guidance effect of PFSA. Considering that cosine annealing has demonstrated robustness in various applications (e.g., learning rate decay), we employ cosine annealing in RepLDM.\nWe appreciate your constructive suggestion. Following your recommendation, we performed additional experiments using linear and exponential annealing strategies at 2048 $\\times$ 2048 resolution to investigate the sensitivity of Attention Guidance to the choice of annealing strategy.\nThe results are shown in the table below:\n\n| Method | $\\text{FID}$ | $\\text{IS}$ | $\\text{FID}_c$ | $\\text{IS}_c$ | $\\text{CLIP}$ |\n| --- | --- | --- | --- | --- | --- |\n| Linear | 66.2 | 21.5 | 47.2 | 20.3 | 25.4 |\n| exponential | 66.8 | 21.8 | 47.0 | 16.3 | 25.3 |\n| cosine | 66.0 | 21.0 | 47.4 | 17.5 | 25.1 |\n\nQuantitatively, comparable performance is observed across different annealing strategies. Qualitatively, the generated results also exhibited comparable improvements in structure and details. This suggests that Attention Guidance is not sensitive to a specific annealing strategy. Due to conference policy restrictions, we are unable to provide access to the qualitative analysis at this time, but we will include these analyses in the next version of the paper.\n\n[Ref4] FreeEnhance: Tuning-Free Image Enhancement via Content-Consistent Noising-and-Denoising Process. ACM MM2024.\\\n[Ref5] Relay Diffusion: Unifying diffusion process across resolutions for image synthesis. ICLR2024.\\\n[Ref6] FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model.\n\n> **Q5: On the generalization of hyperparameters to other pretrained LDMs**\n\nGreat questions! We would like to draw your attention to Section E of Supplementary Material. To verify the generalization of the hyperparameters, we conducted experiments using the guidance hyperparameter settings of SDXL to Stable Diffusion 2.1. The results demonstrate that the hyperparameters used for SDXL are also applicable to SD2.1. \nFurthermore, we tested the same settings on ControlNet and other training-free generative frameworks (HiDiffusion, DemoFusion), with all results confirming the robustness of this hyperparameter set."}, {"review_id": "AdfB6S31pP", "reviewer": "Reviewer_6jTA", "summary": "The authors propose a method for sampling high-resolution images using pretrained diffusion models. Their method consists of progressively making the image larger as they perform the denoising process. Additionally, to better capture the image details they make use of a parameter free self attention operator that shows promise in improving sample quality.", "questions": "My biggest question is trying to understand what attention guidance is doing. Although the authors provided some discussion on the appendix, it is still not clear to me. For ease of this discussion lets consider a black and white image, therefore we only have one channel. Although I understand that this operation is done in the latent space the concept should be the same.\n\nWhen using attention guidance $$AG(v) = \\sigma(vv^T)v$$ the values of v become close to each other and not the other way around. Therefore when we perform the operation $$v  = \\gamma AG(v) + (1-\\gamma) v = v + \\gamma (AG(v) - v)) $$ the pixels with higher values (more black) become smaller and white values become smaller. Therefore the whole image is converging to a more smoothened gray image. \n\nAnalogously this operation in the latent space pulls the \"latent pixels\" closer to each other, which should reduce the details. Therefore that would suggest that there is something particular about the latent space that AG is using to make this work. Could you elaborate on this? \n\nAdditionally, it would be nice to provide a discussion on why do the numerical results are so similar while the qualitative samples look so different\n\nMy final question is regarding the experiments. It seems that you have limited compute and perhaps this could be a cause of issue, but FID and other metrics like IS have very high variance when using a small amount of samples like $1000$. Additionally it seems like the reference statistics are from the SAM dataset, which are not related to the aesthetics generation that SDXL and these models are trained on, therefore the results are a little hard to interpret. Although the qualitative examples do show lots of promise, it would be good to have this be a more robust metric", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- The paper provides a clear explanation of the method\n- The paper provides lots of experiments and comparisons \n\nWeaknesses\n\n- Could the authors define HR and TR. I believe I know there meaning but it would be good for the paper\n- The method requires decoding and encoding multiple times, which is not computationally ideal", "was_revised": true, "final_justification": "Sampling images from diffusion models in high dimensions has remained a very challenging problem. This paper presents a way to do it that qualitatively is producing remarkable results. Although some of the quantitative metrics don't seem to be done in the best way, I believe that it is a great first step towards sampling images in high dimensions.", "rebuttal": "Thank you very much for your insightful comments and suggestions. We try our best to address your questions as follows.\n\n> **W1: On the definitions of training resolution and high resolution**\n\nGreat suggestion. To improve clarity, we will refine the annotation of the abbreviations HR and TR, and include the following explanation as a footnote in the next revision: In this paper, training resolution refers to the resolution used during model training; high resolution refers to a resolution that significantly exceeds the training resolution, to the extent that the model cannot directly generate satisfactory results at that resolution.\n\n> **W2: On the efficiency analysis of intermediate encoding-decoding operations**\n\nTo evaluate the computational cost introduced by multiple encoding and decoding operations, we analyze the time consumption of each component in DemoFusion and RepLDM when generating images at the resolution of $4096 \\times 4096$.\nThe results are shown in the table below:\n\nTable 1: The time consumption of DemoFusion when generating 4096×4096 resolution images.\n| Metric | Denoise 1024 | Denoise 2048 | Denoise 3072 | Denoise 4096 | Decode 4096 | Total |\n| --- | --- | --- | --- | --- | --- | --- |\n| num of steps | $50$ | $50$ | $50$ | 50 | / | $200$ |\n| Time (s) | $12$ | $185$ | $480$ | $901$ | $106$ | $1684$ |\n\nTable 2: The time consumption of RepLDM when generating 4096×4096 resolution images. The intermediate encoding/decoding operations are highlighted in $\\underline{\\text{underline}}$.\n|  | Denoise 1024 | Decode 1024 | Encode 3304 | Denoise 3304 | Decode 3304 | Encode 4096 | Denoise 4096 | Decode 4096 | Total |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| num of steps | $50$ | / | / | $5$ | / | / | $10$ | / | $65$ |\n| Time (s) | $12$ | $\\underline{0}$ | $\\underline{12}$ | $20$ | $\\underline{64}$ | $\\underline{11}$ | $118$ | $106$ | $343$ |\n\nAs shown in Table 1, denoising at high resolutions is a time-consuming process.\nDemoFusion requires substantial generation time because it performs the full denoising process at high resolutions.\nTable 2 shows that RepLDM significantly accelerates generation by substantially reducing the number of denoising steps at high resolutions.\nThis is because RepLDM performs pixel-space upsampling through multiple rounds of encoding and decoding, producing high-quality low-resolution images that serve as better initialization. As a result, RepLDM can significantly reduce the number of sampling steps required for high-resolution generation, thereby accelerating the process. Moreover, Table 2 shows that the additional overhead from multiple intermediate encoding and decoding operations is relatively minor compared to the total generation cost.\n\n> **Q1: On the analysis of the effectiveness of Attention Guidance**\n\nVery insightful comment. To provide an intuitive explanation, we conduct additional experiments to illustrate the effect of Attention Guidance (AG). Specifically, given a clean RGB image $x$, we obtain its latent representation $z$ through VAE encoding.\nWe then apply varying levels of noise to $z$:\n$z_t = \\alpha_t \\cdot \\epsilon + (1 - \\alpha_t) \\cdot z$,\nwhere $t \\in$ {$1,\\ldots,9$}, $\\alpha_t = 0.1\\cdot t$, and $\\epsilon \\sim \\mathcal{N}(0, I)$ is standard Gaussian noise.\nFor $t=9,\\ldots,1$, we compute $z_t^{PFSA}=PFSA(z_t)$ and $z_t^{AG} = \\gamma_t \\cdot z_t^{PFSA} + (1 - \\gamma_t) \\cdot z_t$, where $\\gamma_9=1$ and gradually decays to $\\gamma_1=0$ following the cosine annealing schedule.\nThis operation reflects the progressively diminishing influence of Attention Guidance during the generation process.\nWe then decode $z_t$，$z_t^{PFSA}$, and $z_t^{AG}$ back to the RGB space, obtain $x_t$, $x_t^{PFSA}$, and $x_t^{AG}$, and compute their SSIM with respect to the original image $x$.\nThe results are shown in the table below:\n\nTable 3: The SSIM values of $x_t$, $x_t^{PFSA}$, and $x_t^{AG}$ with respect to the original image $x$. The best results are highlighted in **bold**.\n| t | 9 | 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| $x_t$ | 0.020 | 0.039 | 0.070 | 0.122 | 0.199 | 0.302 | 0.439 | 0.605 | **0.777** |\n| $x_t^{PFSA}$ | **0.038** | 0.066 | 0.083 | 0.106 | 0.110 | 0.120 | 0.153 | 0.205 | 0.279 |\n| $x_t^{AG}$ | **0.038** | **0.068** | **0.106** | **0.164** | **0.241** | **0.337** | **0.455** | **0.608** | **0.777** |\n\nIt can be observed that directly applying PFSA improves SSIM in the early steps of denoising, but leads to a drop in SSIM in the middle and later steps.\nThis explains why directly using PFSA could result in unstable denoising, as noted in Section 3.2.\nBy incorporating the annealed strategy of Attention Guidance, we can leverage PFSA to promote structural recovery while avoiding numerical instability, thereby improving the visual quality of the generated images.\nQualitative results demonstrate that Attention Guidance enables the more rapid removal of noise and the earlier unveiling of image structure. Due to policy constraints, we will provide the visual results in the next version of the paper. The results show that the smoothing effect of PFSA can suppress part of the noise, thereby enhancing structural similarity.\n\nA similar phenomenon can also be observed in a recent work, FreeU [Ref1], where image quality is improved by enhancing low-frequency components while suppressing high-frequency components. Interestingly, despite the suppression of high-frequency components, the generated images exhibit richer colors and details (as shown in their Figs. 10, 17, and 18). To sum up, these findings may suggest that reinforcing structural information during generation provides the model with a stronger prior for synthesizing the remaining fine details, thereby improving the overall visual quality of the images.\n\n[Ref1] FreeU: Free Lunch in Diffusion U-Net. CVPR2024.\n\n> **Q2: On the similar quantitative ablation results of Attention Guidance**\n\nThank you for your suggestion. We would like to point out that FID and IS quantify the statistical differences between two distributions [Ref1,Ref2,Ref3]. Since Attention Guidance mainly enhances visual quality by modifying the mid- and high-frequency components while preserving the low-frequency structure of the image, it has limited impact on the overall distributional statistics. Consequently, the resulting quantitative metrics remain similar.\n\n[Ref1] A Note on the Inception Score. ICML2018.\\\n[Ref2] Improved Techniques for Training GANs. NeurIPS2016.\\\n[Ref3] Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS2017.\n\n> **Q3: On the evaluation metrics and datasets**\n\nDuring our experiments, Laion-5B was removed from Huggingface due to potential content risks. We note that the SAM dataset is of high resolution (average: $3300 \\times 4950$). Although the released version is a downsampled version with the shorter side resized to 1500, the resolution still remains significantly higher than that of many existing vision datasets (e.g., COCO) [Ref1]. Therefore, we chose the SAM dataset as a substitute. For the evaluation metrics and the number of generated samples used to evaluate performance, we follow the settings used in several related works, where 1k prompts are used to generate $4096 \\times 4096$ images for evaluation [Ref2]. This choice is made due to the extremely high inference cost associated with generating high-resolution images.\n\nWe highly appreciate your constructive suggestions regarding the experiments. For a more rigorous comparison, we conducted additional experiments:\n\n- We recently discovered the availability of high-resolution Laion images on Huggingface and downloaded 20K images to serve as the benchmark.\n- To enable more stable evaluation, we doubled the number of prompts used to generate $4096 \\times 4096$ resolution images. Due to the substantial time required to generate 4096 × 4096 resolution images and current hardware constraints, we will provide the relevant experimental results in the next version of the paper.\n- For RepLDM, we performed multiple generations (3 times) at 2048×2048 resolution using different random seeds and computed the mean and standard deviation. Due to time and hardware constraints, we will conduct the same variance analysis for other models as well.\n- Regarding evaluation metrics, since Inception Score (IS) may lead to high variances beyond ImageNet, we follow prior work and adopt Kernel Inception Distance (KID) for more accurate evaluation [Ref6,Ref7].\n\nDue to time and hardware constraints, we have temporarily compared our RepLDM with the top three baselines from Table 1 of the paper: DemoFusion, AccDiffusion, and HiDiffusion.\nWe will continue to extend the experiments and include the results and analyses in the next version of the paper.\nThe results are shown in the table below:\n\nTable 4: **Quantitative comparison results at $2048 \\times 2048$ resolution**. The best results are marked in **bold**.\n| Model | $\\text{FID}$ | $\\text{KID}$ | $\\text{FID}_c$ | $\\text{KID}_c$ | $\\text{CLIP}$ |\n| --- | --- | --- | --- | --- | --- |\n| HiDiffusion | 48.73 | 0.0083 | 36.73 | 0.0111 | $23.19$ |\n| AccDiffusion | $34.91$ | $0.0050$ | $22.60$ | $0.0085$ | $24.93$ |\n| DemoFusion | $34.53$ | $0.0046$ | $21.36$ | $0.0067$ | $25.46$ |\n| RepLDM | **34.08**$\\pm$0.25 | **0.0042**$\\pm$0.0000 | **20.30**$\\pm$0.30 | **0.0049**$\\pm$0.0001 | **25.78**$\\pm$0.03 |\n\nTable 4 shows that RepLDM still achieves state-of-the-art performance on the Laion dataset, further highlighting its strong generalization capability.\n\n[Ref1] Segment Anything. ICCV2023.\\\n[Ref2] DemoFusion: Democratising High-Resolution Image Generation With No . CVPR2024.\\\n[Ref3] A Note on the Inception Score. ICML2018.\\\n[Ref4] ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models. ICLR2024.\\\n[Ref5] Demystifying MMD GANs. ICLR2018."}, {"review_id": "3y4PKx3CbI", "reviewer": "Reviewer_ZoSx", "summary": "The paper proposes a framework named RepLDM to reprogram pretrained latent diffusion models (LDMs) for generating high-resolution images without requiring additional training. The approach integrates a parameter-free self-attention mechanism to enhance the structural consistency of latent representations at the training resolution and progressive upsampling in pixel space to mitigate artifacts caused by latent space upsampling, improving both image quality and generation efficiency.", "questions": "1. PFSA's efficacy appears critically to be dependent on input latent quality.\n(a) How does performance degrade under structured noise?\n(b) Provide robustness analysis using metrics like SSIM drop ≥ 0.2 relative to baseline.\n\n2. The complexity of PFSA's self-attention operation poses significant bottlenecks at high resolutions. Please give the computational load analysis for this issue. \n\n3. For the progressive denoising with pixel space upsampling scheme, \n(a)Does it rely on fixed-size convolutional kernels (e.g., 3×3/5×5) for pixel-space operations? Will it introduce the resolution issues when using different kernels? \n(b) Did you explore integrating blind deblurring modules within the denoising steps?", "limitations": "Please see questions.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1.\tThe idea of reprogramming pretrained LDMs for high-resolution image generation is novel and resource-efficient.\n2.\tUpsampling in pixel space alleviates artifacts and allows for efficient denoising at higher resolutions, significantly reducing inference time.\n3.\tThe paper conducts comprehensive quantitative and qualitative experiments to demonstrate the effectiveness and efficiency of RepLDM compared to state-of-the-art baselines.\nWeaknesses:\n1.\tWhile RepLDM improves efficiency compared to other methods, generating ultra-high resolution images still requires more sub-stages in the progressive upsampling stage, potentially increasing inference time.\n2.\tThe paper does not report error bars or statistical significance tests for the experimental results, making it difficult to assess the robustness of the claims.", "was_revised": true, "final_justification": "After discussions with authors, I have decided to update the score.", "rebuttal": "Thank you very much for your positive comments and efforts in reviewing our manuscript. We try our best to address your questions as follows.\n\n> **W1: On the efficiency of ultra-high-resolution image generation**\n\nFor training-free image generation at ultra-high resolutions, the inference time is expected to increase further, a challenge commonly encountered by existing methods. However, our RepLDM still achieves a significant reduction in overall generation time compared to other frameworks, such as DemoFusion, despite employing a progressive upsampling sub-stage. This is because our RepLDM integrates an upsampling sub-stage within the entire denoising process over T timesteps, whereas in existing methods, the upsampling sub-stage is applied after each denoising process. We effectively reduce the number of denoising steps required at high resolutions, which are the primary source of computational overhead, as illustrated in Fig. 21 of the supplementary material.\n\n> **W2: On the error analysis of the quantitative metrics**\n\nThank you for the constructive suggestion. Due to the high computational cost of high-resolution image generation, repeating experiments to compute standard deviations is expensive. As a result, most prior works do not perform error analysis [Ref1-Ref6]. To achieve a more rigorous evaluation, we conducted additional experiments to analyze the variability of the results. Specifically, for each experiment, we performed multiple generations (3 times) with different random seeds, computed the evaluation metrics for each generation, and reported the mean and standard deviation. Due to limitations in time and computational resources, we first performed this error analysis for RepLDM at a resolution of $2048 \\times 2048$. We will extend this analysis to other resolutions and baseline models, and include the results in the next version of the paper.\nThe results are shown in the table below:\n\nTable 1: Quantitative comparison results. The best results are marked in **bold**, and the second best results are marked by $\\underline{\\text{underline}}$.\n| Model | $\\text{FID}$ | $\\text{IS}$ | $\\text{FID}_c$ | $\\text{IS}_c$ | $\\text{CLIP}$ |\n| --- | --- | --- | --- | --- | --- |\n| SDXL | $99.94$ | $14.22$ | $80.00$ | $16.91$ | $25.03$ |\n| MultiDiff. | $98.83$ | $14.54$ | $67.87$ | $17.14$ | $24.59$ |\n| ScaleCrafter | $98.18$ | $14.19$ | $89.71$ | $13.32$ | **25.37** |\n| UG | $82.22$ | $17.61$ | $65.81$ | $14.64$ | $25.52$ |\n| HiDiff. | $81.01$ | $16.79$ | $64.14$ | $14.15$ | $24.94$ |\n| DemoFusion | $72.29$ | $\\underline{21.62}$ | $53.49$ | $\\underline{19.11}$ | $25.21$ |\n| AccDiff. | $\\underline{71.63}$ | $21.00$ | $\\underline{52.65}$ | $17.00$ | $25.13$ |\n| RepLDM | **66.08**$\\pm$0.02 | **22.13**$\\pm$0.74 | **47.31**$\\pm$0.11 | **20.38**$\\pm$2.03 | $\\underline{25.30}\\pm 0.12$ |\n\nThe results show that FID, FID_c, and CLIP exhibit relatively small variance, while IS shows comparatively larger fluctuations. This may be attributed to the fact that IS is not well-suited for evaluation on datasets other than ImageNet [Ref7]. In our future work, we will adopt more reliable metrics, such as KID, to replace IS for more accurate evaluation [Ref8].\n\n[Ref1] DemoFusion: Democratising High-Resolution Image Generation With No. CVPR2024.\\\n[Ref2] AccDiffusion: An Accurate Method for Higher-Resolution Image Generation. ECCV2024.\\\n[Ref3] FouriScale: A Frequency Perspective on Training-Free High-Resolution Image Synthesis. ECCV2024.\\\n[Ref4] ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models. ICLR2024.\\\n[Ref5] Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis. NeurIPS2023.\\\n[Ref6] HiDiffusion: Unlocking Higher-Resolution Creativity and Efficiency in Pretrained Diffusion Models. ECCV2024.\\\n[Ref7] A Note on the Inception Score. ICML2018.\\\n[Ref8] Demystifying MMD GANs. ICLR2018.\n\n> **Q1: On the efficacy of PFSA**\n\nThank you for your suggestion. We would like to note that PFSA itself does not contain any parameters, and thus its efficacy is not affected by the input. Notably, PFSA influences the efficacy of Attention Guidance through the parameter $\\gamma$, thereby altering the generation results. Therefore, $\\gamma$ is a key parameter that significantly impacts generation quality.\nFurthermore, for evaluation metrics, while the SSIM metric is frequently employed in image super-resolution, our study primarily focuses on text-to-image generation, where ground truth images are unavailable.\nIn this paper, we utilize commonly adopted metrics in generation tasks—FID, IS, and CLIP—to comprehensively assess image quality, consistent with previous methods. In addition, considering that our work targets high-resolution image generation, we follow prior works [Ref1,Ref2] and compute FID and IS on cropped patches of high-resolution images, denoted as $\\text{FID}_c$ and $\\text{IS}_c$.\nThe results in Table 1 demonstrate the robustness of our RepLDM.\nIn Section 5.1, we conduct both qualitative and quantitative ablation studies on $\\gamma$, and provide a more comprehensive analysis in Section F.1 of the supplementary material.\nOur ablation results suggest that, within a certain range, larger values of $\\gamma$ tend to introduce more fine-grained details and lead to better visual quality in the generated results.\n\n[Ref1] DemoFusion: Democratising High-Resolution Image Generation With No $$$. CVPR2024.\\\n[Ref2] AccDiffusion: An Accurate Method for Higher-Resolution Image Generation. ECCV2024.\n\n> **Q2: On the computational complexity analysis of PFSA**\n\nGreat suggestion. We would like to note that PFSA is only applied during the first stage of generation, and therefore does not introduce any computational or memory bottlenecks.\nAssume we have a high-resolution image $x$ with a resolution of $H \\times W \\times C$.\nwe encode the image $x$ into latent space and obtain latent representation $z \\in \\mathbb{R}^{h \\times w \\times c}$.\nBefore feeding $z$ into PFSA, we reshape it to a $(hw) \\times c$ matrice.\nThe computation of PFSA follows a formulation similar to that of self-attention:\\\n$\\text{PFSA}(z)=\\text{Softmax}(\\frac{zz^T}{\\sqrt{c}})z$\\\nThus, the computational complexity of PFSA is $O((hw)^2c)$.\nTaking SDXL as an example, the training resolution is $H=1024$, $W=1024$.\nAfter VAE encoding, $c=4$, $h=H/8=128$, $w=W/8=128$.\nFor each denoising step, the FLOPs of PFSA is approximately $2 \\times (h \\times w)^2 \\times c$, which is around 2.15 GFLOPs—negligible compared to the FLOPs of the denoising network (several TFLOPs per step).\n\n> **Q3: On the pixel space upsampling scheme**\n\n(a) The proposed RepLDM avoids artifacts caused by upsampling in the latent space by decoding latent representations into the pixel space for upsampling.\nIn practice, we use bicubic interpolation to perform this upsampling, which does not rely on a convolutional kernel of fixed size. Alternatively, a super-resolution network can be employed to accomplish this upsampling step, potentially leading to better generation quality.\\\n(b) We believe your proposed approach is insightful and highly feasible. We have previously considered a similar idea—training a neural network to directly perform upsampling in the latent space—which may align with your suggestion of using a blind model to mitigate the blurring artifacts caused by latent-space interpolation. We find your suggestion very constructive. To train such a blind model, a sufficiently large and high-quality high-resolution dataset would be required. Due to the limited time during the rebuttal phase, we will leave this as a promising direction for future exploration."}, {"review_id": "TbS8sRIQPL", "reviewer": "Reviewer_urNj", "summary": "This paper proposes a training-free image generation framework called RepLDM, which aims to generate high-resolution images while preserving global structural consistency. To achieve this goal, the authors introduce a parameter-free self-attention mechanism (PFSA) and a pixel-space-based Progressive Upsampling Stage. Both quantitative and qualitative comparisons, as well as ablation studies, demonstrate the effectiveness of the proposed method.", "questions": "The concerns are as described in the Weaknesses section. Due to the uncertainty regarding the effectiveness of PFSA, I am assigning a borderline score for now out of caution. If the authors can adequately address my concerns during the rebuttal phase, I would consider raising my score.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n(1) The paper proposes a plug-and-play, parameter-free self-attention mechanism (PFSA), which elegantly improves the global structural consistency of generated images at the cost of only a small increase in computational overhead. This effectively addresses a long-standing issue in Stable Diffusion when generating ultra-high-resolution images. If the method proves to be effective, it has the potential to become a general paradigm for high-resolution image generation tasks.\n\n(2) The proposed RepLDM framework requires no additional training and features a simple, intuitive pipeline, making it easy to reproduce.\n\nWeaknesses:\n\n(1) As mentioned in Strengths (1), PFSA could be a promising approach if it truly works. However, due to its highly counter-intuitive nature, I find it difficult to judge whether it is genuinely effective. Why would a simple post-processing self-attention operation significantly improve so-called global structural consistency? Although the authors provide some supporting references (e.g., [44, 52]) and ablation studies (Section 5.1), the evidence remains unconvincing. A more detailed analysis of PFSA is needed—such as numerical evaluation or visualizations—to help readers better understand its mechanism. Moreover, the concept of global structural consistency should be more clearly defined.\n\n(2) While the paper emphasizes high-efficiency in its title, the experimental section only compares inference time, which seems rather one-sided. A more comprehensive comparison including parameters, FLOPs, and NFE would provide a fuller picture of the method’s efficiency.", "was_revised": true, "final_justification": "We primarily focus on the effectiveness of PFSA, as it shows promise as a general plug-and-play method for improving image generation quality within the image synthesis community. The authors have provided thorough theoretical analysis and experimental validation of PFSA in both their rebuttal and the appendix. We believe our concerns have been addressed and the work meets the standard for acceptance. Therefore, we assign a score of \"5: Accept.\"", "rebuttal": "We appreciate your kind words and recognition of the significance of our contributions. We did our best to address your questions as follows.\n\n> **W1: On the validation of PFSA’s effectiveness**\n\nWe fully understand your concerns about the effectiveness of PFSA.\nTo address them, we have conducted the following efforts:\n\n- In Section A of the supplementary material, we provide extensive analysis of the underlying mechanism of PFSA, which we elaborate on in our responses to Q3 and Q4.\n- We incorporate PFSA into ControlNet (see Section C of the appendix), demonstrating its applicability under different generation conditions.\n- We apply PFSA to various high-resolution training-free generation frameworks, including DemoFusion and HiDiffusion (Section D of the appendix), and observe consistent improvements, highlighting its generalizability.\n- We also integrate PFSA into Stable Diffusion 2.1, showing that PFSA is compatible with different versions of SD models.\n- Finally, we commit that, if the paper is accepted, we will release the RepLDM codebase promptly to facilitate reproducibility and further research within the community.\n\n> **W2: On the detailed explanation of global structural consistency**\n\nGreat suggestion. In this paper, global structural consistency refers to the plausibility of the overall layout and the realism of object structures in an image. Specifically, a reasonable layout should follow logical spatial arrangements—for example, the sky should be above the ground. Realistic object structures should align with common sense, such as a cat having four legs instead of five. We will incorporate this explanation in the next version of the paper.\n\n> **W3: On the explanation of the effectiveness of Attention Guidance**\n\nTo illustrate how Attention Guidance enhances the global structural consistency of latent representations, we conducted extensive experiments.\n\n1. As shown in Fig. 12 of the supplementary material, we compute the deviation between the mean of each token and the overall mean across all tokens, and visualize this deviation. In columns (A) and (B) of Fig. 12, it can be observed that PFSA introduces pronounced stripe patterns in the token deviations, suggesting that it performs a clustering effect on tokens. Columns (C) and (D) present the heat maps of the deviation of tokens' mean, where it becomes evident that PFSA effectively aggregates semantically similar tokens, thereby enhancing the structural integrity and contours of the image.\n2. To provide a more intuitive explanation, we conduct additional experiments to illustrate the effect of Attention Guidance. Specifically, given a clean RGB image $x$, we obtain its latent representation $z$ through VAE encoding.\nWe then apply varying levels of noise to $z$:\n$z_t = \\alpha_t \\cdot \\epsilon + (1 - \\alpha_t) \\cdot z$,\nwhere $t \\in$ {$1,\\ldots,9$}, $\\alpha_t = 0.1\\cdot t$, and $\\epsilon \\sim \\mathcal{N}(0, I)$ is standard Gaussian noise.\nFor $t=9,\\ldots,1$, we compute $z_t^{PFSA}=PFSA(z_t)$ and $z_t^{AG} = \\gamma_t \\cdot z_t^{PFSA} + (1 - \\gamma_t) \\cdot z_t$, where $\\gamma_9=1$ and gradually decays to $\\gamma_1=0$ following the cosine annealing schedule.\nThis process reflects the gradually diminishing effect of Attention Guidance (AG) during generation.\nWe then decode $z_t$，$z_t^{PFSA}$, and $z_t^{AG}$ back to the RGB space, obtain $x_t$，$x_t^{PFSA}$, and $x_t^{AG}$, and compute their SSIM with respect to the original image $x$.\nThe results are shown in the table below:\n\n| t | 9 | 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| $x_t$ | 0.020 | 0.039 | 0.070 | 0.122 | 0.199 | 0.302 | 0.439 | 0.605 | **0.777** |\n| $x_t^{PFSA}$ | **0.038** | 0.066 | 0.083 | 0.106 | 0.110 | 0.120 | 0.153 | 0.205 | 0.279 |\n| $x_t^{AG}$ | **0.038** | **0.068** | **0.106** | **0.164** | **0.241** | **0.337** | **0.455** | **0.608** | **0.777** |\n\nIt can be observed that directly applying PFSA improves SSIM in the early steps of denoising, but leads to a drop in SSIM in the middle and later steps. This explains why directly using PFSA could result in unstable denoising, as noted in Section 3.2.\nBy incorporating the annealed strategy of Attention Guidance, we can leverage PFSA to promote structural recovery while avoiding numerical instability, thereby improving the visual quality of the generated images.\nQualitative results demonstrate that Attention Guidance enables the more rapid removal of noise and the earlier unveiling of image structure. Due to policy constraints, we will provide the visual results in the next version of the paper. The results show that the smoothing effect of PFSA can suppress part of the noise, thereby enhancing structural similarity.\n\nA similar phenomenon can also be observed in a recent work, FreeU [Ref1], where image quality is improved by enhancing low-frequency components while suppressing high-frequency components. Interestingly, despite the suppression of high-frequency components, the generated images exhibit richer colors and details (as shown in their Figs. 10, 17, and 18). To sum up, these findings may suggest that reinforcing structural information during generation provides the model with a stronger prior for synthesizing the remaining fine details, thereby improving the overall visual quality of the images.\n\n[Ref1] FreeU: Free Lunch in Diffusion U-Net. CVPR2024.\n\n> **W4: On the numerical analysis of PFSA**\n\nTo understand the effect of PFSA on the distribution of latent representations, we conducted a numerical analysis comparing latent features with and without the application of PFSA. Specifically, in Fig. 13 of Section A in the supplementary material, we compute the standard deviation and the Fourier amplitude spectrum of the latent representations with and without applying PFSA at different denoising timesteps.\nWe observe that during the early and middle stages of denoising, PFSA increases the variance of the latent representations and amplifies the high-frequency components. In contrast, during the later stages, PFSA suppresses high-frequency components and reduces the standard deviation.\nThis partially explains why attention guidance is more effective when applied during the early stages of denoising, but becomes less impactful in the later stages (as discussed in Fig. 9 of Section 5.1).\n\n> **W5: On the further visualizations for better understanding PFSA**\n\nAccording to your suggestion, we conducted an additional visualization experiment: We randomly selected a latent token as the query and visualized its attention weights as a heatmap over the generated RGB image to highlight the regions most related to the query token.\nThe results show that even without the linear projection layers for queries and keys used in conventional self-attention, the query token can still identify the key tokens that are most semantically relevant to the query token. This explains why PFSA is able to cluster semantically related tokens and enhance global structural consistency, even in the absence of learnable linear layers.\n\nUnfortunately, we were informed via email that external links are not permitted in the rebuttal, and we are therefore temporarily unable to show these visualizations. However, we will incorporate the experiment in the next version of our paper.\n\n> **W6: On the efficiency comparison across different models**\n\nGreat suggestion. To the best of our knowledge, existing training-free high-resolution generation frameworks [Ref1,Ref2,Ref3] evaluate efficiency solely based on inference time. This is primarily because these frameworks utilize the exact same pretrained model, ensuring identical parameters across comparisons. Nevertheless, we agree with your point that a more comprehensive set of metrics should be used to reflect efficiency. Following your suggestion, we have collected the NFE and FLOPs of different models when generating a single image at 2048 × 2048 and 4096 × 4096 resolutions. As all models employ training-free methods and have identical parameter counts, we do not report the number of parameters.\n\nTable 1: Inference cost of generating a 2048 × 2048 Image for different models.\n\n| Model | SDXL | MultiDiff. | ScaleCrafter | HiDiff. | UG | DemoFusion | AccDiffusion | RepLDM |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| NFE | 50 | 50 | 50 | 50 | 80 | 100 | 100 | 60 |\n| TFLOPs | 3010 | 5420 | 2437 | 1857 | 3608 | 9015 | 8597 | 1140 |\n| Time (min) | 1.0 | 3.0 | 1.0 | 0.8 | 1.8 | 3.0 | 3.0 | 0.6 |\n\nTable 2: Inference cost of generating a 4096 × 4096 Image for different models.\n\n| Model | SDXL | MultiDiff. | ScaleCrafter | HiDiff. | UG | DemoFusion | AccDiffusion | RepLDM |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| NFE | 50 | 50 | 50 | 50 | 80 | 200 | 200 | 65 |\n| TFLOPs | 12026 | 29566 | 9759 | 5211 | 12624 | 72167 | 74225 | 7140 |\n| Time (min) | 8.0 | 15.0 | 19.0 | 3.4 | 11.1 | 25.0 | 26.0 | 5.7 |\n\nTables 1 and 2 show that RepLDM significantly reduces the NFE and FLOPs required for inference by decreasing the number of denoising steps at high resolutions, thereby substantially reducing the time needed to generate high-resolution images.\n\n[Ref1] AccDiffusion: An Accurate Method for Higher-Resolution Image Generation. ECCV 2024.\\\n[Ref2] HiDiffusion: Unlocking Higher-Resolution Creativity and Efficiency in Pretrained Diffusion Models. ECCV2024.\\\n[Ref3] DemoFusion: Democratising High-Resolution Image Generation With No . CVPR 2024."}]

Author rebuttal:

Meta-review: All four reviewers recommend acceptance. The paper introduces a novel and efficient framework for high-resolution image generation from pretrained latent diffusion models. The work's central innovation is a novel, parameter-free self-attention mechanism which is both elegant and able to enforce structural consistency without retraining. Its logical two-stage design first learns the image's overall structure before progressively adding detail in pixel space, managing to avoid common artifacts. The proposed method results in significant improvements in both image quality and inference speed, outperforming state-of-the-art techniques with almost 5x speedup. The paper's impactful solution, clever design, and broad applicability make it a strong contribution that's worth spotlighting.
Decision: Accept (spotlight)