ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models
Authors: Yingqing He
Shaoshu Yang
Haoxin Chen
Xiaodong Cun
Menghan Xia
Yong Zhang
Xintao Wang
Ran He
Qifeng Chen
Ying Shan
Venue: ICLR 2024 spotlight (2024)
Keywords: text-to-image generation
diffusion models
high resolution generation
Primary area: generative models
TL;DR: A tuning-free method that can generate images with resolution of 4096x4096 based on pre-trained diffusion models, which is 16 times higher than the original training resolution.

Abstract: In this work, we investigate the capability of generating images from pre-trained diffusion models at much higher resolutions than the training image sizes. In addition, the generated images should have arbitrary image aspect ratios. When generating images directly at a higher resolution, 1024 x 1024, with the pre-trained Stable Diffusion using training images of resolution 512 x 512, we observe persistent problems of object repetition and unreasonable object structures. Existing works for higher-resolution generation, such as attention-based and joint-diffusion approaches, cannot well address these issues. As a new perspective, we examine the structural components of the U-Net in diffusion models and identify the crucial cause as the limited perception field of convolutional kernels. Based on this key observation, we propose a simple yet effective re-dilation that can dynamically adjust the convolutional perception field during inference. We further propose the dispersed convolution and noise-damped classifier-free guidance, which can enable ultra-high-resolution image generation  (e.g., 4096 x 4096). Notably, our approach does not require any training or optimization. Extensive experiments demonstrate that our approach can address the repetition issue well and achieve state-of-the-art performance on higher-resolution image synthesis, especially in texture details. Our work also suggests that a pre-trained diffusion model trained on low-resolution images can be directly used for high-resolution visual generation without further tuning, which may provide insights for future research on ultra-high-resolution image and video synthesis. More results are available at the anonymous website: https://scalecrafter.github.io/ScaleCrafter/

SCALECRAFTER: TUNING-FREE HIGHER-RESOLU- TION VISUAL GENERATION WITH DIFFUSION MODELS
Yingqing H, Shaoshu Yang Haoxin Chen', Xiaodong Cun?, Munn,Mnnhhan Xia?
Yong Zhang Xintao Wang2, Ran He?, Qifeng Chen Ying Shan
'Hong Kong University of Science and Technology
Chinese Academy of Sciences
TTeneent Al Lab
ABSTRACT
In this work, we investigate the capability of generating images from pre-trained diffusion models at much higher resolutions than the training image sizes. In addition, the generated images should have arbitrary image aspect ratios. When generating images directly at a higher resolution, 1024 x 1024, with the pre-trained Stable Diffusion using training images resolution 512 x 512, we observe per- sistent problems of object repetition and unreasonable object structures. Existing works for higher-resolution generation, such as attention- based and joint-diffusion approaches, cannot well address these issues. As new perspective, examine the structural components of the U-Net diffusion models and identify the crucial cause the limited perception field of convolutional kernels. Based on this key observation, we propose a simple yet effective re-dilation that can dynamically adjust the convolutional perception field during inference. We further propose the dispersed convolution and noise noise-damped classifier free guidance, which can image generation (e.g,, 4096 x 4096). Notably, our
enable seira-high-resoluute approach does not require any training or optimization. Extensive experiments demonstrate that our approach can address the repetition issue well and achieve state-of-the-ant performance
higher-resolution image synthesis, especially in texture details. Our work also suggests that a pre-trained diffusion model trained low-resolution images can be directly used for high-resolution visual gener- ation without further tuning, which may provide insights for future research on altra-high-resoluee image and video synthesis. More results available at the project website: https://yingginghe..oo wiw.eeggggguubbb.saeeeeaaaa
INTRODUCTION
In recent two years, the rapid development of image synthesis has attracted tremendous attention from both academia and industry, especially the most popular text-to-image generation models, such Stable Diffusion (SD) (Rombach et al. 2022). SD-XL (Podell al. 2023), Midjournyy (Mid), and IF (IF). However, the highest resolution of these models is 1024 X 102, which is far from the demand of applications such advertisements.
Directly sampling an image with a resolution beyond the training image sizes of those models will encounter severe object repetition issues and unreasonable object structures. As shown in Fig. [ when using a Stable Diffusion (SD) model trained on images of 512 x 512, to sample images of 512 1024 and 1024 x 1024 resolutions, the object repetition appears. The larger the image size, the more severe the repetition.
A few methods attempt generate images with larger size than the training image size of SD, e.g., Multi-Diffussee (Bar-Ta Tal et al. (2023) and SyncDiffusion (Lee et al. 2023). In Multi- Diffusion, images generated from multiple windows are fused using the averaged features across the windows
"quual Contribution
Corresponding Authors

---

Direct inference
Altering Attn-Se
MultiDiffusion
SyncDiffusion
Ours
512 x 1024
knight riding wooden horse
race course,
Gogh painting style"
"A photo teddy
riding bike in Time Square"
Figure 1: Structure repetition issue of higher-resolution generation (Train: 5122: Inference: 512x 1024 and 10243). Altering the scaling factor of attention (Jin et al. 2023). and joint diffusion approaches including MultiDiffusion (Bar-Tal et al. 2023) and SyncDiffusion (Lee et al. 2023 fails to address this problem. While our simple re-dilation successfully solves this problem and yields structure and semantic correct images, and at meanwhile require no optimization and tuning cost.
all the reverse steps. While SyncDiffusion improves the style consistency Multi-Diffusion by using anchor window. However, they focus the smoothness of the overlap region and cannot solve the repetition issue, as shown in Fig. [ Most recently, (Jin et al.) 2023) studies the SD adaptation for variable-sized image generation through the view of attention entropy. However, their method has a negligible effect on the object repetition issue when increasing the inference resolution.
To investigate the pattern repetition, we sample a set of images of 1024 and 512² from the pre- trained SD model trained with 5122 images for comparison. Zooming in on the images, we observe that the images of 1024 have no blur effects and the image quality does not degenerate like the Bilinear upsampling, though their object structures become worse. indicates that the pre-trained SD model has the potential to generate higher- highlriresolution images without sacrificing image definition.
We then delve into the structural components of SD to analyze their influence, e.g., convolution, self-attention, cross-attention, etc. Surprisingly, when we change convolution to dilated convolution in the whole U-Net using the pre-trained parameters, the overall structure becomes reasonable, i.e., the object repetition disappears. However, the repetition happens to local edges. We then carefully analyze where, when, and how to apply the dilated convolution, i.e., the influence of U-Net blocks, timesteps, and dilation radius. Based on these studies, propose tuning- dynamic re-dilation strategy to solve the repetition. However, as the resolution further increases (c.g., 16x), the de- creased generated quality and denoising ability arise. To tackle it, then propose novel dispersed convolution and noise-damped classifier- guidance for hltra-high-resoluttinn generation.
Our main contributions follows:
We observe that the primary cause of the object repetition issue is the limited convolutional receptive field rather than the attention token quantity, providing a new viewpoint compared to the prior works.
2

---

Figure 2: Our method can generate 4096 x 4096 images, 16x higher than the training resolution.
Based this observation, we propose the simple yet effective re-dilation for dynamically increasing the receptive field during inference time. We also propose dispersed convolution and noise-damped classifier-free guidance for ultra-bigh-resolutie generation.
We empirically evaluate our approach various diffusion models, including different ver- sions of Stable Diffusion, and a text-to-video model, with varying image resolutions and aspect ratios, demonstrating the effectiveness of our model.
RELATED WORK
2.1 TEXT-TO-IMAGE SYNTHESIS
Text-to-image synthesis has gained remarkable attention recently due to its impressive generation
performance (Saharia et al., 2022 Ramesh al. 2002]] Chang et al., 2023 Ding et 2022. Rom-
bach et al. 2022). Among various generative models, diffusion models are popular for their high- quality generation capabilities (Xing al. 2023 He al. 20226 Ho et al. 2022a He al. 2023)
Following the groundbreaking work of DDPM (Ho et al. 2222)) numerous studies have focused on
diffusion models for image generation (Nichol et al. 2022 Nichol & Dhariwal 2021 Kingma &
Ba 2014 Ho et al.) 2022b Gong et al. 2023 Ramesh et al. 2022 Gu et al.) 2022).In particular, Latent diffusion models (LDM) (Rombach et al. 2002) have become widely used, as their compact
latent space improves model efficiency. Subsequently. series of Stable Diffusion (SD) models are open-sourced, building upon LDMs and offering high sample quality and creativity (Ma et al. 2023]
Chen et al. 2023). Despite their impressive synthesis capabilities, the resolution remains limited to

---

Re-dilation dilated convolution
Dispersed
Quegud
Harnel h
1-1
2-
Neature
Training
Re dllation during inference
Siruclune- leval calibration
organa lost may
seotefed foot Intery, (h)
pon sonter
Output feeture
feetire
Output
Kameel
Disprren Rk
CoWW
4-2
Poud feature
Hout feature
Allgn scale
convolution
Figure 3: (a) The first shows re-dilation. Given pre-trained kernel trained on low-resolution data, we fix the parameters and insert spaces into kernel elements during test time. The second row shows fractional dilated convolution. For each entry of the convolution kernel, we compute the input feature with features near the kernel entry center with bilinear interpolation. This is equivalent to stretch input feature maps and uses rounded-up dilation scale before the convolution operation. (b) Dispersed convolution can enlarge pre-trained kernel with specific scale. We use structure-level calibration adapt to new perception field when the input feature dimension larger and use pixel-level calibration to preserve local information processing ability.
the training resolution, e.g., 5122 for SD 2.1 and 1024 for SD XI., necessitating mechanism for higher resolution generation (e.g., 2K, 4K, etc.).
2.2 HICH-RESSO.UTION SYNTHESIS AND ADAPTATION
High-resolution image synthesis is challenging due to the difficulty of learning from higher-
dimensional data and the substantial requirement of computational resources. Prior work can mainly
be divided into two categories: training from scratch (Teng et al. 2023 Hoogeboom et al.
2023
Chen 2023 and fine-tuning (Zheng al. 2023 Xie et al. 2023). Most recently, (Jin et 2023
studies a training-fre free method for variable-ste sized adaptation. However, it fails to tackle the higher-
resolution generation. Multi-Diffusion (Bar-
al., (023) and SyneDiffusion (Lee et al., (023)
focus on smoothing the overlap region to avoid inconsistency between windows. However, object repetition still exists their results. MultiDiffusion can avoid repetition by using the region and text conditions given by users. However, those extra inputs not available in the scenario of text- to-image generation. Differently, we propose a tuning-free free method from the view the network structure that fundamentally solve the repetition issue in higher-resolution image synthesis.
METHOD
3.1 PROBLEM FORMULATION AND MOTIVATION
Without the loss of generality, the formulation of this paper considers the e-prediction paradigm of diffusion models. Given a base diffusion model (-)-) parameterized by 0. It is trained on fixed pre- defined low-resolution image x E 3x3xxx our goal is to adapt the model (.).) in a training-free manner to synthesize higher resolution images ă E xx
previous work (Jin et al.) 2023) attributes the degradation performance to the changse in the number of attention tokens and proposes scale the features in the self-attention layer according to the imput resolution. However, when applying it to generate 1024² images, the object repetition is still there (see Fig. T). We observe that the local structure of each repetitive object seems reasonable and the unreasonable part is the object number when the resolution increases. This encourages us to investigate whether the receptive field of any network component does not fit the larger resolution.

---

Hence, modify the components of the SD U-Net to analyze their influence, such as attention, convolution, normalization, etc. We develop re- re-dilation to dissect the effect of the receptive field of attention and convolution, respectively. Re-dilation aims to adjust the network receptive field on a higher-resolution image to maintain the same as the original lower-resolution generation. For re-dilated attention, we partition the feature map into slices with each slice collected via feature dilation having the same token quantity as training. Then, these slices are fed into the QKV atten- tion parallel, after which they are merged into the original arrangement. Details are illustrated in the supplementary. However, maintaining the receptive field of attention yields results with indis- tinguishable differences compared with direct inference. Differently, when increasing the receptive field of convolution in all blocks of the U-Net, fortunately, we observe that the number of objects correct though there are many artifacts such as noisy background and repetitive edges. Based on the observation, we then develop a more elaborate re-dilation strategy considering where, when, and how to apply dilated convolution.
3.2 RE-DILATION
Note that we ignore the feature channel dimension and convolution bias for simplicity in the fol- lowing. Considering a hidden feature h E Rmxx before a convolution layer (i(()) of the network. Given the convolution kernel & E _x*x and the dilation operation & Pa(-) with factor d. The dilated convolution (-) is computed with
(h) h @ Pa(k), (h @ Pa(k))
E h(p) (00--kt -(()). s+d.tpp
(1)
where p. q are spatial locations used to index the feature and kernel, and @ denotes convolution operation. Notably, different from traditional dilated convolutions, which share a common dilation factor during training and inference. Our proposed approach dynamically adjusts the dilation factor only in inference time, leading us to term it as re-dilation. Since the dilation factor can only be an integer, traditional dilated convolution cannot address a fractional multiple of the perception field (i.e.. 1.5x), We propose fractional re-dilated convolution. Without loss of information, we round up the target scale to an integer dilation factor and stretch the input feature map to a size where the perception field meets the requirement. Specifically, let denote the stretch scale and let interp, (-) denote a resizing interpolation function (i.e., bilinear interpolation) with scale s. We upsample the feature h with the stretch scale to (n)erp.(h). The new re-dilation supporting fractional dilation factors is computed as follows:
f (h) = interpi/s (interp,(h) @ P (k)), s = [d]/d,
where [.] is the round-up operator. A visualization of re-dilated convolution is shown in Fig. 3 Considering the properties of the diffusion model with T timesteps and L layers, we further gen- eralize the re-dilation factor to become layer-and-timeste-avaaa
yielding d = D(t, 1). where t 10, T 1).7 [0, T. 1], and 1) is pre-defined dilation schedule function. Empirically, we find that the re-dilation achieves better synthesis quality when the dilation radius is progressively decreased from deep layers to shallow layers, well from noisier steps to less noisy steps, than the fixed dilation factor across all timesteps and layers.
3.3 CONVOLUTION DISPERSION
While the aforementioned simple re-dilation effectively handles the generation of higher-resolution generation at 4x resolution, it suffers from the periodic downsampling problem (e.g., grinding ar- tifacts) (Wang & Ji, 2018). i.e., the features will not consider information from different dilated convolution splits, which leads to the generated image appears repetitive edges under much higher resolution. This problem arises when adapting a diffusion model to generate a much higher reso- lution, e.g., 8x and 16x. To alleviate the problem, we propose to increase the receptive field of a pre-trained convolution layer by dispersing its convolution kernel. Our convolution dispersion method is shown in Fig. 3 Given a convolution layer with kernel & E grxr and a target kernel size r' (if the required perception field multiple is d and r is odd, then r' = d(r - 1) + 1), our method applies linear transform R e "²' x²x= to get dispersed kernel k' Rk. We apply structure- level and pixel-level calibration to enlarge the convolution kernel while keeping the capability of the original convolution layer.
5

---

Our practiinn of 250
a0p prodicted by 24(00
Ineraasse most
da predicted by 2
Denoising
Figure 4: left: Samples by increasing perception field in middle blocks and most blocks (middle and outskirt blocks). The middle blocks-only setting fails to produce the correct small object struc- tures. right: The first row shows the predicted original sample using noise-damped classifier guidance. The second and third rows show the prediction using ég (x y) and ēo (2:)) Co( y) and (x;;)) fails to remove noise during sampling. However, their predictions exhibit a very similar noise pattern. The fourth row illustrates (ē y) (=:))) The erroneous noise prediction vanishes and we can utilize the remaining useful information.
We use structure-eevel calibration to preserve the performance pre-trained convolution layer when the size of the imput feature map changes. Consider arbitrary convolution layer fx(() and the input feature map h. Structure-level calibration requires the following equation:
ifterp.((f((h)) fw (interp,,h)),, k' Rk
(3)
where (-) receives the interpolated feature map and keeps its output the as the interpolated original output (f (h))... Eqn. 3 is underdetermined since the enlarged kernel k' has more elements than &. To solve this equation, we introduce pixel-level calibration ensure the enlarged new convolution kernel behaves similarly on the original feature map h. Mathematically, pixel-level calibration requires (h) fw (h). Then, we combine this with Equ. formulate linear least square problem:
R = arg min (f*) far (interp + I1 (1) fkr (h)
R
where 7/ is a weight controlling the focus of dispersed convolution. We derive an enlarged kernel with convolution dispersion by solving the least square problem. Note that R is not relevant to the exact numerical value of the input feature or the kernel. One can apply it to any convolution kernels be used along with the re-dilation
to enlarge it from x to r' r'. Convolution dispersion technique to achieve a much larger perceptual field without suffering from the periodic sub-sampling problem. To achieve fractional perception field scale factor, we replace the dilation operation with convolution dispersion in fractional dilated convolution introduced above.
3.4 NOISE-DAMPED FLASSIFIER-FRES GUIDANCE
To sample at a much higher resolution (i.e., 4x in both height and width), we need to increase the perception field in the outer blocks in the denoising U-Net to generate the correct structure in small objects as shown in Fig. However, we find that the outside block in the U-Net contributes a lot to estimating the noise contained in the input. When try to increase the convolution perceptual field in these blocks, the denoising capability of the model is damaged. As result, it is challenging to generate the correct small structures while maintaining the denoising ability of the original model.

---

Res
Method
1.5
SD
SD XL. 1.0
KID,
FID,
KID,
FID,
FID,
FID,
FID,
KID,
Direct-Inf
38.50
0.014
29.30
0.008
29.89
0.010
24.21
0.007
67.71
0.029
45.55
0.014
Attn SF
38.59
0.013
29.30
0.008
28.95
0.010
22.75
0.007
68.93
0.028
46.07
0.013
Ours
32,67
0.012
24.93
0.007
20.88
0.008
16.67
0.005
64.75
0.024
28.15
0.009
6.25 1:1
Direct-Inf
55.47
0.020
48.54
0.015
52.58
0.018
48.13
0.014
93.91
0.041
54,90
0.020
0.020
93.92
54.89
Ours
52.11
0.019
45.86
0.014
33.36
0.010
30.66
0.008
80.72
0.032
47.15
0.015
Direct
74.52
0.032
68.98 98
0.027
89.89
0.029
55.48
0.020
122.41
0.062
82.51
0.037
8x 1:2
Attn-SF
74.42
0.032
68.81
0.027
68.97
0.029
53.97
0.020
122.21
0.062
82.35
0.037
Ours
58.21
0.022
52.76
0.017
58.57
0.021
49,41
0.015
119.58
0.057
50.70
0.019
Direct-Inf
111.34
0.046
106.70
0.042
104.70
0.043
104.10
0.040
153,33
144,99
16x 1:1
Atto-SF
110.10
0,046
105.42
0.042
104.34
0.043
103,61
0.041
153.68
0.070
144.84
0.061
Ours
78.22
0.027
65.86
0.023
59.40
0.021
57.26
0.018
131,03
0.063
124.01
0.055
Table 1: Quantitative comparisons among different methods for higher- generation.
4x
16x
gelnely
dog wnaring a Suportere outfic capa
professional phonograhh deig
"phato
Wicrr with aão and Eifol Tower"
Bultin Stoigg Cacament Nirio Cartera Cart Mind Fas
XL
chery Micooce
VICRLAME
surnsanniing waterfall o1 painting
Figure 5: Qualitative comparisons between our method (@), direct inference (2), and altering the attention scaling factor (3) under 4x, 8x, and 16x settings and three different pretrained models.
We propose noise-damped classifier-fre guidance to address the difficulties. Our method incor- porates the two model priors, a model with strong denoising capabilities ee(-), and a model with re-dilated or dispersed convolution most blocks that generates correct content structures Eo(-). Then, the sampling is performed via linear combination of the estimations with a scale w:
(σ(x;) + w- (Eo(2), y) - εo(x:)),
(5)
where y is the input text prompt. Eqn. 5 includes a base prediction (xxx)) that ensures effective de- noising during the sampling process. The guidance term čω ( y/ (x(x)) includes two poor noise predictions. However, in Fig. 4 our experiments demonstrate that the erroneous noise predictions éo (x)) and éo (x,, y) are very similar. Such erroneous noise prediction vanishes Co(#:; y) (xr)), and the remaining information is useful for generating correct object structures.
results of
4 EXPERIMENTS
Experiment setup. We conducted evaluation experiments on text-to-image models, Stable Diffu- sion (SD), including three prevalent versions: SD 1.5 (Rombach al. 2022). SD 2.1 (Diffusion)

---

diperned
Figure 6: Visual comparisons with SD-SR.
Figure 7: Qualitative ablation results.
Method
FID,-4x
KID,-4x
TD-16x
Method
FVD,-4x
KVD,-4x
SD+SR 12.59
0.005
38%
Direct Inference
674.14 14
78.31
Ours 20.88
0.008
62%
Ours
418.80
31.78
Table Quantitative comparisons with SR.
Table 3:
Quantitative results on T2V.
2022], and SD XL 1.0 (Podell et al. 2023) in inferring four unseen higher resolutions. We exper- imented with four resolution settings, which are 4 times, 6.25 times, 8 times, and 16 times more pixels than the training. Specifically, for both SD 1.5 and SD 2.1, the training size is 512², and the inference resolutions
1024?. 1280 2048 x 1024, 2048 respectively. For SD XL. the training resolution 1024², and the inference resolutions are 2048², 2560², 4096 2048, 4096². We also evaluate our approach on text-to-video model for times higher resolution generation. Please our supplementary for the detailed hyperparameter settings.
Testing dataset and evaluation. We evaluate performance on the dataset of Laion-5B (Schuh- mann et al., 2022) which contains 5 billion magge-caption pairs. When the inference resolution 10242, we sample 30k images with randomly sampled text prompts from the dataset. Due to massive computation, we sample 10k images when the inference resolution is higher than 1024². main evaluations, we experiment with normal higher resolution setting (4x, 1:1), a fractional scaling resolution (6.25, 1:1), varied aspect ratio (8x, 1:2), and an extreme higher resolution (16x, 1:1). In other experiments, we evaluate the metrics with the aspect ratio of 1:1 unless otherwise specified. Following the standard evaluation protocol, we measure the Frechet Inception Distance (FID) and Kernel Inception Distance (KID) between generated images and real images to evaluate the gener- ated image quality and diversity, referred to as FID, and KID,.. We adopt the implementation of clean-fid (Parmar et al. 2022) to avoid discrepancies in the image pre-processing steps. Since the pre-trained models have the capability of compositing different concepts that do not appear in the training set, also measure the metrics between the generated samples under the base training res- olution and inference resolution, referred to as FID, and KID,, This evaluates how well our method can preserve the model's original ability when sampling under new resolution.
4.1 EVALUATION
Comparision with training-free methods. We compare our method with the vanilla text-to- image diffusion model (Direct-Inf) and tuning-free free method (Jin al. 2023) via altering the at- tention scaling factor (Attn-FF). As Multi-Diffusioo (Bar- Tal et al. 2023) and SyncDiffusion (Lee) al. 2023) cannot alleviate the repetition issue, they not compared here. Our results shown in Tab. Compared to baselines, we achieve better scores in all settings, indicating our method preserves the generation ability of a pre-trained diffusion model much better in a new scale. Visual comparisons are shown in Fig. 5 Direct inference and Attn-SF tend to generate repeated contents, resulting in an unnatural image structure. On the contrary, our method can generate plausible struc- tures and highly detailed textures in unseen image resolutions.
Comparison with the diffusion super-resolution model (SR). Although our approach does not require any extra datasets or extensive training efforts, to comprehensively evaluate our performance, we compare our approach with a pre-trained Stable Diffusion super-resolution (SD-SR) model: SD 2. 1-upscaler-4x (Ups) at 4x and 16x higher resolution. Both our approach and the SR are combined with the SD 2.1-512 Qualitative and quantitative results are shown in Fig, and Tab. 2 As in Fig. 6 our method synthesizes better details and textures, e.g., the shoes and cushion. Note that the calculation of FID and KID requires image downsampling 2292 which cannot measure the
8

---

Text "A solems tortaise alowdy anbils along, ite horn on ihs badk.
Original Lou- Plaee
Text "A charming racoson
steulitiy rannages
Low-llas
minugh a hrreewaee'' toch con
Direct Direct Inference
Curs
Figure 8: Quantitative results 4x resolution of T2V.
definition of texture. Hence, we perform user preference study on the 16 x setting to ask users to choose an image with a better texture definition between ours and SR. The responses to 300 questions collected. We summarize the results with the percentage of the user's choices, referred to as Texture Definition (TD) (4th column in the Tab. Though our FID and KID are slightly worse than the pretrained SD-SR, method synthesizes high-resolution images with lower-resolution generative model without any extra training, while the SR requires substantial data and computation train and exhibits worse texture details. This demonstrates the pretrained SD already learns the rich texture priors. With proper utilization, we can leverage this prior to synthesizing high-quality and higltrone images directly.
4.2
ABLATION STUDY
We conduct ablation studies on SD 1.5 and 16 X (1:1) resolution to generate 20482. Visual results of removing three key technical compo- nents, including timestep-win wisppp-weee progressive dilation (progression), convolution dispersion, and noise-damped classifier-free guidance (nd- cfg), can be seen in Fig. 7 highlights that the progression improves visual quality, especially in image details. Nd-cfg ensures less noise in the final image, and dispersed convolution im- proves the fidelity of object structures. Tab. 4 shows quantitative results, which show that without these components, the FID dropped by 16.68, 27.96, and 33.93, respectively. Hence, every technical component brings considerable improvements.
Method
FID,--66x
KID,--6x
Ours
78.22
0.027
w/o progression
94.90
0.040
w/o conv dispersion
106.18
0.051
w/o nd-cfg
112.15
0.055
Table 4: Ablation study on SD 1.5.
4.3 APPLY ON VIDEO DIFFUSION MODELS
To verify the generalization ability of our method for video generation models, we apply it to a pre-trained text-to-video model, LVDM (He et al. 2022a) As shown in Fig. our method can generate higher-resolution videos without image definition degeneration. 'he quantitative results in Tab, 3 Metries computed using the video counterpart Frechet Video Distance (FVD) (Un-l Terthiner et al. 2018) and Kernel Video Distance (K VD) (Unterthiner el al. 2019) with 2048 sampled videos and are evaluated on the Webvid-10M. (Bain al. (021).
CONCLUSIONS
We investigate the possibility of sampling images at much higher resolution than the training resolution of pre-trained diffusion models. Directly sampling a higher-resolution image can preserve the image definition but will encounter severe object repetition issues. We delve into the architecture of the SD U-Net and explore the receptive field of its components. Fortunately, we observe that convolution critical for sampling higher-resolution images. We then propose an elaborate dynamic re-dilation strategy to remove the repetition and also propose the dispersed convolution and noise- damped classifier-free guidance for titsalhigh-reoolutionn generation. Evaluations are conducted to demonstrate the effectiveness of our methods for different text-to-image and text-to- video models.

---

REFERENCES
If. URL http:pps//gttttb...o/////eeeooo/
Accessed: 9 28, 2023.
Midjourney. URL.
/ttpt////////////oooooo/oo/////r Accessed:
28, 2023.
Upscaler.
URL
httpa///uuggincaac..c///tttaat
Stable-diffusee x4 upscaler
Accessed: 9 29, 2023.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1728-177 1738, 2021.
Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. 2023.
Lucy Chai, Michael Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang. Any-resolution train- ing for high-resolution image synthesis. In European Conference on Computer Vision, pp. 170- 188. Springer, 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- eration via masked generative transformers. arXiv preprint arXiv:2301 007044 2023.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high- quality video generation. arXiv preprint arXiv:2310.1 19512, 2023.
Ting Chen. On the importance of noise scheduling for diffusion models. arXiv:2301.1097. 2023.
arXiv preprint
Stable Diffusion. Stable diffusion 2-1 base. st stable-diffusae 2 base baase/blin www.b 2022.
ckpt
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. Advances Neural Information Processing Systems, 35:16890-16902. 2022.
Yuan Gong, Youxin Pang. Xiaodong Cun, Menghan Xia, Haoxin Chen, Longyue Wang, Yong Zhang, Xintao Wang. Ying Shan, and Yujiu Yang. Talecrafter: Interactive story visualization with multiple characters. arXiv preprint arXiv:2305. 182427777.8877 2023.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10696-10706, 2022.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion mod- els for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221 2022a.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion mod- els for high-fidelity video generation with arbitrary lengths. arXiv preprini 2022b.
Yingqing He, Menghan Xia, Haoxin Chen, Xiaodong Cun, Yuan Gong, Jinbo Xing, Yong Zhang, Xintao Wang, Chao Weng, Ying Shan, et al. Animate-a-storn a-story: Storytelling with retrieval- augmented video generation. arXiv preprint arXiv:2307.069400 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 -6551, 2020.
10

---

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02:003 2022a.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23:
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. arXiv preprint arXiv:2301.11093 2023.
Zhiyu Jin, Xuli Shen, Bin Li, and Xiangyang Xue. Training-free diffusion model adaptation for variable-sized text-to-image synthesis. arXiv preprint arXiv:2306.08645, 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Yuseung Lee, Kunho Kim, Hyunjin Kim, and Minhyuk Sung. Syncdiffusion: Coherent montage via synchronized joint diffusions. arXiv preprint arXiv:2306.05/72 2023.
Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Ying Shan, Xiu Li, and Qifeng Chen. Fol- low your pose: Pose-guided text-to-video generation using pose-free videos. arXiv preprint arXiv:2304.01 1866, 2023.
Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint WxXi::113.03841, 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 11Xiv:2112.10711, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffuuson probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Gaurav Parmar, Richard Zhang, and Lun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11410-1 11420, 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 0rXiv:2204.061255, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.114887 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.
Jiayan Teng, Wendi Zheng, Ming Ding, Wenyi Hong. Jianqiao Wangni, Zhuoyi Yang, and Jie Tang. Relay diffusion: Unifying diffusion process across resolutions for image synthesis. arXiv preprint arXiv:2309.033550 03350, 2023.
11

---

Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01177, 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marein Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. ICLR, 2019.
Zhengyang Wang and Shuiwang Ji. Smoothed dilated convolutions for improved dense prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2486-2495, 2018.
Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning, arXiv preprint arXiv:2304.0664h 2023.
Jinbo Xing, Menghan Xia, Yuxin Liu, Yuechen Zhang, Yong Zhang, Yingqing He, Hanyuan Liu, Haoxin Chen, Xiaodong Cun, Xintao Wang, al. Make-your- Customized video genera- tion using textual and structural guidance. arXiv preprint arXiv:2306.0094 2023.
Qingping Zheng, Yuanfan Guo, Jiankang Deng, Jianhua Han, Ying Li. Songeen Xu, and Hang Xu. Any-size-diffusion Toward efficient text-driven synthesis for any-size hd images. arXiv preprint ar0iv:2308. /6582, 2023.
12

---

APPENDIX
A EXPERIMENT DETAILS
A.1 MODEL LAYERS IN OUR METHOD
The U-Net Stable Diffusion (SD) v1.5, SD 2.1, and SD XL 1.0 share a similar convolution layer layout. We explain which layer to use re-dilated or dispersed convolutions without loss list of convolution layers contained in U-Net block is shown in Tab. 5 The attention projection layers and convolution shorteut layers will not use re-dilation or dispersion since the convolution kernel in these layers 1x1. Note that the first and the last convolution in the U-Net (conv in and conv out) will not they do not contribute to generating image contents. Also, the spatial part of the text-to-video model used shares the same architecture the SD. Therefore, layers of the following mentioned also the same as our video experiment.
of generality. We follow the naming layers in diffuer
method since
Layer name
Exist in all blocks
Use method
attentions.O.pp i.proooss.........i. attentions.O..p 0.proj out
XX ANAIKA
attentions. 1. -proj_in attentions. 1 _proj_ouu
attentions.2.prg in
..temtiooss2.proj...
resnets.O.oo
resnets.0.conv2
resmets...comv..hhrttue
r.snets.1.convl
1.snets.1.conv2
downsamplers...comv
Table 5: The layers to use our method in a U-Net block. The second column shows the existence condition since some layers cannot be seen in specific U-Net blocks.
A.2
HYPERPARAMETERS
1x
1x
7
1
X
X
16
64
64
64
DB0
DB1
DB2
MB
UB0
UB1
UB2
UB3
Skip connections
Figure 9: Reference block names in the following experiment details. The fractional multiples above blocks are the spatial pixel number of feature maps within the block compared to the network input. ie, the input latent has 64 spatial dimension, then the size of feature maps in DB3 to UB0 is 8².
13

---

We explain selection for hyperparameters in this section. All samples are generated using the default classifier- guidance scale the corresponding pre-trained model (i.e. SD 1.5 and SD 2.1 use 7.5, SD XL 1.0 uses 5.0). Our SD 2.1 experiments use similar setting to SD 1.5. We list the hyperparameters for SD 1.5 only for brevity. The evaluation settings for SD 1.5 are shown in Tab.
T3 reference for block names and their exact location in the U-Net can be found in Fig. The tables show detailed settings about which block use re-dilation conv and dispersed conv. Dilation scale rb. means the dilation scale for re-dilated blocks and dilation scale db. defines the dilation scale for dispersed blocks. If the sampling uses noise-damped classifier-free guidance, we construct cg(.) with strong denoising capability by turning some outskirt blocks that use re-dilated and dispersed convolution to the original blocks. The chosen ones that become the original blocks are listed in noise-damped
g The settings for SD XI. 1.0 are shown in Fig. TO T1 12
blocks.
Params
Values
Params
Values
latent resolution re-dilated blocks dilation scale rb.
4x 128x 128 [D33, MB, UBO]
latent resolution re-dilated blocks dilation scale rb. dispersed blocks
4x 160x 160 [DB3, MB, UBO]
[2, 2, 2]
[2.5, 2.5, 2.5]
dispersed blocks
0
X
X
progressive
progressive
noise-damped cfg. inference timesteps
X
noise-damped cfg. inference timesteps
X
50
50
30
30
Table 6: 1 SD 1.5 experiment settings.
Table 7: 1280 SD 1.5 experiment settings.
Params
Values
latent resolution re-dilated blocks
4x 128 256 [DB0, DB1, DB2, DB3, MB, UBO, UB1, UB2, UB3] 2, 2, 2,
dilation scale rb. dispersed blocks
@ X
progressive noise-damped cfg. noise-damped blocks
[DB0, DB1, DB2, UB1, UB2, UB3]
inference timesteps
50
T
30
Table 8: 2048 1024 SD 1.5 experiment settings,
Params
Values
latent resolution re-dilated blocks dilation scale rb.
4x256x 256 [DB0, DB1, UB2, UB3]
[2, 4, 4,
dispersed blocks dilation scale db. dispersed kernel size
DB2, DB3, MB, UB0, UB1]
[2, 2, 2, 2,
3 x - 5
progressive noise-damped cfg. noise-damped blocks
[DBO, DBI, UB2, UB3]
inference timesteps
50
35
Table 9: 2048 SD 1.5 experiment settings.
A.3 SYNCHRONIZE STATISTICS BETWEEN TILES IN GROUPNORM
14

---

Params
Values
Params
Values
latent resolution re-dilated blocks
4x256x25 x266
latent resolution re-dilated blocks dilation scale rb. dispersed blocks
4x 320x320 [DB1, DB2, DB3, MB, UBO, UB1, UB2] 22,2, 2.5, 2.2
DB3, MB, UB0]
dilation scale rh.
[2, 2.
dispersed blocks
progressive
X
progressive noise-damped efg.
noise-damped cfg.
X
noise-damped blocks inference timesteps
[DBI, DB2, UBL, UB2]
inference timesteps
50
50
30
30
Table 10: 2048 SD XL 1.0 settings.
Table 11: 2560 SD XL 1.0 experiment settings.
Params
Values
Params
Values
latent resolution
xxx12x512
latent resolution re-dilated blocks dilation scale rb. dispersed blocks
2x25xx512
re-dilated blocks dilation scale rb. dispersed blocks
[D22, UBI] [2.
[DBI, DB2, DB3, MB, UB0, UBL, UB2
2, 2. 2, 2, 0 X
[DB3, MB, UB0
dilation scale db. dispersed kernel size
[2.2]
progressive noise-damped cfg. noise-damped blocks inference timesteps
3 5
progressive
[DB1, DB2, UB1, UB2]
noise-damped cfg. maiseddampdd hlocks
[DB2, UB1]
50 30
inference timesteps
50
35
Table 12: 4096xx044 2048 SD XI. 1.0 experiment settings.
Table 13: 40962 SD XL. settings.
When the generated image size is large (i.e., >> 2048 2048), the VAE of SD requires enormous VRAM for decoding and usually not applicable on personal GPU. A simple solution is decoding in tiles. However, tiled decoding usually causes abrupt changes between different tiles shown in Fig. 10 To solve this, one can make overlapped regions between tiles and inter- polate the overlapped regions. However, another problem tiled decoding is the inconsistent color tone between tiles. We figure out this is caused by the in- dependent computation of GroupNorm (GN) layers in VAE between tiles. We propose to synchronize the fea- ture statistics in GN in different tiles. Specifically, we compute the mean and std using all tiles instead of ing only current ones. As shown in Fig. [0 it elimi- nates the color tone difference efficiently.
Tiled Decode
Oure
Direct tiled decode causes abrupt changes in tile borders and differ- ent color tones in tiles. We synchronize the statistics VAE GroupNorm between tiles to address this problem.
Figure 10:
B RE-DILATED ATTENTION
Here, we introduce the experimented re-dilated attention. We aim to keep the original receptive field of attention, e.g., the attention token quantity. Thus, before calculating the attentional features, we first split the input feature map into four slices (the resolution is 4x higher than the training), and for each slice, we flat them into token sequences and feed them into the QKV attention. After the attention calculation, we merge them back to form the original fea- ture arrangement. This operation strictly controls the token length of attention to be the same as training. However, this cannot solve the structure issue of the generated image, as shown in the 2nd row of Fig. [] However, when applying the redilation on the convolu- tional kernel, the structure totally correct. This demonstrates that the key cause of structure repetition lies in convolutional kernels.
Attention He-Ditatin
Kernel Re- Dilvtion
CiOd Attention
MLteen Aancee Divectore APros wardh"
Figure 11: Illustration and re- sults of two re-dilations.
15

---

C MORE COMPARISONS
Method
FID,-
KID,
pFID,
pKID,
sFID,,
sKID,
Time (s)
#param
SD XL
18.50
0.005
29.63
0.014
16.68
0.007
3.5B
SD 2. 1+S
15.39
0.005
17.30
0.005
14.57
0.007
9.5
(1.5+8)
2.2B
Ours (SD 2.1)
18.73
0.005
20.97
0.005
10.17
0.004
5.6
1.3B
Ours (SD 2.1)+ LR
9.96
0.003
19.27
0.007
11.05
0.004
6.3(1.5+4.8)
1.3B
Table 14: Comparison results with state-of-the-art image generation models and super-resolution (SR) model under the resolution of 10242. Time indicates the second used for synthesizing one image
A100 GPU with 16-bit precision). #param stands for the number of model parame-
ters. pFID, and pKID, represent patch-FII and patch-KID, respectively, pFID, CCai et al. (2022).
pKID, Chai et al. 20222, sFID, Nash et al. (2021), and sKID, Nash al. 2221 are used to mea- sure the texture details of generated samples.
We make a comprehensive comparison regarding general generation quality (FID,. KID,), texture details (pFID,, pKID, sFID,, sKID,), inference time and number of model parameters with SD+SR and high-resolution image generation method SI) XI., as shown in Tab. [4] Specifically, pFID/pKII avoids the downsampling operation and instead uses cropping in the metric calculation. sFID/sKID uses the features before the global average pooling to retain low-level details in the feature for the metric calculation, well avoids downsampling. The evaluation dataset is 30k subset from La with a resolution larger than 1024?.
Results show that our training- free method (with low-resolution reference image) achieves al- most the comparable generation performance compared with well-trained SD+SR. Additionally, we achieve better texture details than SD SR (see the sFID and sKID metrics, as well the user study in the main paper). With the low-resolution generated samples as guidance, our method achieves much better results than SD+SR. At the same time, our method has 59% inference time and 59% model parameters less than SD+SR, showing our better efficiency. Compared with SD XL, we achieve both better metrics and lower inference time and parameter numbers.
D MORE ABLATIONS
We conduct additional ablation experiments to investigate the impact of increasing inference reso- lutions. As depicted in Figure [2] it is evident that as the inference resolution increases, the degree of structure distortion becomes more pronounced. Despite these challenges, our method is capable of effectively addressing and mitigating these issues, even in extremely demanding settings.
1280
1536
Direct Inference
Oriinall Resolution
Increasing Resolution
Ours
Figure 12: Performance change when increasing the image resolution. When increasing the resolu- tion. the problem becomes more challenging. Our method is still capable of addressing these issues and maintaining a correct image structure.
16

---

E LIMITATIONS
Ensuring the accuracy of the local structure remains challenging, as demonstrated in Figure 13 particularly with regard to intricate details like the fingers of the robot and the legs of the chairs. It worth noting that this issue is not exclusive to our method but also exists in the original lower- resolution model. As approach is training-free free, inherits the limitations of the original model.
Ours
SD 2.1 original
Ours
1
2
Figure 13: Failure cases on local structures. Our method has failures in the local structure of the generated image. We observed that the original lower-resolution model also struggled with this problem.
F CLOSED-FROM SOLUTION FOR DISPERSED CONVOLUTION
We follow the notation in our main paper. Given convolution layer with kernel E Rexr and a target kernel size r'. We find a dispersion transform R E "r*x"² to get a dispersed kernel k'. Considering input feature map h Roxx, ignore the bias in convolution, since will not influence our results. Then structure-level calibration and pixel-level calibration can form an equation
interpa = Jw (interpa(h)) nfa(h) - (h),
where (Ju()) Bndxad (h) E Rax. The equation has (nd)² n² equations. Each equation is made up of the sum of terms kighiy (elements in & and h) where the coefficient of the terms are linear combinations of R. (elements R) and constants. For example, when 3, = 3 and r' = 5, the 5-th equation in pixel-level calibration is:
kn1hn k12h12 k13h13 (R7 + R7,2k12 + R7.3k13 3 + ...... + Rr (Rs,1k11 R8.2k12 + R8.3h13 + Rg.9k33 h112
k3shes
...t
(R19,1k11 Ra9,2h1n R19,3k13 +... R19
We rearrange the equation and get: R7,1 )h hn (- R122) + (1 R19.9) 0.
(8)
To ensure this equation for all kiyhij. one can let every coefficient be zero. Getting a linear equation set for the 5-th equation in pixel-level calibration.
R7,1 =
R7,2 0
R19,9 -1
17

---

We do this for every equation in Eq. (6) to derive larger set of linear equations of Ryj. Note that the equation sets are linear equations of Ri and have no h or k, making it applicable to all conv kernels and any input feature. Let's go back to the general case where R E xr² Let Astructure denote the coefficient of the linear combination for Rij in structure-leve calibration, and let Dstructure denote the fight-hand-side constants in the equation set of structure-Ievel calibration. Similarly, we define Apixel, Spizel, and we construct least square problem.
R1,1
Astructure
Ustructree nppiell
R1,2
Ax b A AApiel
b=
x
(5)
R
The solution x (A (A-1") This can be easily solved by math software, i.e. MATLAB.
G SEARCHING RE-DIILATION SCHEDULE
We first test series of hand-crafted hyperparameters and figure out some empirical results: 1) Using re-dilation/dispersion in inner U-Net blocks and using original convolution in outer U-Net blocks produces good results. 2) Sharing the same re-dilation scale in block instead of using different re-dilation scales for every convolution layer within a block produces better results.
Then, search for hyperparameters using automatic strategy. For each target resolution, sample 50 examples from LAION-5B to build an evaluation set. The MSE loss noise estimation, used in the training of diffusion models, serves metric for the hyperparameter search each diffusion timestep. The hyperparameter that achieves the lowest loss is chosen for the corresponding timestep.
To speed up the search process, we prune the hyperparameter set using the previously established empirical rules. The pruning method is as follows: Use the same re-dilation scale in all con- volution layers within a block. 2) The blocks within start block and an end block will use re- dilatiinndlispersion. For example, SD v2.1 blocks are shown in Fig. 9 If the start block is DB2 and the end block is UB1, then the blocks that use re-dilation/ddiseesionn are DB2, DB3, MB, UBO, and UBI. 3) If the search includes noise-damped classifier-fre free guidance, then the blocks that use re- dilation/dispersion in is continuous subset of the blocks that use re-dilation/dispersion in ég. For example, if Eg re-dilation/ddspersion blocks are DB2, DB3, MB, UB0, UB1, that in EΦ can be DB3, MB. UB0. 4) The maximum re- re-dilation scale does not exceed the enlargement scale of the target resolution (i.c., generating 2048x2048 image using a 512x512 model, the maximum re-dilation scale is 4). The search step of the re-dilation scale is 1.
18

---

H OTHER VISUALIZATIONS
2:1 (2048H1022))
A corgi oits on beachh chair on a butttiulll beach, with padm trees bebind. high Aetaing
16:9 (1024x576)
sbateboowding snouy
Fanney
11.5:1 (284)
"Fieer in dream land'
1:1 (1024x1024)
1800112
'Ai riding bicycle
skateboarding
Figure 14: More generated results with our method and SD 2.1 with arbitrary aspect ratios and sizes.
19

---

Kng
oloar
(1024x10
"A natic woeden catln netted a snowe foreet
(2048x2048)
(20-18x1024) 3:4 (960x1280)
3::4 1280)
2:1 (2018x1024)
Figure 15: More generated results with our method and SD 1.5 with arbitrary aspect ratios and sizes.
20

---

(2048x2045)
(086x4038)
abense or Johin Lbaing
1:11 (0448820488
corncle corvaatadd oftounchap
MEW MENT Fattamnse
2:1 (4096x2048) 1:1 (248x20418)
racisaius bow wooden tabile tcheer stock sholl"
"Spring Magnotlo bouquet. Featuring mag volla leawes, pussywillwn
2:1 (2048x2045)) 2:1 (4096w20-48)
Prer is pox Mockuie saan trow art oa name petniing modem home decre piclure print
buing
Widh richo
Figure 16: More generated results with our method and SD XL 1.0 with arbitrary aspect ratios and sizes.
21


Reviews:
[{"review_id": "3lYv9VuCkj", "reviewer": "Reviewer_7LTr", "summary": "This submission proposed the generation of high-resolution images from pre-trained diffusion models. The authors claimed that their approach can address the persistent issues with the generated images. A re-dilation method was proposed to dynamically adjust the convolutional perception field during inference. Moreover, the authors implemented ultra-high-resolution image generation with a dispersed convolution and noise-damped classifier-free guidance. The experiments were conducted to demonstrate the advantages of the proposed solutions.", "questions": "The significance of the proposed method is still unclear after the presentation of the experimental results. The method can achieve similar or slightly better results. However, it is not sufficient to justify the significance of the work. This is a major concern.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "As claimed by the authors, the proposed solution is simple and efficient, but it is powerful enough to generate ultra-high-resolution images and videos. The method does not require any training and optimization.", "weaknesses": "The major issue is with the experiments. It is unsure whether the results in Table 1 tell about the differences among SD solutions or the differences between the \"Method\" in the second column. And when the resolution changes, how did the results vary accordingly? Are the metrics still acceptable if a higher-resolution image is generated? What do the metrics mean for the application in addition to their use for comparison? \n\nAs the authors mentioned the results in Table 2 were not better than the SR+SR method, what is the justification to use the proposed solution? How could the user balance the efficiency and the quality in terms of the application's needs? It is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ijFAOEtrkx", "reviewer": "Reviewer_6a3j", "summary": "This submission proposes to generate images from pre-trained diffusion models at a much higher resolution without the problems of object repetition and unreasonable object structures. In particular, it presents the key problem of the limited perception field of convolutional kernels and represents the new dispersed convolution and noise-damped classifier-free guidance. The experimental results, including ultra-high-resolution image and video synthesis, are very impressive.", "questions": "Please refer to the above weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper addresses a significant challenge in image generation by investigating the generation of images at much higher resolutions than the training image sizes, while also allowing for arbitrary aspect ratios. This is a novel and important problem in the field of image synthesis.\n- The proposed approach of using re-dilation to dynamically adjust the convolutional perception field during inference is a key contribution. This addresses the identified limitation of convolutional kernels, which is a crucial insight for improving high-resolution image generation.\n- The paper's extensive experiments demonstrate the effectiveness of the proposed approach in addressing the repetition issue and achieving state-of-the-art performance, particularly in texture details. This provides strong empirical support for the proposed techniques.\n- Overall, the paper appears to make a valuable contribution to the field of image synthesis, specifically in the context of generating high-resolution images with arbitrary aspect ratios. The proposed techniques and insights are well-motivated and supported by extensive experiments.", "weaknesses": "- Fig. 5-7 and Table 2-4 could be presented in a former manner.\n- More ablation studies could be presented in the main text and limitations are suggested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YLFaaKVGZ8", "reviewer": "Reviewer_URVR", "summary": "This paper conduct research on the capability of generating higher resolution image from pre-trained diffusion models using lower resolution training data. The authors find that in the existing structures, the perception field of convolutional kernels is limited. Based on this, the authors propose a re-dilation method that can adjust the convolutional perception field during inference. The proposed method in this paper can generate high resolution images with 4096x4096 resolution without extra training. Experimental results show that the proposed method is able to achieve the SOTA results. This paper is well organized and easy to follow.", "questions": "Here are my detailed comments and suggestions:\n\nThis article is generally good, but I still have some questions about the design of the model and the experimental process.\n\n1.\tThe section on \"re-dilation\" mentions the use of a predefined dilation schedule function D(t,l). How is this function defined, and have any experiments been conducted to validate or optimize its selection?\n\n2.\tIn the \"convolution dispersion\" section, a linear transformation R and structural-level calibration are used to enlarge the convolution kernel. Does this step add to the model's computational complexity? If so, by how much?\n\n3.\tIn the \"noise-damped classifier-free guidance\" section, a guidance scale w is mentioned. How is this parameter chosen? Does it require adjustments for different tasks or datasets?\n\n4.\tThe authors used FID and KID as the metrics that require downsampling images to 229 × 229, does this imply that these metrics are not very suitable for evaluating high-resolution images?\n\n5.\tHas the paper considered comparisons with other recently published methods for high-resolution image generation? Would such comparisons further validate the effectiveness of the model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Here are the strength points of this paper:\n\nThe authors proposed an observation and explanation for the objective repetition problem. They also designed a re-dilation method to address this problem for high-resolution image generation. The proposed method is applied and evaluated on multiple models, the results show that the proposed method is effective.", "weaknesses": "Here are the weak points of this paper:\n\nSee the detailed comments and questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wuBzJNUdEe", "reviewer": "Reviewer_3ixD", "summary": "This paper proposes an (almost) training-free solution for adapting a pre-trained diffusion model to generate images of much higher resolution than the image size used during training.\nTo achieve this goal, the authors consider the procedure of convolution dispersion which aims to increase the receptive field (in terms of pixels) of convolutional blocks while preserving the properties of the original layer.\nIn addition, the presented technique of noise-damped classifier-free guidance leverages both the original model and its modification with dispersed convolutions in order to combine the generative power of the former and better denoising in high resolution of the latter.\n\nAccording to the qualitative evaluation, the proposed approach successfully eliminates the issue of object repetition and implausible object structures. This is also confirmed with quantitative measurements.", "questions": "1. Please, address the weaknesses mentioned above.\n1. Since the dispersed convolution is a dense operation with a larger receptive field, it introduces additional computational costs. How significant is the computational overhead of the method?\n1. Is the re-dilation described in Sec. 3.2 actually used in the method? As far as I understood, in practice only dispersed convolutions are applied in the modified UNet. Probably, this part of the description needs more polishing.\n1. Is it possible to find a closed-form solution for the optimization problem in Eq. 4? If so, it would be nice to add it to the paper or supplementary. Otherwise, the method implementation is not entirely training-free (although still pretty cheap), and the claims should be adjusted.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem tackled in this work is quite important for the community. In general, this paper is clearly written. The motivation is explained well, and the details of the method are concise (see minor remarks below). The samples provided by the authors look quite plausible. Also, I appreciate the implementation details specified in the Appendix.", "weaknesses": "1. As far as I can judge, there is no discussion in the paper of the ability to magnify the output produced with the original diffusion model in standard resolution. I suggest adding it to the paper since it seems that it can be relatively straightforward to add it to the existing guidance (Eq. 5). This can help to make the comparison with SD+SR (Tab. 2) more conclusive.\n1.  None of the reported quantitative metrics except for human assessment estimates the plausibility of generated high-frequency details, since common implementations of FID/KID downsample the input images to the ImageNet resolution before the feature extraction. Therefore, I recommend reporting e.g. patch-FID [1]  to assess high-resolution textures in addition. Also, some other variations like sFID [2] are known for better assessment of spatial variability.\n\n1. Minor remarks:\n    1.  Eq. 1 probably has a typo: the variable $o$ exists in LHR but not in RHS.\n    1.  Probably, one needs to replace $\\min$ with  $\\arg\\min$ in Eq. 4.\n    1.  Please highlight the best metrics in tables consistently (see Tab. 2).\n    1.  There is no such block as MB3 in Fig. 1 of Appendix A2, please edit the caption.\n\nReferences:\n\n[1] Chai et al. Any-resolution Training for High-resolution Image Synthesis. In ECCV, 2022.\n\n[2] Nash et al. Generating images with sparse representations. In ICML, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)