Title: Is One GPU Enough? Pushing Image Generation at Higher-Resolutions with Foundation Models.
Venue: NeurIPS 2024
Accepted: yes
Keywords: diffusion model

generative AI

image generation

foundation models

higher resolution
Abstract: In this work, we introduce Pixelsmith, a zero-shot text-to-image generative framework to sample images at higher resolutions with a single GPU. We are the first to show that it is possible to scale the output of a pre-trained diffusion model by a factor of 1000, opening the road to gigapixel image generation at no extra cost. Our cascading method uses the image generated at the lowest resolution as baseline to sample at higher resolutions. For the guidance, we introduce the Slider, a mechanism that fuses the overall structure contained in the first-generated image with enhanced fine details. At each inference step, we denoise patches rather than the entire latent space, minimizing memory demands so that a single GPU can handle the process, regardless of the image's resolution. Our experimental results show that this method not only achieves higher quality and diversity compared to existing techniques but also reduces sampling time and ablation artifacts.

Is One GPU Enough? Pushing Image Generation at Higher-Resolutions with Foundation Models.

Abstract

In this work, we introduce Pixelsmith, a zero-shot text-to-image generative framework to sample images at higher-resolutions with a single GPU. We are the first to show that it is possible to scale the output of a pre-trained diffusion model by a factor of 1000, opening the road for gigapixel image generation at no additional cost. Our cascading method uses the image generated at the lowest resolution as a baseline to sample at higher-resolutions. For the guidance, we introduce the Slider, a tunable mechanism that fuses the overall structure contained in the first-generated image with enhanced fine details. At each inference step, we denoise patches rather than the entire latent space, minimizing memory demands such that a single GPU can handle the process, regardless of the image's resolution. Our experimental results show that Pixelsmith not only achieves higher quality and diversity compared to existing techniques, but also reduces sampling time and artifacts. 1

1 Introduction

Recent advances in diffusion models (DMs) have revolutionized the field of high-fidelity image generation. Foundational works like Ho et al. [2020], Sohl-Dickstein et al. [2015], Song and Ermon [2019] established the groundwork, leading to significant breakthroughs demonstrated by Dhariwal and Nichol [2021]. These models have evolved rapidly, with innovations such as new sampling techniques Lu et al. [2022], Song et al. [2020] and capabilities for inpainting Lugmayr et al. [2022] and image editing Brooks et al. [2023], Mokady et al. [2023], Nichol et al. [2021]. However, even though DMs have shown remarkable results on high-fidelity image generation, scaling them to high-resolutions is still an open challenge. The introduction of the latent diffusion model (LDM) Rombach et al. [2022b] has made high-resolution image synthesis more accessible. However, most pre-trained DMs based on the LDM are limited to generating images with a maximum resolution of

1024 2 pixels due to constraints in computational resources and memory efficiency. Following this approach, aiming to ultra-high-resolution generation would incur additional training and data costs, and the model would be unable to run on a single GPU.

Recently, several works have focused on scaling pre-trained models to higher-resolutions, highlighting new possibilities and challenges in the field Aversa et al. [2024], Du et al. [2023], Gu et al. [2023], He et al. [2023]. This allows already available models to be used with no extra costs and no additional carbon footprint. However, most methods addressing this problem either require expensive GPUs, as memory demands increase with resolution or prolonged generation times. Moreover, scaling the native resolution of a generative foundation model to higher-resolutions introduces artifacts due to the direct mapping into a prompt-image embedding space. For a specific prompt, we would expect to sample an image matching the description and of the same size as the training data images. Consequently, even scaling up by a factor of 2 would lead to the duplication of the image produced by the prompt across the higher-resolution image. However, since the images in the training set contain information at different resolutions, we can leverage this prior knowledge implicitly learned by the diffusion model to guide the generation at higher-resolutions and enhance fine details. Addressing these challenges is crucial for applications that demand ultra-high-resolution images, such as gigapixel photography, medical imaging, satellite imagery and high-definition digital art.

To overcome the challenges in ultra-high-resolution image generation, we introduce Pixelsmith, an adaptable framework that utilizes pre-trained generative models for scalable gigapixel synthesis.

Our contributions are outlined as follows: 1. We introduce Pixelsmith, the first framework capable of generating gigapixel-resolution images using a single GPU. 2. We develop the Slider, a dynamic tool that allows users to adjust the balance between overall image structure and fine-detail enhancements in the generation process. 3. We enhance and adapt the random patch denoising strategy to text-to-image pre-trained diffusion models, leading to minimized memory usage. 4. We provide a masking method that, combined with the Slider, reduces the number of artifacts at higher-resolutions.

2 Related Work

Pre-trained DMs are trending toward increasing native resolutions. Notable examples include Stable Diffusion (SD) Rombach et al. [2022b], which started at 512 2 , then 768 2 in SD 2, and 1024 2 in SDXL Podell et al. [2023], SD Cascade Pernias et al. [2024], and SD 3 Esser et al. [2024]. Similarly, DALL-E Ramesh et al. [2021] continues to increase resolution with each version OpenAI [Nov 6 2023]. This trend shows a growing demand for higher-resolution generation.

Currently, high-resolution image generation often involves a super-resolution model applied after the initial text-to-image generation Saharia et al. [2022]. This additional model increases costs due to training and domain-specific fine-tuning requirements.

2.1 Trained Models

Trained models are those that are specifically designed for multi-resolution generation. Recent models like Matryoshka Gu et al. [2023] can generate various resolutions up to 1024 2 through a progressive training schedule. However, scaling beyond this is limited. CogView3 Zheng et al. [2024b], based on Relay Teng et al. [2023], upsamples from a base resolution of 512 2 , though higher-resolutions like 4096 2 remain a future goal. Fine-tuning methods such as DiffFit Xie et al. [2023] are costly, requiring 51 V100 GPU days. ASD Zheng et al. [2024a] introduces a memory-efficient sampling method capable of theoretically generating images up to 18432 2 resolution. Patch-DM Ding et al. [2023], by training on 64 2 patches, can generate resolutions like 1024×512. LEGO Zheng et al. [2023] and Inf-DiT Yang et al. [2024] also rely on training.

2.2 Adapted Models

Adapted models, on the other hand, modify pre-trained models to generate higher-resolutions without additional training. MultiDiffusion Bar-Tal et al. [2023] offers controllable generation through multiple processes but is slow and prone to structural errors Du et al. [2023], He et al. [2023], Zheng et al. [2024a]. ScaleCrafter He et al. [2023] improves resolution by altering the convolution kernel dilation but faces memory limitations. DemoFusion Du et al. [2023] adapts MultiDiffusion

A

1000

x16 (4096×4096)

LX

x64

x64(8192×8192

x64 (6144×10880)

x1

x64

(65×65)

(33×33)

(33×25)

x16(5440×30

x1024(32382×32382)

(2080×2080)

（1056×1056)

(1056×800)

for constant memory use, though it is time-consuming. ElasticDiffusion Haji-Ali et al. [2023] and SyncDiffusion Lee et al. [2023] aim for specific high-resolution goals, but object repetition remains a problem at large scales Jin et al. [2024]. Recently, HiDiffusion Zhang et al. [2023] introduced modifications to the UNet architecture to prevent object duplication; however, it is limited to a maximum resolution of 4096 2 due to high GPU memory requirements. AccDiffusion Lin et al. [2024] employs patch-content-aware prompts and adjusts the attention masks within the UNet to suppress artifacts, but these changes result in blurry images. Similarly, Fouriscale Huang et al. [2024] is another model that alters the UNet but exhibits issues at higher-resolutions, particularly at 4096 2 . Current approaches excel in some areas but fall short in others. To enable high-resolution generation on a single consumer GPU, models must be adapted to avoid retraining costs. Efficient memory usage is crucial to prevent the need for expensive, high-memory GPUs, and the process must be fast and artifact-free. We propose a flexible framework that addresses these challenges based on text conditions.

Latents

ZT

ZT-1

20

(i,))

(l,m)

（l,m)

，

Timesteps per pixel

timesteps

3 Foundations

3.1 Diffusion models

DMs Ho et al. [2020], Sohl-Dickstein et al. [2015] are probabilistic generative models that first add noise to a distribution during diffusion and then learn to remove this noise during denoising. This way, during training, a Gaussian probability distribution is learnt and during inference sampling from the Gaussian leads to the data probability distribution. Executing this process in the latent space Rombach et al. [2022b] is more resource efficient allowing for faster training and inference times. In a Latent Diffusion Model, let z 0 ≡ E θ ( x ) represent the clean training data x ∼ q ( x ) mapped to the latent space by the VQ-VAE encoder E θ . Beginning from z 0 , at each timestep t ∈ { 1 , . . . , T } , noisy latents z 1 , . . . , z T ∼ q ( z t | z t -1 ) := N ( z t ; √ 1 -β t z t -1 , β t I ) are sampled according to a noise schedule function based on the timestep β t ∈ [0 , 1] . During training, the model learns to denoise these latent variables using a conditional prompt c . At inference time, starting from a random latent z T ∼ N (0 , I ) , the model generates by iteratively sampling z 0 ∼ p θ ( z 0 | z T , c ) = ∏ T i =1 p θ ( z t -1 | z t , c ) .

3.2 Patch sampling

The default LDM denoising process samples the entire latent space at each timestep, which becomes resource-intensive as the resolution increases. Methods like Li et al. [2024] distribute this load across multiple GPUs, but this requires expensive hardware. To address this, we adapted and refined the DiffInfinite sampling method Aversa et al. [2024] for text-to-image DMs, enabling ultra-highresolution generation efficiently on a single GPU.

At each timestep, random patches are selected for denoising, and this process is repeated until the entire latent space is denoised. The randomness is controlled for efficiency. We track which pixels have been denoised at each timestep, and select areas where pixels have not been denoised yet. This sampling process avoids excessive inference times, especially at very high-resolutions like 32768 2 . In our experiments, we sample images at higher-resolution using fixed 128 2 patches in the latent space. To handle overlapping patches, pixel values are temporarily reverted when already denoised pixels are encountered, ensuring correct processing. After the current patch is denoised, the pixels that were reverted are restored to their denoised values. Only the first denoising of each pixel is retained for each timestep, preventing redundant denoising (see Fig. 2).

Despite its advantages, the DiffInfinite sampling method still relies on segmentation masks to condition each patch, providing rich spatial information. However, in text-to-image DMs, where a global text prompt conditions the entire latent space, this method is insufficient on its own. Applying the same text prompt to each patch results in repetitive content and poor-quality generations. To overcome this, we introduce a guiding mechanism that incorporates structural information between patches alongside the text prompt, improving the quality and diversity of the generated images.

IMAGEGUIDANCEGENERATION

2. Upsampling process

pguid

e.g.x4

Zoomed-inbaseresolution

‘BustsculptureofaraccoonGoddessmadeof marble”

1. Text-to-image generation

T

T-1

0

Decoding

Upscaler

HIGHER-RESOLUTIONIMAGEGENERATION

Encoding

Zoomed-inhigher-resolution

3.Imageguidancepreparation

guid

T

guid

2t-1

Slider

guid

2k-1

guid

20

：

口

4.Image generation

Guided

Puregeneration

Prompt

：

mechanism

Guidance

Denoising

：

ZT

Zt

Zt-1

2k

2k-1

20

Decoding

Guidancemechanism

gid

FFT

Φ

2t-1

Zt

Mean-iFFT→

IFFT

Masking

Prompt

FFT

个

Denoising

4 Method

4.1 Problem statement

Currently, pre-trained text-to-image LDMs generate a latent variable z 0 ∼ p θ ( z 0 | z T , c ) with a fixed size using a diffusion process, which is then decoded into pixel space using the decoder D θ of a V AE. In this section, we show how to leverage foundational generative models to sample arbitrarily large images based on a given prompt. Without requiring additional training or fine-tuning, the LDM, trained on images with size H × W × 3 , is adapted to generate images with size mH × nW × 3 , where m,n ≥ 1 are the scaling factors for achieving higher-resolution.

We present Pixelsmith, a framework with a flexible number of cascaded steps (Section 4.3) designed to generate images at ultra-high-resolution on a single GPU. The implementation of the Slider (Section 4.4) provides control over the generation process using patches (Section 4.2), making it resource-efficient regardless of the resolution.

4.2 Framework overview

In this section, we describe the workflow of Pixelsmith, detailing how the framework adapts pretrained text-to-image LDMs to generate images with higher-resolutions on a single GPU (see Fig. 3). In order to generate ultra-high-resolution images without artifacts, we introduce these key components: the Slider (see Sec. 4.4), patch averaging (see Sec. 4.5) and masking (see 4.6).

Text-to-image generation First, given a conditional prompt c , we use SDXL to sample the latent variable ˜ z 0 and decode it into pixel space as ˜ x = D θ (˜ z 0 ) , generating a 1024 2 resolution image.

Upsampling process After the image generation, we apply an upsampling algorithm to increase the image resolution (see Sec. 4.3 for details). In our case, we used Lanczos interpolation Lanczos [1950] to scale up to the desired resolution. However, upsampling leads to a blurred output and lack of additional content. The upsampled image, x guid , will serve as guidance for our generative process.

Image guidance preparation Once the guidance image is encoded in the VAE's latent space z guid 0 = E θ ( x guid ) , we can easily sample each latent variable of the diffusion process through the forward diffusion process z guid t ∼ q ( z guid t | z guid 0 ) .

Image generation The generative process starts from z T ∼ N (0 , I ) , which has the same dimensions as z guid T . At each step, a random patch is cropped as described in Section 3.2:

where C i,j : R 4 ,h,w → R 4 ,p h ,p w is a cropping function that extracts patches of size ( p h , p w ) from the latent variables z t and z guid t of size ( h, w ) at the coordinates ( i, j ) . For simplicity, we will refer to the cropped latent patch ˆ z ( i,j ) t as ˆ z t and ˆ z guid, ( i,j ) t as ˆ z guid t throughout this discussion.

The Slider's position (see Sec. 4.4 for details), indicated by a blue line in Fig. 3, determines whether the guidance mechanism (see Sec. 4.4.1 for details) or unguided patch denoising will be applied. In the unguided mode, each patch is based solely on the previous one and the text condition, similar to a conventional patch denoising process. The Slider allows control over whether a generated image will be slightly or significantly altered compared to the previous resolution.

After the denoising process has ended, the latents z 0 are decoded and the higher-resolution image is generated. Using a cascade upsampling approach, the generated image can be upsampled again, repeating the process to achieve an even higher-resolution image.

4.3 Higher generation in one step

Most of existing higher-resolution generative methods rely on cascade sampling approaches Denton et al. [2015], Ho et al. [2022], Menick and Kalchbrenner [2018]. Our approach, however, differs from previous works like Du et al. [2023], Guo et al. [2024] in two significant ways.

First, we perform upsampling directly in the pixel space rather than in the latent space. Manipulating the latent space distribution with transformations like upsampling can introduce distortions that the scheduler has not accounted for Chang et al. [2024], leading to degraded image quality Hwang et al. [2024]. Second, our flexible method enables higher-resolution image generation in two generative steps: first by producing the base image, then by enhancing it to the desired resolution. For instance, Du et al. [2023] requires passing through intermediate resolutions ( 1024 2 , 2048 2 , 3072 2 ) to achieve a 4096 2 resolution, with even higher-resolutions necessitating additional steps. This results in prolonged waiting times and multi-scale duplications, as each step may produce duplicates at different resolutions (Appendix D).

In contrast, our framework can generate any resolution directly from the base one, making it feasible to scale up to a 32768 2 image directly from the base resolution of 1024 2 . However, intermediate steps can sometimes yield better small-sized details (Appendix E). Our flexible approach supports both two-step generation and cascade generation, allowing users to select their preferred sampling method depending on the use case.

basemodel

withmasking

withoutmasking

artifacts

4.4 Slider

Currently, higher-resolution generative models often encounter issues such as duplications and atypical anatomical features or structures He et al. [2023]. To mitigate these artifacts, we introduce the Slider (indicated by a blue line in Figure 3). The Slider is a parameter that determines at which denoising step there is the transition between the guidance mechanism introduced in Sec. 4.4.1 and traditional unguided generation. The Slider can either constrain the denoising process to eliminate undesirable outputs or allow for less constrained generation to add details.

When set to zero, the generation is entirely unconstrained. This results in an image that might resemble the 1024 2 base image but will likely contain multiple duplicates. At t = T , the generation is highly guided, leading to an image that is almost identical to the previous resolution. The optimal value for the Slider varies depending on the base image. Adjusting the Slider manually is necessary to achieve the best output. This manual adjustment allows for fine-tuning the balance between constrained and unconstrained generation, ensuring that the resulting image maintains the desired quality and level of detail. The ideal Slider setting is influenced by factors such as image complexity, the presence of fine details, and the desired resolution. Experimentation and iterative adjustments are often required to find the most effective value for each specific image and resolution.

An example demonstrating the impact of the Slider's position can be found in Appendix G.

4.4.1 Guidance mechanism

The guidance mechanism combines the random patches ˆ z guid t , ˆ z t , and ˆ z guid t -1 to generate the updated patch ˆ z t -1 . First, the patches ˆ z guid t and ˆ z t are transformed into the Fourier space using the Fast Fourier Transform ( FFT ), where their phase components ϕ t and ϕ guid t are averaged:

Here, A t and A guid t represent the amplitudes, while ϕ t and ϕ guid t are the phases. 2 The averaged phase ϕ t is then combined with the amplitude A t from ˆ z t to form a new Fourier representation ˆ z FFT t = A t e iϕ t . While the amplitude A t preserves the intensity and contrast information, the phase averaging process ensures that the resulting phase contains structural characteristics from both

the generated and guiding images. This modified Fourier representation is transformed back to the spatial domain using the inverse Fast Fourier Transform ˆ z i FFT t = i FFT ( ˆ z FFT t ) . The output ˆ z i FFT t is then used as the condition for the reverse diffusion process to generate the next patch ˆ z i FFT t -1 ∼ p θ ( ˆ z i FFT t -1 | ˆ z i FFT t , c ) . This solution suppresses local artifacts and helps maintain lowfrequency structural consistency between the reverse diffusion steps, reducing the likelihood of significantly altering the global properties of the generated image. However, the guidance mechanism alone still generates some long-range discrepancies across the image due to prompt sharing between the patches. To mitigate these artifacts and prevent prompt duplications across the latent space, we introduce a masking method(see Section 4.6).

4.5 Patch averaging

Overlapping patches can sometimes cause visible differences at their borders. To address this, we introduce a transition zone where the values of overlapping patches at timestep t are averaged, producing a smooth and seamless denoised output. In Appendix C, we illustrate the artifacts caused by patch overlap and show how averaging effectively removes them.

4.6 Masking

One of the main reasons patch-based image generation in diffusion models Bar-Tal et al. [2023] suffers from artifacts at higher-resolutions is the use of the overall image text prompt for each patch during the denoising process. This leads to duplicate structures forming in the latent space. We partially addressed the spatial coherence with the guidance mechanism in Section 4.4.1. To further improve results, we combine the sampled ˆ z iFFT t with the image guidance ˆ z guid t using a chess-like mask Λ (Equation 3).

In Fig. 4, we show the comparison between a higher-resolution image generated with and without masking with the image guidance. The image on the right, which is not constrained by pixels from ˆ z guid t , exhibits more freedom in generation and, as a result, introduces notable artifacts. For example, clouds transform into waves, and the blue sky morphs into the sea. Additionally, a small, seemingly artificial lake or river appears in the middle of the village, situated unnaturally close to the beach.

5 Experiments

Pixelsmith is tested on a single RTX 3090 GPU, with all tested resolutions requiring 8.4 GB of memory. Performance is evaluated on the LAION-5B dataset Schuhmann et al. [2022] by randomly sampling 1,000 image and text prompt pairs. The metrics used for evaluation are Fréchet Inception Distance (FID) Heusel et al. [2017], Kernel Inception Distance (KID) Bi´ nkowski et al. [2018], Inception Score (IS) Salimans et al. [2016], and CLIP Score Radford et al. [2021], with the FID metric computed using the clean-FID approach Parmar et al. [2022] (for further comparisons, see Appendix B). Additionally, we compare our results with a super-resolution model in Appendix F.

5.1 Ablation study

We conducted a quantitative examination of our framework at a resolution of 2048 × 2048 pixels, focusing on key factors that influence its performance: the Slider position, the role of amplitude and phase in the latent space, the importance of masking during guidance, and the impact of averaging overlapping patches. Our results are shown in Table 1.

Slider position Our findings highlight the critical role of the Slider position in the quality of the generated images. Setting the Slider to position 30 ( proposed method) provides an optimal trade-off between guidance and generation, outperforming other positions like 1 ( SP1 , no guidance), 24 ( SP24 , mid-point), and 49 ( SP49 , full guidance). A Slider position of 1 introduces numerous artifacts due to insufficient guidance, while a position of 49 lacks fine detail because it relies too heavily on lower-resolution. Position 24 has been chosen as mid-point in the diffusion process, it is closer to the optimal value but does not yield the best results across our dataset.

ScaleCrafter

DemoFusion

FouriScale

AccDiffusion

20482

制

40962

气

20482

40962

Amplitude and phase In exploring the role of amplitude and phase within the latent space, we experimented with different configurations. The proposed method averages the phase of the guidance latents and the current latents while using the amplitude of the current latents, as detailed in the Methods section (see Figure 3). We compared it with other setups: one that averages the amplitude of both latent spaces while using the phase of the current latents (denoted as A in Table 1), and another that averages both amplitude and phase from both latent spaces (denoted as A & ϕ ). The proposed method demonstrated superior performance in maintaining image quality and preserving fine details. Averaging the two phases provides guidance through structural information, while averaging the amplitudes only conveys the intensity of the frequencies. Averaging both offers information on both structure and intensity which overly influences the final image, leading to a lack of fine details expected in a higher-resolution output.

Masking Masking emerges as a significant factor in enhancing image quality. The quantitative metrics indicate that the chess-like mask leads to slightly better scores compared to using no mask (as seen in the No Mask column of Table 1 compared to the proposed one). Visual inspections reveals that incorporating the mask is crucial for removing artifacts (see Figure 4).

Overlapping patches Lastly, we assessed the impact of averaging overlapping patches. Implementing patch averaging improved the performance compared to not using it, as evidenced by the comparison between the proposed method and the No Averaging column in Table 1. Although, despite the small difference in the metrics, averaging overlapping patches effectively eliminates artifacts that occur at patch borders due to overlapping regions, resulting in smoother and more coherent images (see Appendix C). This discrepancy underscores the limitations of metrics like FID in detecting high-resolution artifacts, highlighting the necessity of qualitative assessments.

5.2 Comparison

We compare Pixelsmith to state-of-the-art frameworks in the higher-resolution image generation task, as well as to SDXL, which serves as the base model for our adaptation. To ensure a fair comparison, we include only models with native resolutions comparable to ours, excluding those trained at resolutions smaller than 1024 2 . While Pixelsmith outperforms other methods in most metrics (see Table 2), its real advantage lies in its ability to adapt to the unique characteristics of each individual image generation. This flexibility enables precise control to emphasize details and reduce artifacts. In Figure 5, we qualitatively show how Pixelsmith maintain the image structure by adding fine-details while other models introduce duplications or artifacts. There, we highlighted with red arrows the duplications and with purple arrows spurious artifacts. Additional comparisons can be found in Appendix B.

* Inference time not available for FouriScale at 4096 2 resolution due to out of memory on an RTX 3090.

6 Discussion and Considerations

Pixelsmith can generate images at arbitrarily high-resolutions, but as the resolution increases, adding more generative details without introducing artifacts becomes difficult due to the smaller denoising patch size relative to the latent space. For example, a 32768 2 image maps to a 4096 2 latent space with a 128 2 denoising patch. The higher the resolution to generate in a single step, the higher the Slider value is required to suppress artifacts. However, increasing the Slider value reduces the finer detail generation and return an image which closely resembles the previous resolution, limiting the true quality of the higher-resolution image. Generating an image through a cascade generation approach allows for using a lower Slider value at each stage, providing more details and fewer artifacts compared to single-step generation, as the lower Slider settings can be more effective in this incremental process. Additionally, appropriate metrics for evaluating high-resolution images are lacking, and further research is necessary to improve quantitative evaluation methods, as highlighted in previous studies Jin et al. [2024] as well.

7 Conclusions

In this work, we introduced Pixelsmith, a text-to-image generative framework that leverages a pretrained foundation model to generate higher-resolution images. Our flexible method uses a base image generated at the lowest resolution, combined with the Slider mechanism, to ensure both structural coherence and fine detail enhancement in the resulting images. By employing patch-based denoising, we have significantly reduced memory demands, making the generation of ultra-highresolution images feasible on consumer-grade GPUs. Experimental results demonstrate that our model outperforms current state-of-the-art methods in both image quality and generation efficiency.

References

Marco Aversa, Gabriel Nobis, Miriam Hägele, Kai Standvoss, Mihaela Chirica, Roderick MurraySmith, Ahmed M. Alaa, Lukas Ruff, Daniela Ivanova, Wojciech Samek, Frederick Klauschen, Bruno Sanguinetti, and Luis Oala. DiffInfinite: Large mask-image synthesis via parallel random patch diffusion in histopathology. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 78126-78141. Curran Associates, Inc., 2024.

Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. MultiDiffusion: Fusing diffusion paths for controlled image generation. In Proc. 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.

Mikołaj Bi´ nkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. arXiv preprint arXiv:1801.01401, 2018.

Tim Brooks, Aleksander Holynski, and Alexei A Efros. InstructPix2Pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402, 2023.

Pascal Chang, Jingwei Tang, Markus Gross, and Vinicius C. Azevedo. How I Warped Your Noise: a Temporally-Correlated noise prior for diffusion models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=pzElnMrgSD .

Emily L Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using a Laplacian pyramid of adversarial networks. Advances in neural information processing systems, 28, 2015.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.

Zheng Ding, Mengqi Zhang, Jiajun Wu, and Zhuowen Tu. Patched denoising diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2023.

Ruoyi Du, Dongliang Chang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. DemoFusion: Democratising high-resolution image generation with no $$$. arXiv preprint arXiv:2311.16973, 2023.

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206, 2024.

Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M Susskind, and Navdeep Jaitly. Matryoshka diffusion models. In The Twelfth International Conference on Learning Representations, 2023.

Lanqing Guo, Yingqing He, Haoxin Chen, Menghan Xia, Xiaodong Cun, Yufei Wang, Siyu Huang, Yong Zhang, Xintao Wang, Qifeng Chen, et al. Make a cheap scaling: A self-cascade diffusion model for higher-resolution adaptation. arXiv preprint arXiv:2402.10491, 2024.

Moayed Haji-Ali, Guha Balakrishnan, and Vicente Ordonez. ElasticDiffusion: Training-free arbitrary size image generation. arXiv preprint arXiv:2311.18822, 2023.

Yingqing He, Shaoshu Yang, Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang, Ran He, Qifeng Chen, and Ying Shan. ScaleCrafter: Tuning-free higher-resolution visual generation with diffusion models. In The Twelfth International Conference on Learning Representations, 2023.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in neural information processing systems, 30, 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.

Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.

Linjiang Huang, Rongyao Fang, Aiping Zhang, Guanglu Song, Si Liu, Yu Liu, and Hongsheng Li. Fouriscale: A frequency perspective on training-free high-resolution image synthesis. arXiv preprint arXiv:2403.12963, 2024.

Juno Hwang, Yong-Hyun Park, and Junghyo Jo. Upsample guidance: Scale up diffusion models without training. arXiv preprint arXiv:2404.01709, 2024.

Zhiyu Jin, Xuli Shen, Bin Li, and Xiangyang Xue. Training-free diffusion model adaptation for variable-sized text-to-image synthesis. Advances in Neural Information Processing Systems, 36, 2024.

Cornelius Lanczos. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. 1950.

Yuseung Lee, Kunho Kim, Hyunjin Kim, and Minhyuk Sung. SyncDiffusion: Coherent montage via synchronized joint diffusions. Advances in Neural Information Processing Systems, 36:5064850660, 2023.

Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Ming-Yu Liu, Kai Li, and Song Han. DistriFusion: Distributed parallel inference for high-resolution diffusion models. arXiv preprint arXiv:2402.19481, 2024.

Zhihang Lin, Mingbao Lin, Meng Zhao, and Rongrong Ji. AccDiffusion: An accurate method for higher-resolution image generation. arXiv preprint arXiv:2407.10738, 2024.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461-11471, 2022.

Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. arXiv preprint arXiv:1812.01608, 2018.

Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038-6047, 2023.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.

Will Depue/ OpenAI. What's new with dall·e 3?, Nov 6 2023. URL https://cookbook.openai. com/articles/what_is_new_with_dalle_3 .

Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11410-11420, 2022.

Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Marc Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. In The Twelfth International Conference on Learning Representations, 2024.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022a.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022b.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. Advances in neural information processing systems, 29, 2016.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5B: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294, 2022.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.

Jiayan Teng, Wendi Zheng, Ming Ding, Wenyi Hong, Jianqiao Wangni, Zhuoyi Yang, and Jie Tang. Relay diffusion: Unifying diffusion process across resolutions for image synthesis. arXiv preprint arXiv:2309.03350, 2023.

Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. DiffFit: Unlocking transferability of large diffusion models via simple parameter-efficient finetuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4230-4239, 2023.

Zhuoyi Yang, Heyang Jiang, Wenyi Hong, Jiayan Teng, Wendi Zheng, Yuxiao Dong, Ming Ding, and Jie Tang. Inf-DiT: Upsampling any-resolution image with memory-efficient diffusion transformer. arXiv preprint arXiv:2405.04312, 2024.

Shen Zhang, Zhaowei Chen, Zhenyu Zhao, Zhenyuan Chen, Yao Tang, Yuhao Chen, Wengang Cao, and Jiajun Liang. HiDiffusion: Unlocking high-resolution creativity and efficiency in low-resolution trained diffusion models. arXiv preprint arXiv:2311.17528, 2023.

Huangjie Zheng, Zhendong Wang, Jianbo Yuan, Guanghan Ning, Pengcheng He, Quanzeng You, Hongxia Yang, and Mingyuan Zhou. Learning stackable and skippable lego bricks for efficient, reconfigurable, and variable-resolution diffusion modeling. In The Twelfth International Conference on Learning Representations, 2023.

Qingping Zheng, Yuanfan Guo, Jiankang Deng, Jianhua Han, Ying Li, Songcen Xu, and Hang Xu. Any-size-diffusion: Toward efficient text-driven synthesis for any-size HD images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7571-7578, 2024a.

Wendi Zheng, Jiayan Teng, Zhuoyi Yang, Weihan Wang, Jidong Chen, Xiaotao Gu, Yuxiao Dong, Ming Ding, and Jie Tang. CogView3: Finer and faster text-to-image generation via relay diffusion. arXiv preprint arXiv:2403.05121, 2024b.

A Improving the base generation

We show the qualitative improvements Pixelsmith provides to the base model through an example here. We observe that Pixelsmith improves the generation at higher image generation resolutions. In Figure 6, we can see that relatively small and larger areas in the image are also improved after each iteration of the image generation. The final image resembles the actress more in terms of facial attributes and has more finer details in terms of multiple physical characteristics.

×1(1024×1024)

x4(2048×2048)

x16 (4096×4096)

B Additional comparisons

ScaleCrafter

DemoFusion

FouriScale

AccDiffusion

HiDiffusion

Pixelsmith (Ours)

C Averaging Overlapping Patches

Patch denoising may introduce artifacts as seen in Figure 8. This occurs because we sample overlapping patches from the latent space, which can result in parts of the same area being denoised differently across patches. We address this issue by averaging along the edges where the patches meet.

Figure 9 demonstrates our approach visually. Here, the image on the left represents the first noised latents. During the first denoising step (the next image), we sample a random patch (shown in blue) for the denoising process. After every denoising step, our algorithm checks for overlap between patches to perform averaging to remove any potential artifacts that may occur. The second denoising step (the third image), demonstrates a random patch sampled (in green) which overlaps with the patch in the previous denoising step. The third image shows an area of overlap where artifacts can be potentially generated. To alleviate this issue, we then take a tolerance of 10 pixels in the overlap but within the bounds of the first (blue) patch, and take an average across that region (as shown in yellow in the final image in the figure). From our experiments, we note that averaging across the edges of the patches, forces that region to create a smooth structure instead of horizontal and vertical lines (as shown in the second image in Figure 8).

i

O

In order to prevent degrading the quality of the final generated image, we do not include pixels exclusively from the patch of the second denoising step (pixels that are immediately adjacent to the edge of the random patch from the first denoising step and fall within the second patch) as they primarily contain noise and this degrades the quality of the entire image.

In Figure 8, for illustration purposes, we keep the higher-resolution at 1536 2 . The Slider here is at position 0 as we want to encourage the algorithm to create artifacts that are easily distinguishable, as a stronger guidance results in them being less noticeable. We only use three inference steps here to keep the comparison as close to the base resolution as possible.

D Multi-Scale Duplications

Existing patch diffusion approaches risk introducing artifacts with each denoising patch. This phenomena is more prominent at higher image resolutions as the increase in the image resolution leads to an increase in the size of the latent space while the size of the denoising patch still remains constant. Duplications are especially prominent in approaches like Du et al. [2023] where a fixed number of intermediate resolution steps may amplify this issue progressively with every step. We demonstrate this in Figure 10. Although we appreciate that using many steps helps refine the image gradually, depending on the generation the steps may have the negative effect of progressively amplifying a generated artifact at a lower resolution. For this reason we created Pixelsmith to be flexible in its image generation capabilities. Depending on the required result and text-prompt, we allow generating a higher-resolution image without generating any intermediate resolutions first. Image generation at intermediate steps (see also Appendix E is treated as a hyperparameter in our framework.

10242

20482

30722

DemoFusion

Ours

E Multi-Step Generation

1360x768

5440x3072

3400x1920

5440x3072

In Pixelsmith, generating the required resolution directly after the base model helps with removing multi-scale duplications of unwanted artifacts (Appendix D). However, allowing to generate multiple intermediate higher-resolution images before the required resolution also leads to a much better definition of objects present in the image at the higher-resolution. We demonstrate this with an example, as shown in Figure 11. We generate a 5440 × 3072 image from a base image of resolution 1360 × 768 directly, as well as going through the intermediate step of generating a 3400 × 1920 resolution image before the final resolution. We amplify the same region in all generated images to show the advantage of having an intermediate generation. Both 5440 × 3072 images are highly

detailed, but comparing the amplified region ( zoom in the image to see in more detail), we can clearly see the high frequency components in the image, such as the grass, are a lot more well defined even though they appear as pixelated regions in the base image. There can however, be cases where this difference in details in the images are not needed/noticeable and a one step image generation would be the preferable and faster choice in such a case.

F Super-Resolution Comparison

In addition to comparing our performance with current state-of-the-art models for higher-resolution generation, we also compare our results with a diffusion super-resolution model. Following ScaleCrafter by He et al. [2023], we compare our results with a latent upscaler, but as our base model generates images at a 1024 2 resolution, choosing the 4 × upscaler leads to creating a lot of computational overhead. Hence, we choose the 2 × upscaler Rombach et al. [2022a] and evaluate on the 2048 2 image resolution. Table 3, shows that even though the super-resolution model is trained to upscale, we still outperform it across all metrics.

Super-Resolution

Ours

G Slider Position

A generated image depends on many factors, namely, the input text-prompt, the input negative text-prompt, the seed, and the number of inference steps to name a few. The generated images can differ across these factors and even across intermediate image resolutions in terms of shape, texture, colour and a plethora of other varying attributes. This is especially seen when using a model that was not trained on the data for the task at hand directly - making the task a lot more challenging. Varying sizes of objects in the image such as arbitrary big structures, differ from the smaller ones and miscellaneous textured surfaces in the images are different from smooth ones. To account for these changes, and to control and constraint image generation at higher-resolutions, we introduce The Slider. The Slider helps provide a varying amount of control to a user to create image generations depending on how much they want the previously generated scene to influence the next generation.

In a real world scenario, this is especially useful as depending on the type of image being generated and the prominence of artifacts that could potentially be introduced in the image, the user can switch between allowing more guidance from the previous resolution, whereas for truly unconstrained image generation, the Slider provides the added benefit of more freedom. We demonstrate the expressive power of The Slider in Figure 13.

We see more bees generated in the image on the left when we position the Slider value at 1 . This results in certain artifacts and changes the structure of the image. At the other extreme, setting the max value of the Slider ( 49 ) results in preserving the overall structure of the base image, while not introducing any novel aspects into the generated image. Setting the Slider to 27 allows the framework to trade-off between image generation and preserving the global structure where it generates more realistic look high frequency components like the hair on the bees' body. Depending on the different settings for the Slider we can constraint the image generation to be more free.

We observe from our experiments that lower values tend to generate images that are rich in fine detail but at the same time, are prone to changing the global structure of the image while creating duplications and artifacts. Higher values tend to keep the overall global structure of the image intact, but do not refine the image to create finer details. The Slider is a tunable hyperameter that provides a means to trade-off between freedom and constraints to create the best possible generation of an image from the text-prompt.

2304×4080

49

H Patch Denoising Masks

I Additional visualizations

x1 (1360x768)

x4 (2720×1536)

x16 (5440x3072)

x16 (4096×4096)

x1 (1024×1024)

x16(4096×4096)

x1 (1360×768)

x4 (2720×1536)

x1(1360×768)

x4 (2720×1536)

x1（1024×1024)

x256(16384×16384)

回

x953(31616x31616)

(988×988)

(988x988)

(988x988)

(988×988)

(988×988)

(988×988)

(988×988)

(988×988)

J Text-prompts

Figure 1:

lego astronaut, riding a lego horse on Mars

pearly gates of heaven, ethereal, intricate details, angels, cloud, 7th sky, eternal bliss, in an otherworldly universe, detailed, realistic, 8k uhd, high quality

photo realistic, ultra details, natural light ultra detailed portrait of a female necromancer, skeleton face volumetric fog, Hyperrealism, breathtaking, ultra realistic, ultra detailed, cyber background, cinematic lighting, highly detailed, breathtaking, photography, stunning environment, wide-angle

Landscape of a dark far away planet, rock and organic soil, glowing trees, UHD, masterpiece, trending on artstation, sharp focus, studio photo, intricate details, highly detailed, by Greg Rutkowski

photo of the dark sky with visible but very distant swirling galaxies and stars

Figure 2:

cat, close-up

Figure 4:

A village on a cliff, realistic transparent sea, fishing boats

Figure 5:

Photo of a zebra in a forest, black and white

Photo realistic oil painting of Robert De Niro

Figure 6:

photo of Emma Watson, on a bike

Figure 7:

Darth Maul in a white tuxedo, in a forest, photo realistic

Raw photo of a roman warrior looking at the camera, close-up portrait, high detail, masterpiece

ultimate Japanese style, ink wash. Black and White, japanese Writing in Red and Blue, trending on ArtStation , intricate details, masterpiece, best quality, Use Dream Diffusion Secret Prompt, Epic

Figure 8:

Close up photo of the human eye iris

Figure 10:

Astronaut on Mars

Figure 11:

masterpiece, best quality, high quality, extremely detailed CG unity 8k wallpaper, scenery, outdoors, sky, cloud, day, no humans, mountain, landscape, water, tree, blue sky, waterfall, cliff, nature, lake, river, cloudy sky, award winning photography, Bokeh, Depth of Field, HDR, bloom, Chromatic Aberration, Photorealistic, extremely detailed, trending on artstation, trending on CGsociety, Intricate, High Detail, dramatic, art by midjourney

Figure 13:

close-up photo of a bee, macro, 100mm lens, bokeh, depth of field, high quality

Figure 12:

samurai iron man, dark black samurai armor, Artgerm, photorealism, Unreal engine 5 highly rendered, glowing red eyes, Tilt-shift 4k,White Balance, High Contrast, Low Saturation, Bracketing Detailed.

macro photography of a sakura blossom flower, epic

photo of a cute fluffy cat, close-up, high detail

Figure 15:

a turian on andromeda, 50mm lens

Figure 16

portrait photo of a beautiful Greek girl, with black pigtails, wearing fluffy white coat, wide mid shot, derelict building, soft lighting, Hasselblad, Voigtländer 50mm lens, in style of Oleg Oprisco

photography in the style of detailed hyperrealism ,cinematic composition, dramatic light, a glass with a forest in it, surrealism 8k, graphic of enchanted terrarium, 4k highly detailed digital art, 3D render digital art, 3D render stylized, stylized 3D render, highly detailed scene, 3D landscape, 3D landscape, 4 k surrealism, sci-fiish landscape, 3 D artistic render, surreal 3D render. Dreamlike, mysterious, provocative, symbolic, intricate, detailed, expressive,hyper detailed,intricate,poster,artstation

The gladiator stands after the battle, fear and horror on his face, tired and beaten, sand on his face mixed with sweat, an atmosphere of darkness and horror, hyper realistic photo. In post-production, enhance the details, sharpness, and contrast.

aerial photo, amazon forest

mosaic of van Gogh

Figure 17:

photo of the dark sky with visible but very distant swirling galaxies and stars

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims in the paper are studied, there are explanations for all of them.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: see Section 6

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: There is no theoretical result.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In Section 5 we detail how we did the experiments. All images can be easily generated and given resources the metrics as well.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Yes the code will be open-sourced and the data to evaluate is already opensource.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The training is irrelevant as we did not train any model. Details about evaluating are provided (see Section 5). We cannot share the subset to evaluate as we do not hold the rights but an official download link will be provided. Code will be provided as well.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We follow previous works in the field. We use relevant metrics and rely on qualitative comparisons.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: the used GPU is mentioned. Time of inference is also provided.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We use a pre-trained diffusion model which should be used as every other generative model, with care. We do not fine-tune or change it in our work.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: Our work does not change the societal impact of diffusion models as the generation remains the same as the pre-trained model.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: We use a pre-trained model without altering the weights.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Everything is cited and the pre-trained model is free to be used for research.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: A pre-trained model is used, we are not realeasing it for the first time Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We use an open sourced pre-trained model

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We use an open sourced pre-trained model

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors propose a number of engineering tricks which enable generating at higher resolutions from a pre-trained txt2img diffusion model.\nNotably, the requirements for the proposed method are relatively low.\n\nIn the proposed approach, an image of a standard resolution is generated first.\nAfter that, is it upsampled (in RGB space) to the desired size and used as a guidance for the new, truly high-resolution image which is generated as a sequence of overlapping patches.\nThe guidance mechanism is implemented by mixing the imaginary component of Fourier decomposition of the currently denoised and guidance patches.\nThis procedure takes place not for all the denoising steps, but only up to certain noise level referred to as Slider.\nThe value of Slider controls how similar are the original standard-resolution image and its higher-resolution version.\n\nPerformance of the method is evaluated with commonly used metrics such as FID, KID, IS and CLIP-similarity.\nAlso, a lot of samples are provided for visual inspection.\n\nSTRENGTHS:\nAccording to the provided metrics and visual results, the methods performs quite well in comparison with recent baselines. \nThe fact that it requires only 7.4 GB of memory makes it very affordable for the community. \nThe idea of mixing in the spectral domain, to the best of my knowledge, is novel enough in context of ultra-high resolution sampling from a pre-trained model. \nExploiting the checkerboard pattern for mixing with guidance is also an interesting engineering trick.\n\nWEAKNESSES:\n1. While the authors several times mention that \"the imaginary part in the frequency space contains most of the low-frequency information of the image\" (line 187) and \"imaginary part provides more structural information than the real part\" (line 249), they do not provide any references for this statement or supporting experiments. To me, this fact does not look evident enough. I am sure this needs more thorough explanation since otherwise it looks just like an empirical trick.\n\n1. From the text, it is unclear which Slider value was used to obtain the numbers from Table 2. Was it the value of 30, as defined in line 244? Also, what exactly is \"using next inference step (NIS)\" (see line 253)? Is this about checkerboard mask mixing?\n\n1. The presentation of Fig. 3 is not easy for understanding, I advise considering its redesign. For example, forward diffusion and taking imaginary part of FFT are denoted in the same way. \n\n1. It is unclear from the text what the guidelines are for selecting the size for the zone of averaging for overlapping patches (line 805).\n\nQUESTIONS:\nI ask the authors to address the weaknesses listed above during the rebuttal period. In particular, I am interested in the deeper justification of  using imaginary Fourier coefficients than empirical evidence.\n\nPOST-REBUTTAL UPDATE: I keep my initial score. I think that although the method is very simple, this does not mean that the submission is bad. I believe that with improved presentation this paper can be interesting for the practitioners.\n\nLIMITATIONS:\nNo actions needed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

comments: SUMMARY:
This paper proposes a training-free method for diffusion models to sample images at higher resolution with limited GPU memory, which introduces several tricks, such as fourier merging, chess-mask deduplication, and slider control. Experiments show the effectiveness of the proposed method.

STRENGTHS:
1. The fourier merging seems interesting with imaginary part to maintain the global structure;
2. The chess mask seems effectively eliminates the duplicate artifacts in high resulotion generation;
3. The ablations are sufficient and comparisons show the superiority of the proposed method.

WEAKNESSES:
1. Is there any literatures to support that the imaginary part of the fourier transform corresponding to the low-frequency of the signal, or is there any deeper analysis beyond the ablation results in Tab.1 to illustrate this? By the way, it is suggested to provide the visual ablations of this part.

QUESTIONS:
1. How to determine the patch orders in high-resolution generation, is it random orders or follow some principles.
2. From Tab. 2, it is not clear why Pixelsmith slower than Scalecrafter at 2048 resolution, while nearly 2x faster at 4096 resolution. Is there any trend that the bigger, the more efficient?
3. Is the content of the final scaled high resolution output significantly differ from the base output, whether the new objects will be generated?

LIMITATIONS:
The authors have adequately discussed the limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Pixelsmith, a framework designed to utilize pre-trained diffusion models to enable high-resolution image generation using only a single GPU.\nPatch-based denoising ensures that the entire generation process can be accommodated on a single GPU.\nThe Slider mechanism balances the trade-off between finer details and overall structure by controlling the transition from guided generation to native generation.\nGuided generation operate on the latent fused by one upscaled higher-res image patch and one native-generated higher-res image patch in Fourier space, helping to maintain global structures as claimed by the authors.\nExperimental results show that Pixelsmith not only produces high-quality and diverse images but also improves sampling time and reduces artifacts compared to existing techniques.\n\nSTRENGTHS:\n1. Memory Efficiency with Limited Computational Resources: Achieving high-resolution image generation on a single GPU is a good contribution, and the image generation quality is not compromised by this restricted setting.\n2. High-Quality Generation: The proposed Fourier space fusion maintains fine-grained details and global structures, while also preventing some artifacts that occur in other methods.\n\nWEAKNESSES:\n1.\tPaper Structure Issues: The content before the methods section is too lengthy, making the key method and experiments sections comparatively short and harder to fully understand. These are elaborated upon in the following points.\n2.\tMethod Description is Hard to Follow: For instance, the overview in Lines #171-173 does not fully describe all the components involved in the method, requiring readers to review the paper multiple times to understand the entire pipeline. Additionally, the intuition behind combining $\\hat{z}^{iFFT}{t-1}$ and $\\hat{z}^{guid}{t-1}$ is not explained, which is confusing since $\\hat{z}^{iFFT}_{t-1}$ already incorporates information from $\\hat{z}^{guid}_t$.\n3.\tErrors and Mismatches in the Experiment Section: According to Table 1, using the real part is better, which contradicts Line #248. Line #252 describes the wrong column for \"the next inference step\". Also, the term “base model” in Table 2 is unclear—does it refer to the same model as Table 2’s Pixelsmith and Figure 1’s base model?\n4.\tFlaws in Figure Illustrations: In Figure 1, the base model image for “x256 (16384x16384)” is not displayed. Is this base model the same as the one mentioned in Table 1? In Figure 3, the illustration should be more concrete and easier to understand; however, some variable names are too small and there are too many blank areas. The fusion part is also unclear, with no mention of the real part of the Fourier transform and no explicit explanation of $\\mu( . , . )$.\n\nQUESTIONS:\nThe aim of achieving high-resolution image generation on a single GPU is commendable, and the proposed method successfully accomplishes this goal. However, the presentation quality is relatively poor, especially with some key parts described incorrectly. I hope the authors can address the questions mentioned in the weaknesses section and resolve my concerns.\n\nLIMITATIONS:\nThe authors have discussed the trade-off between achieving finer details and suppressing artifacts. They have also suggested proposing appropriate metrics for evaluating high-resolution image generation, which would be beneficial for the community.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework for generating high-resolution images from text prompts using pre-trained diffusion models. The key innovations are: A cascading approach that uses lower-resolution generated images as guidance for higher resolutions. A \"Slider\" mechanism to control the balance between following the guidance and allowing novel generation. Patch-based denoising to enable generation of arbitrarily large images on a single GPU. Averaging techniques to reduce artifacts from patch-based generation. The authors demonstrate that Pixelsmith can generate images up to 32,768 x 32,768 pixels on a single GPU, outperforming existing methods in terms of quality and efficiency.\n\nSTRENGTHS:\nThe proposed method enables generation of ultra-high resolution images without additional training, addressing an important limitation of current models.\n\nGood results.\n\nThe patch-based approach allows for generating massive images on consumer GPUs, which is a significant practical advantage.\n\nWEAKNESSES:\n1. Poor presentation. The writing of this paper needs substantial improvement to be published in a venue like NeurIPS. First, the language of the paper needs to be improved. Second, the paper spends a lot of space on unnecessary content. The method in this paper is not difficult, but it is very difficult to understand the method in one reading. In terms of writing, the authors did not succeed in emphasizing the core concept of their proposed method. In addition, the length of the paper is too long. Such a simple method, but the experiments are not introduced until page 8, which should not happen.\n2. The core idea of ​​this paper seems to be to generate once and then upsample the generated image by patch-wise processing. (By the way, Figure 3 is difficult to understand and Eq. 4 is not explained). This method is essentially not much different from other operations that use diffusion for upsampling. So what is the advantage of this method? Why is this method feasible?\n3. The role of FFT Transformation is not well demonstrated. No exploratory experiments are used to demonstrate the motivation and effect of this method.\n\nQUESTIONS:\nSee Weaknesses please.\n\nLIMITATIONS:\nThe authors claim they discussed the limitation but I didn't find it in the text. Correct me if I am wrong.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-18"}