Is One GPU Enough? Pushing Image Generation at Higher-Resolutions with Foundation Models.
Authors: Athanasios Tragakis
Marco Aversa
Chaitanya Kaul
Roderick Murray-Smith
Daniele Faccio
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion model
generative AI
image generation
foundation models
higher resolution
Primary area: diffusion_based_models
TL;DR: Pushing Image Generation at Higher-Resolutions with Foundation Models using a single GPU

Abstract: In this work, we introduce Pixelsmith, a zero-shot text-to-image generative framework to sample images at higher resolutions with a single GPU. We are the first to show that it is possible to scale the output of a pre-trained diffusion model by a factor of 1000, opening the road to gigapixel image generation at no extra cost. Our cascading method uses the image generated at the lowest resolution as baseline to sample at higher resolutions. For the guidance, we introduce the Slider, a mechanism that fuses the overall structure contained in the first-generated image with enhanced fine details. At each inference step, we denoise patches rather than the entire latent space, minimizing memory demands so that a single GPU can handle the process, regardless of the image's resolution. Our experimental results show that this method not only achieves higher quality and diversity compared to existing techniques but also reduces sampling time and ablation artifacts.

Is One GPU Enough? Pushing Image Generation at Higher-Resolutions with Foundation Models.
Athanasios Tragakis" University of Glasgow
Marco Aversa" Dotphoton
Glasgow, United Kingdom
Zug, Switzerland
tragakis. @@research, .gw.glll.... uk
marco www..aavrraaadotooooooooom
Chaitanya Kaul University of Glasgow
Roderick Murray-Smith
University of Glasgow
Glasgow, United Kingdom
Glasgow, United Kingdom
chai thaitanya. kaul@glasgon ac uk
Roderick Murray www.aray--ttttggggggooooooooooo ac. uk
Daniele Faccio
University of Glasgow
Glasgow, United Kingdom
Daniele. Ftw.eleeaFacciggllsggoooooo ac
Abstract
In this work, we introduce Pixelsmith, zero-shot text-to-image generative frame- work to sample images at higher-resolutions with single GPU. We are the first show that is possible scale the output of pre-trained diffusion model by factor of 1000, opening the road for gigapixel image generation at no additional cost. Our cascading method uses the image generated the lowest resolution as a baseline to sample at higher-resolutions. For the guidance, we introduce the Slider, tunable mechanism that fuses the overall structure contained in the gest-generated image with enhanced fine details. At each inference step, we denoise patches rather than the entire latent space, minimizing memory demands such that single GPU can handle the process, regardless of the image's 's resolution. Our experimental re- sults show that Pixelsmith not only achieves higher quality and diversity compared to existing techniques, but also reduces sampling time and artifacts
Introduction
Recent advances in diffusion models (DMs) have revolutionized the field of high-fidelity image generation. Foundational works like Ho et al. 2020), Sohl-Dickstein et al. [2015). [ong and Hrmon
[2019] established the groundwork, leading significant breakthroughs demonstrated by Dhariwal
and Nichol [2021] These models have evolved rapidly, with innovations such as new sampling techniques Lu al. 2022)] Song
2020 and capabilities for inpainting Lugmayr et al.) [2022 and image editing Brooks et al. 2023] Mokady et al. (2023), Nichol et al. [2021]. However, even though DMs have shown remarkable results high-fidelity image generation, scaling them to
high- nigh-resolutions is still an open challenge. The introduction of the latent diffusion model (LDM)
Rombach et al. [2022b] has made high-resolution image synthesis more accessible. However, most
pre- pre-trained DMs based on the LDM are limited to generating images with maximum resolution of
"Equal Contribution.
'The code for work is available at http::/hhann--ddhiiiitba.aaaiiie
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1024 pixels due to constraints in computational resources and memory efficiency. Following this approach, aiming to ultra-high-resolution generation would incur additional training and data costs, and the model would be unable to run single GPU.
Recently, several works have focused scaling pre-trained models to higher-resolutions, highlighting new possibilities and challenges in the Aversa et al.) [2024), Du et al. 20023] Gu et al. [2023], Hc al. 2023]. This allows already available models to be used with no extra costs and no additional carbon footprint. However, most methods addressing this problem either require expensive GPUs, as memory demands increase with resolution or prolonged generation times. Moreover, scaling the native resolution of a generative foundation model to higher- resolttoss introduces artifacts due to the direct mapping into prompt-image embedding space. For a specific prompt, we would expect to sample an image matching the description and of the same size as the training data images. Consequently, even scaling up by a factor of would lead to the duplication of the image produced by the prompt across the higher-resolution image. However, since the images in the training set contain information at different resolutions, we can leverage this prior knowledge implicitly learned by the diffusion model to guide the generation at nigher-resolution and enhance fine details. Addressing these challenges crucial for applications that demand ulrra-high-resolution images, such as gigapixel photography, medical imaging, satellite imagery and high-definition digital art.
To overcome the challenges ultra- setrr-high-reee nigh-tteolution image generation, we introduce Pixelsmith, an adaptable framework that utilizes pre- pre-trained generative models for scalable gigapixel synthesis.
Our contributions outlined follows: 1. We introduce Pixelsmith, the first framework capable of generating gigapixel-resolute images using a single GPU. 2. We develop the Slider, a dynamic tool that allows users to adjust the balance between overall image structure and fine-detail enhancements in the generation process. 3. We enhance and adapt the random patch denoising strategy to texl-to-image pre-trained diffusion models, leading minimized memory usage. We provide masking method that, combined with the Slider, reduces the number of artifacts at higher-resolutions.
2 Related Work
Pre-trained DMs trending toward increasing native resolutions. Notable examples include Stable
Diffusion (SD) Rombach et al. (2022b), which started at 5122, then 7682 in SD 2, and 10242 in SDXL Podell al. [2023]. SD Cascade Pernias et al./ [2024), and SD 3 Esser al. (2024). Similarly, DALL-E Ramesh et al. [2021] continues to increase resolution with each version OpenAl Nov
2023 This trend shows a growing demand for higher-resolution generation.
Currently, high-resolution image generation often involves a super-resolution model applied after the initial text-to-image generation (Saharia et al. [2022]. This additional model increases costs due to training and -omain-specific fine-tune the-tuning requirements.
2.1 Trained Models
Trained models are those that are specifically designed for multi-resolution generation. Recent models like Matryoshka Gu et al. [2023) can generate various resolutions up 10242 through progressive training schedule. However, scaling beyond this is limited. Cog View3 Zheng et al. (022b)) based on Relay (Teng al. ). upsamples from base resolution of 5124 though higher-resolutions like 4096² remain a future goal. Fine-tuning methods such as DifiFit Xie et al. [2023] are costly, requiring 51 V100 GPU days. ASD Zheng et al. [2024a] introduces memory-efficient sampling method capable of theoretically generating images up to 18432² resolution. Patch-DM Ding et al. 2202).. by training on 642 patches, can generate resolutions like 1024x512. LEGO Zheng al. 2023 and Inf-DiT Yang et al. [2024] also rely on training.
2.2 Adapted Models
Adapted models, on the other hand, modify pre- praitedi models to generate higher-resolution, without additional training. MultiDiffusion Bar-Tal et al. (0033) offers controllable generation through multiple processes but is slow and prone to structural errors Du et al. (202). He et al. (2023). Zheng et al. [2024a]. ScaleCrafter He al. (0023 improves resolution by altering the convolution
kernel dilation but faces memory limitations. DemoFusion Du et al. 20233 adapts MultiDiffusion

---

10001
20554
X64 1819228192
x64 x6144x 10880)
&1
185
033x331
33x+2
x16 15440*30E
x1024 (32382<32382)
2080 + 20BCI 11056 1056] 1056x8001
Figure Examples of generated images using Pixelsmith. The proposed framework generates images on higher-resolutions than the pre-trained model without any fine-tuning. Images different resolutions are shown with cut-out areas for both Pixelsmith and the base model. The higher- resolution images are in scale with the images generated by the base model. Only the lower resolution version of the gigapixel image has been resized for a better visualisation. Some cut-outs of the gigapixel generation have resolution close to the base model which is 10241 and it can be seen that the images are comparable in aesthetics showing that our framework is capable of true gigapixel generations (zoom to see in better detail).
for constant memory use, though it is time-consuming. ElasticDiffusion Haji-Ali et al. 2023 and
SyncDiffusion Lee al. 2023 aim for specific high-resolution goals, but object repetition remains a problem at large scales Jin et al. [024]] Recently, HiDiffuusion ZZhang et al. 2023 introduced modifications the UNet architecture
limited maximum resolution of 40962 due to high GPU memory requirements. AccDiffusion Lin et al. [2024] employs patch-contentt-waare prompts and adjusts the attention masks within the UNet to suppress artifacts, but these changes result in blurry images. Similarly, Fouriscale Huang et al. [2024] another model that alters the UNet but exhibits issues fighhe--rsolutionns particularly at 4096 Current approaches excel in some areas but fall short in others. To enable high-resolution generation on single consumer GPU, models must adapted to avoid retraining costs. Efficient memory usage is crucial to prevent the need for expensive, high-memory GPUs, and the process must be fast and artifact-free. We propose a flexible framework that addresses these challenges based on text conditions.
prevent object duplication; however,

---

Latents
2y'
#--
but
Timesteps per pixe
dimesieps
Figure 2: Overview of the patch denoising process proposed by DiffInfinite: The top row represents the latent space, while the bottom row tracks the timesteps for each pixel. Each pixel should be denoised only once per timestep, so when overlapping occurs, already denoised pixels revert their previous values from the prior timestep. After denoising, these reverted pixels are restored to their original denoised state from the current timestep.
3 Foundations
3.1 Diffusion models
DMs Ho et al. 2020], Sohl-Dicksteii et al. [2015] are probabilistic generative models that first add noise to distribution during diffusion and then learn to remove this noise during denoising. This way, during training, a Gaussian probability distribution learnt and during inference sampling from the Gaussian leads to the data probability distribution. Executing this process in the latent space Rombach et al. 22222b is more resource efficient allowing for faster training and inference times. In Latent Diffusion Model, let 20 Ex(2) represent the clean training data x q(x) mapped to the latent space by the VQ- VAE encoder Eg. Beginning from 2g. at each timestep t E {1...., T', noisy latents 21...., 2T g² 2t-1) == N(zz; /1 /1- 2 3,I) are sampled according
noise schedule
function based on the timestep 3r E [o. 1]. During training, the model learns to denoise these latent variables using conditional prompt c. At inference time, starting from random latent 2T NN
the model generates by iteratively sampling 2q ~ po(2o Zr; c) = II P= 2t, c)-
3.2 Patch sampling
The default LDM denoising process samples the entire latent space at each timestep, which becomes resource- -inturccinttesivee the resolution increases. Methods like Li et al. [2024] distribute this load across multiple GPUs, but this requires expensive hardware. To address this, we adapted and refined the DiffInfinite sampling method Aversa et al. [2024] for text-to-image DMs, enabling ultra-high- resolution generation efficiently on a single GPU.
At each timestep, random patches are selected for denoising, and this process is repeated until the entire latent space is denoised. The randomness controlled for efficiency. We track which pixels have been denoised at each timestep, and select areas where pixels have not been denoised yet. This sampling process avoids excessive inference times, especially very high-resolutions like 32768 In our experiments, we sample images at higher-resolution using fixed 128 patches in the latent space. To handle overlapping patches, pixel values are temporarily reverted when already denoised pixels are encountered, ensuring correct processing. After the current patch is denoised, the pixels that were reverted are restored to their denoised values. Only the first denoising each pixel is retained for each timestep. preventing redundant denoising (see Fig. 2p.
Despite its advantages, the Difflnfinite sampling method still relies on segmentation masks to condition each patch, providing rich spatial information. However, in text-to- text-to-image DMs, where global text prompt conditions the same text prompt to each patch results in repetitive content and poor-quality generations. To overcome this, we introduce guiding mechanism that incorporates structural information between patches alongside the text prompt, improving the quality and diversity of the generated images.
entire latent space, this method insufficient on its own. Applying

---

IMAGE GUIDANCE GENERATION
Upsampling process
"Bust soulpture lacooon
Text to Image generation
Decoding
Upocaler
HIGHER- RESOLUTION IMAGE GENERATION
HIGHER-RESOLUTION
Zoameck-n higher-reaclution
Image guidance proparation oni
piif
guld
pult
Balan
4. Image generation
Quided
Pure aaiinaation
Prorret
machardem
Donolsing
Ducoding
Guidaacee mechanism
IPPT
Meon
Manki no
Frompt
Denclsing
Figure 3: Proposed framework overview. Text-to-image Generation: A pre-trained text-to-image diffusion model generates initial image based on the input text prompt. 2. Upsampling process: The generated image is upscaled (in this use case by factor x4) and encoded into the latent space to guide the creation of higher-resolutee is degraded through the diffusive forward model, creating the guidance latents. 4. Image generation: the Slider (indicated by blue line) adjusts the extent of guidance. Left of Slider (Guided Generation): guidance latents control the image generation. The framework fuses guidance latents (green patches) with high- nigh-resolution latents (purple patches) using the Fast Fourier Transformation (FFT). The phases are averaged and combined with the amplitude, then transformed back via the inverse FFT (iFFT). A chess-like mask integrates information from the successive guidance step (orange), resulting in fully processed patches (cyan). Right of Slider (Pure Generation): : the generation relies only on the prompt. Higher-Resolution Comparison: while the base model upscales the bust with disfigured hands, the proposed method enhances details, corrects distortions, and prevents new artifacts.
image. 3. Image guidance preparation: The encoded image
4 Method
4.1 Problem statement
Currently, pre-trained text-to-image LDMs generate a latent variable 20 ~ po(zol c) with fixed size using diffusion process, which is then decoded into pixel space using the decoder D, of VAE. In this section, we show how to leverage foundational generative models to sample arbitrarily large images based
given prompt. Without requiring additional training or fine-tuning, the LDM, trained on images with size H x W x 3, is adapted to generate images with size mH x nW 3, where m, are the scaling factors for achieving higher-resolution.
present Pixelsmith, a framework with flexible number of cascaded steps (Section 4.3) designed to generate images at altroltigh-resolutionn (Section 4.4) provides control over the generation process using patches (Section making resource-efficient regardless of the resolution.
a single GPU. The implementation of the Slider

---

4.2 Framework overview
In this section, we describe the workflow of Pixelsmith, detailing how the framework adapts pre- trained text-to-image LDMs to generate images with higher-resolution on a single GPU (see Fig. B. In order generate ullra-high-resolution images without artifacts, we introduce these key components: the Slider (see Sec. 4.4), patch averaging (see Sec. 4.5) and masking (.6.
Text-to-image generation First, given conditional prompt c, we use SDXL sample the latent variable 2g and decode it into pixel space as = Dz(2)) generating a 1024 resolution image.
Upsampling process After the image generation, we apply an upsampling algorithm to increase the image resolution (see Sec. 4.3 for details). In our case, we used Lanczos interpolation Lanczos) [950) to scale up the desired resolution. However, upsampling leads a blurred output and lack of additional content. The upsampled image, guuid will serve as guidance for our generative process.
Image guidance preparation Once the guidance image is encoded the VAE's latent space =0 = E we can easily sample each latent variable of the diffusion process through the forward diffusion process guid q(= gguid)
Image generation The generative process starts from 2T NO, I), which has the same dimen- sions as 2T ggudd
At each step, a random patch is cropped as described in Section 3.2
2 2(( C.J (z)), squid
-iss guid
where Cis, R4.h,, R.pp.p is a cropping function that extracts patches of size (ph; Pw) from the latent variables z, and =t zquid of size (h, w) at the coordinates (i.j). For simplicity, we will refer to the cropped latent patch =; as 2, and 2t (Q.5 sguid throughout this discussion.
The Slider's position (see Sec. 4.4 for details), indicated by blue line in Fig. 3, determines whether the guidance mechanism (see Sec. 4.4.T for details) or unguided patch denoising will be applied. In the unguided mode, each patch based solely on the previous one and the text condition, similar to conventional patch denoising process. The Slider allows control over whether a generated image will be slightly or significantly altered compared the previous resolution.
After the denoising process has ended, the latents 2g are decoded and the is generated. Using cascade upsampling approach, the generated image can be upsampled again, repeating the process to achieve an even higher-resolution image.
image
4.3 Higher generation in one step
Most of existing nigher-resolutin generative methods rely on cascade sampling approaches approaches(Denton Denton et al. 2015), Ho et al. [2022, Menick and Kalchbrenner (2018). Our approach, however, differs from previous works like Du al. [2023] Guo et al. 2024 two significant ways.
First, we perform upsampling directly the pixel space rather than in the latent space. Manipulating the latent space distribution with transformations like upsampling can introduce distortions that scheduler has not accounted for Chang et al. (2024), leading to degraded image quality[Hwang et al. [2024). Second, our flexible method enables higher-resolution image generation in two generative steps: first by producing the base image, then by enhancing to the desired resolution. For instance, Du et al. [2023] requires passing through intermediate resolutions (1024 20482, 3072²) to achieve a 4096* resolution, with even higher-resolutin necessitating additional steps. This results in prolonged waiting times and multi-scale duplications, as each step may produce duplicates at different resolutions (Appendix D).
In contrast, our framework can generate any resolution directly from the base one, making it feasible to scale up to a 32768 image directly from the base resolution of 10242. However, intermediate steps can sometimes yield better small-sized details (Appendix Eb. Our flexible approach supports both two-step generation and cascade generation, allowing users select their preferred sampling method depending on the use case.

---

base model
with masking
without masking
artifacts
Figure 4: Masking effects higher-resolution generation (x16 the original resolution). (left) Image generated using SDXL. (center) Image generated with Pixelsmith with masking. (right) Image generated with Pixelsmith without masking. We highlighted the artifacts introduced by generating at higher scales. These artifacts demonstrate the challenges of maintaining coherence and accuracy when scaling up the resolution without additional guidance.
4.4 Slider
Currently, higher-resolution generative models often encounter issues such as duplications and atypical anatomical features or structures He et al. [2023]. To mitigate these artifacts, introduce the Slider (indicated by a blue line in Figure 3). The Slider a parameter that determines at which denoising step there is the transition between the guidance mechanism introduced in Sec. and traditional unguided generation. The Slider can either constrain the denoising process to eliminate undesirable outputs allow for less constrained generation to add details.
and
When set to zero, the generation entirely unconstrained. This results in an image that might resemble the 1024 base image but will likely contain multiple duplicates. At = T, the generation highly guided, leading to an image that almost identical to the previous resolution. The optimal value for the Slider varies depending on the base image. Adjusting the Slider manually is necessary to achieve the best output. This manual adjustment allows for fine-tuning the balance between constrained and unconstrained generation, ensuring that the resulting image maintains the desired quality and level of detail. The ideal Slider setting is influenced by factors such as image complexity, the presence of fine details, and the desired resolution. Experimentation and iterative adjustments often required to find the most effective value for each specific image and resolution.
An example demonstrating the impact of the Slider's ' position can be found in Appendix G
4.4.1 Guidance mechanism
The guidance mechanism combines the random patches =g 2,, and a to generate the updated patch -1 First, the patches zquid and & are transformed into the Fourier space using the Fast Fourier Transform (FFT), where their phase components de and oi averaged:
FFT (2i)
A
3. arg ( + eropun
FFT
sguid'
guid e iogui
Here, Ar and A Auuid represent the amplitudes, while de and obuud are the phases The averaged phase o, is then combined with the amplitude A, from & form a new Fourier representation 3FFT = A, While the amplitude Ae preserves the intensity and contrast information, the phase averaging process ensures that the resulting phase contains structural characteristies from both
*Drectlly averaging phases, as dittd can lead to issues due to the periodic nature of sinusoidal functions. 2
For example. averaging 0 and é 2n should result in phase near zero, not æ. To correctly average phases, compute the arg(-) arctan(y/a (v/x) of the sum of the complex numbers' phase components.
7

---

the generated and guiding images. This modified Fourier representation is transformed back to the spatial domain using the inverse Fast Fourier Transform 2 iiFF (2 The output 2FFF is then used as the condition for the reverse diffusion process to generate the next patch 2: ~ Po (2 2/ c). This solution suppresses local artifacts and helps maintain low- frequency structural consistency between the reverse diffusion steps, reducing the likelihood of significantly altering the global properties of the generated image. However, the guidance mechanism alone still generates some long-range discrepancies across the image due prompt sharing between the patches. To mitigate these artifacts and prevent prompt duplications across the latent space, we introduce a masking method(see Section 4.6
4.5 Patch averaging
Overlapping patches can sometimes cause visible differences at their borders. To address this, we introduce a transition zone where the values of overlapping patches at timestep are averaged, producing a smooth and scamless denoised output. In Appendix [ we illustrate the artifacts caused by patch overlap and show how averaging effectively removes them.
4.6 Masking
One of the main reasons patch-based image generation in diffusion models Bar- Tal et al. 2023 suffers from artifacts at higher-resolutions is the use of the overall image text prompt for each patch during the denoising process. This leads to duplicate structures forming in the latent space. We partially addressed the spatial coherence with the guidance mechanism in Section 4.4.1 To further improve results, we combine the sampled 2/ mask A (Equation 3)
gguid using a chess-like 2/FTT with the image guidance 2f
gmasked ASFFFT + (1 - A) where A = Ao.j = { if is even
(3)
if if is odd
In Fig. 4 we show the comparison between a higher-resolution image generated with and without masking with the image guidance. The image on the right, which is not constrained by pixels from eguid exhibits more freedom in generation and, as a result, introduces notable artifacts. For example, ot
clouds transform into waves, and the blue sky morphs into the sea. Additionally, a small, scemingly artificial lake or river appears in the middle of the village, situated unnaturally close to the beach.
Experiments
Pixelsmith is tested on single RTX 3090 GPU, with all tested resolutions requiring 8.4 GB of memory. Performance is evaluated on the LAION-5B dataset dataset|Schullee Schuhmann et al. [022] by randomly sampling 1,000 image and text prompt pairs. The metrics used for evaluation are Fréchet Inception
Distance (FII) Heusel et al.] [2077], Kernel Inception Distance (KID) Binkowski et al.. (2018), Inception Score (IS) /Salmaas et al. (2016), and CLIP Score Radford et al. [202]]. with the FID metric computed using the clean-FID approach Parmar et al. 2022] (for further comparisons, see Appendix B Additionally, we compare our results with super-resolution model in Appendix F
5.1 Ablation study
We conducted quantitative examination of our framework resolution of 2048 x 2048 pixels, focusing on key factors that influence its performance: the Slider position, the role of amplitude and phase in the latent space, the importance of masking during guidance, and the impact of averaging overlapping patches. Our results are shown in Table
Slider position Our findings highlight the critical role of the Slider position in the quality of the generated images. Setting the Slider to position 30 (proposed method) provides an optimal trade-oft off between guidance and generation, outperforming other positions like 1 (SPI, no guidance), 24 (SP24, mid-point), and 49 (SP49, full guidance). Slider position of introduces numerous artifacts due to insufficient guidance, while a position of 49 lacks fine detail because it relies too heavily on lower-resolution. Position 24 has been chosen as mid-point the diffusion process, is closer the optimal value but does not yield the best results across dataset.
8

---

ScaleCrafter
DemoFusion
FouriScale
AccDiffusion
HiDiffusion
Pixelsmith (Ours)
2048
4096
2048
40962
Figure 5: Qualitative comparisons: This figure highlights how other models suffer from duplications (red arrows) and introduce artifacts in areas with complex, high-freew frequency patterns (purple arrows). In contrast, Pixelsmith effectively eliminates these issues. (zoom in see better detail).
Amplitude and phase In exploring the role of amplitude and phase within the latent space, we experimented with different configurations. The proposed method averages the phase of the guidance latents and the current latents while using the amplitude of the current latents, as detailed in the Methods section (see Figure 3). We compared with other setups: one that averages the amplitude of both latent spaces while using the phase of the current latents (denoted as AA in Table [b. and another that averages both amplitude and phase from both latent spaces (denoted as A&z)). The proposed method demonstrated superior performance in maintaining image quality and preserving fine details. Averaging the two phases provides guidance through structural information. while averaging the amplitudes only conveys the intensity of the frequencies. Averaging both offers information on both structure and intensity which overly influences the final image, leading to a lack of fine details expected in higher-resolution output.
Masking Masking emerges as significant factor enhancing image quality. The quantitative metrics indicate that the chess-like mask leads to slightly better scores compared to using no mask (as seen in the No Mask column of Table compared the proposed one). Visual inspections reveals that incorporating the mask is crucial for removing artifacts (see Figure
Overlapping patches Lastly, assessed the impact of averaging overlapping patches. Imple- menting patch averaging improved the performance compared to not using it, as evidenced by the comparison between the proposed method and the No Averaging column in Table Although, despite the small difference in the metrics, averaging overlapping patches effectively eliminates artifacts that occur patch borders due overlapping regions, resulting in smoother and more coherent images (see Appendix C). This discrepancy underscores the limitations of metrics like FID in detecting high-resolution artifacts, highlighting the necessity of qualitative assessments.
Table I: A quantitative examination of our framework through ablations.
Metrie
SP1
SP24
SP49
A
A&o
No Mask
No Aver.
Proposed
FID1
74,890
63.168
64.195
63.476
63.173
64.473
63.493
63.116
KIDI
0.008
0.002
0.003
0.002
0.002
0.003
0.003
0.002
ISt
19,928
19.941
19,290
19,933
19.897
19,892
19.626
24.242
CLIP
32.198
32.451
32.425
32.730
32.432
32.355
32.416
33.429

---

5.2 Comparison
We compare Pixelsmith to state-of-the-art frameworks in the higher-resolution image generation task, as well as to SDXL, which serves as the base model for our adaptation. To ensure a fair comparison, include only models with native resolutions comparable to ours, excluding those trained at resolutions smaller than 1024 While Pixelsmith outperforms other methods in most metrics (see Table 2 individual image generation. This flexibility enables precise control to emphasize details and reduce artifacts. In Figure we qualitatively show how Pixelsmith maintain the image structure by adding fine-details while other models introduce duplications or artifacts. There. we highlighted with red arrows the duplications and with purple arrows spurious artifacts. Additional comparisons can be found in Appendix B
its real advantage lies its ability adapt the unique characteristies of each
Table 2: Quantitative comparisons with existing works.
Resolution
Model
FIDI
KID.
IST
CLIP
Time (sec).
SDXL Podell et al. [2023]
111.452
0.020
12.046
29.804
71
ScaleCratter He ct al. 2023
77.543
0.006
17.112
30.904
80
DemoFusion Du
2025
64.422
0.002
19.307
32.648
219
2048
HiDiffussonn Zhang el al. 2023
74.773
0.0051
17.572
31.250
FouriScale Huang cl al. 2024
75.378
0.007
17.987
31.028
162
AccDiffusion Lin al. Pixclsmith (Ours)
2024
65.728
0.004
19.776
31.789
231
63.116
0.002
24.242
33,429
130
SDXL Podell et al.
2023
195.117
0.069
7.709
24.565
515
ScaleCralter He el
202.
105.132
0.018
13.542
27.767
1257
DemoFusion Du et al. 2023
66.186
0.003
18,940
32.319
1632
4096²
HiDiffusion Zhang
2023
97.614
0.015
13,681
27.708
255
FouriScale Huang et al. 2024
125.390
0.028
11,837
26.802
AccDiffusion lin ct al. Pixelsmith (Ours)
2024
67.084
0.003
19.323
32.010
1710
63.686
0.002
19.741
32.369
549
available FouriScale 4096 resolution
memory
RTX
Discussion and Considerations
Pixelsmith can generate images arbitrarily high-resolutions, but the resolution increases, adding more generative details without introducing artifacts becomes difficult due to the smaller denoising 4096" latent space
patch size relative the latent space. For example, 32768 image maps with a 128" denoising patch. The higher the resolution to generate in a single step, the higher the Slider value is required suppress artifacts. However, increasing the Slider value reduces the finer detail generation and return an image which closely resembles the previous resolution, limiting the true quality the higher-resolution image. Generating an image through a cascade generation approach allows for using a lower Slider value at each stage, providing more details and fewer artifacts compared to single-step generation, as the lower Slider settings can be more effective in this incremental process. Additionally, appropriate metrics for evaluating high-resolution images are lacking, and further research is necessary to improve quantitative evaluation methods, as highlighted in previous studies Jin et al. (2024] as well.
7 Conclusions
In this work, we introduced Pixelsmith, text-to- tinlto--mmageeeeee generative framework that leverages a pre- trained foundation model to generate higher-resolution images. Our flexible method uses a base image generated at the lowest resolution, combined with the Slider mechanism, to ensure both structural coherence and fine detail enhancement in the resulting images. By employing patch-based denoising, we have significantly reduced memory demands, making the generation of ultra-high- resolution images feasible on consumer-grade GPUs. Experimental results demonstrate that our model outperforms current state-of-the the-art methods in both image quality and generation efficiency.
10

---

8 Acknowledgements
D.F. A.T. acknowledge support from Royal Academy Engineering through the Chairs in Emerging Technologies scheme. D.F., R.M-S., C.K. acknowledge funding from QuantIC Project funded by EPSRC Quantum Technology Programme (grant EP/MO1326X/1, EP/T00097X/I), and Google. R.M-S, C.K. acknowledge funding from EP/R018634/1, EP/T02 1020/1, and EP/Y029178/1. For the purpose of open access, the author(s) has applied a Creative Commons Attribution (CC BY) license to any Accepted Manuscript version arising.
References
Marco Aversa, Gabriel Nobis, Miriam Hägele, Kai Standvoss, Mihaela Chirica, Roderick Murray- Smith, Ahmed M. Alaa, Lukas Ruff. Danicla Ivanova, Wojciech Samek, Frederick Klauschen, Bruno Sanguinetti, and Luis Oala. DiffInfinite: Large mask-image synthesis via parallel random patch diffusion in histopathology. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 781 18126-788 78141. Curran Associates, Inc., 2024.
Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. MultiDiffusion: Fusing diffusion paths for controlled image generation. In Proc. 40th International Conference on Machine Learning. ICML'23. JMLR.org, 2023.
Mikolaj Binkowski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifinng MMD GANs. arXiv preprint arXiv: 01440, 2018.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. InstructPix2Pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 15392-18402, 2023.
Pascal Chang, Jingwei Tang, Markus Gross, and Vinicius C. Azevedo. How Warped Your Noise: noise prior for diffusion models. In The Twelfth International Conference on Learning Representations, 2024. URL httttpp:://ppreereteee
Emily L Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using Laplacian pyramid of adversarial networks. Advances in neural information processing systems, 28, 2015.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Zheng Ding, Menggi Zhang, Jiajun Wu, and Zhuowen Tu. Patched denoising diffusion mod- els for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2023.
Ruoyi Du, Dongliang Chang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. DemoFusion: Democratising high-resolution image generation with no $$$. arXiv preprint 1rX1v:2311.16973, 2023.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.033: 033006 2024.
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M Susskind, and Navdeep Jaitly. Matryoshka diffusion models. The Twelfth International Conference on Learning Representations, 2023.
Lanqing Guo, Yingqing He, Haoxin Chen, Menghan Xia, Xiaodong Cun, Yufei Wang, Siyu Huang, Yong Zhang, Xintao Wang, Qifeng Chen, et Make cheap scaling: self-cascade diffusion model for higher-resolution adaptation. arXiv preprint arXiv:2402.10491 2024.
Moayed Haji-Ali, Guha Balakrishnan, and Vicente Ordonez. FlasticDiiflssin: Training-free arbitrary size image generation. arXiv preprint arXiv:23 1 18822, 2023,

---

Yingqing He, Shaoshu Yang, Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang. Ran He, Qifeng Chen, and Ying Shan. ScaleCrafter: Tuning-free -ree higher-resolution visual genera- tion with diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local Nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47): 2022.
Linjiang Huang, Rongyao Fang, Aiping Zhang, Guanglu Song, Si Liu, Yu Liu, and Hongsheng Li. Fouriscale: A frequency perspective on training-free high-resolution image synthesis. arXiv preprint arXiv:2403. 12963, 2024.
Juno Hwang, Yong-Hyun Park, and Junghyo Jo. Upsample guidance: Scale up diffusion models without training. arXiv preprint arXiv:2404.01709, 2024.
Zhiyu Jin, Xuli Shen, Bin Li, and Xiangyang Xue. Training-free diffusion model adaptation for variable-sized text-to-image synthesis. Advances in Neural Information Processing Systems, 36, 2024.
Comelius Lanezos. An iteration method for the solution of the eigenvalue problem of linear differen- tial and integral operators. 1950.
Yuseung Lee, Kunho Kim, Hyunjin Kim, and Minhyuk Sung. SyncDiffusion: Coherent montage via synchronized joint diffusions. Advances in Neural Information Processing Systems, 36:50648- 50660, 2023.
Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Ming Yu Liu, Kai Li, and Song Han. DistriFusion: Distributed parallel inference for high-resolution diffusion models. arviv preprint arXiv:2402,1948 2024.
Zhihang Lin, Mingbao Lin, Meng Zhao, and Rongrong Ji. AcDDiffusion An accurate method for higher-resolution image generation. arXiv preprint arXiv:2407. 10733 2024.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-solver: fast ODE solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5783 2022.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461-11471, 2022.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. arXiv preprint arXiv: wtxXiv:181..00 1812.01608, 2018.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1938-6047, 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint
2021.
Will Depue/ OpenAI. What's new with d 32, Nov 2023. URL https: httpss:/cookbook.. openai com/articles /watiitttlsssseeeeeeeeeea what_ is new with dalle_3
12

---

Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages I 11110111420 11400, 2022.
Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Mare Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. In The Twelfth International Conference on Learning Representations, 2024.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 7748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEER/VVF conference on computer vision and pattern recognition, pages 10684-10695, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEER/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022b.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances neural information processing systems, 35:36479-36494 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. Advances in neural information processing systems, 29, 2016.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5B: open large-scale dataset for training next generation image-text models. Advances Neural Information Processing Systems, 35:25278-25294 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference machine learning. pages 1256-2263 PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Jiayan Teng, Wendi Zheng, Ming Ding, Wenyi Hong, Jianqiao Wangni, Zhuoyi Yang, and Jie Tang. Relay diffusion: Unifying diffusion process across resolutions for image synthesis. arXiv preprint arXiv:2309.033500 2023.
Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. DiffFit: Unlocking transferability of large diffusion models via simple parameter-efficent fine- tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4230-4239 2023.
13

---

Zhuoyi Yang, Heyang Jiang, Wenyi Hong, Jiayan Teng, Wendi Zheng, Yuxiao Dong, Ming Ding, and Jie Tang. Inf-DiT: Upsampling any- any-resolution image with memory-efficient diffusion transformer. arXiv preprint XXiv:2405.04312 2024.
Shen Zhang, Zhaowei Chen, Zhenyu Zhao, Zhenyuan Chen, Yao Tang, Yuhao Chen, Wengang Cao. and Jiajun Liang. HiDiffusion: Unlocking high-resolution creativity and efficiency in low-resolution trained diffusion models. arXiv preprint arXiv:2311.17522 2023.
Huangjie Zheng, Zhendong Wang, Jianbo Yuan, Guanghan Ning, Pengcheng He, Quanzeng You, Hongxia Yang, and Mingyuan Zhou. Learning stackable and skippable lego bricks for efficient, re- configurable, and variable-resofution diffusion modeling. In The Twelfth International Conference on Learning Representations, 2023.
Qingping Zheng. Yuanfan Guo, Jiankang Deng. Jianhua Han, Ying Li, Songcen Xu, and Hang Xu. Any-size-dee size-diffusion: Toward efficient text-driven synthesis for any-size HD images. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7571-7578 2024a.
Wendi Zheng, Jiayan Teng. Zhuoyi Yang. Weihan Wang, Jidong Chen, Xiaotao Gu, Yuxiao Dong, Ming Ding, and Jie Tang. Cog View3: Finer and faster text-to-image generation via relay diffusion. arXiv preprint arXiv:2403.0512 001000000511..1 2024b.
14

---

A Improving the base generation
We show the qualitative improvements Pixelsmith provides to the base model through an example here. We observe that Pixelsmith improves the generation at higher image generation resolutions. In Figure O each iteration of the image generation. The final image resembles the actress more in terms of facial attributes and has more finer details in terms of multiple physical characteristics.
we can see that relatively small and larger areas in the image are also improved after
7
(28 048)
#16 4093
Figure 6: We show an example of a two-step image generation process here. The base model provides a 10242 image which our method then uses to iteratively generate a 2048² and a 40962 image. It evident from this example that the higher-resolutions overall have increasingly improved facial characteristics, demonstrating the effectiveness of our framework (zoom in to see in better detail).
15

---

B
Additional comparisons
ScaleCrafter
DemaFusion
FouriScale
AccDiffusion
HiDiffusion
Pixelsmith (Ours)
Figure 7: We show additional qualitative comparisons against our competitors here. All generated images have resolution of 40962. Other models suffer from artifacts (red arrows) along with highly prominent high frequency noise (purple arrows), some of which are indicated by arrows (zoom in to in better detail).
16

---

C Averaging Overlapping Patches
Patch denoising may introduce artifacts seen in Figure 8 lapping patches from the latent space, which can result in parts of the same area being denoised differently across patches. We address this issue by averaging along the edges where the patches meet.
This occurs because we sample over-
base model 10242
without averaging
with averaging 15362
15362
Figure 8: An illustration of patch artifact. The left image with resolution 1024² is generated by the base model. We generate the middle and right images resolution of 1536². In the middle image an artifact is visible due to overlapping patch edges. By averaging the overlapping patches, such artifacts removed, seen in the image the right (zoom in to see in better detail).
Figure 9 demonstrates our approach visually. Here, the image on the left represents the first noised latents. During the first denoising step (the next image), we sample random patch (shown in blue) for the denoising process. After every denoising step, our algorithm checks for overlap between patches to perform averaging to remove any potential artifacts that may occur. The second denoising step (the third image), demonstrates a random patch sampled (in green) which overlaps with the patch in the previous denoising step. The third image shows an area of overlap where artifacts can be potentially generated. To alleviate this issue, we then take a tolerance of 10 pixels in the overlap but within the bounds of the first (blue) patch, and take average across that region (as shown in yellow the final image in the figure). From our experiments, we note that averaging across the edges of the patches, forces that region create smooth structure instead of horizontal and vertical lines (as shown in the second image in Figure
Figure 9: An example of overlapped patches. The left image shows the noised latents. The second image is the first denoising step, and the third is the second denoising step. The third image shows the area where the patch artifacts may occur. We overcome this by averaging the area across the line (as shown in yellow in the last image) in order to make the transition between patches smoother.
In order to prevent degrading the quality of the final generated image, we do not include pixels exclusively from the patch of the second denoising step (pixels that are immediately adjacent to the edge of the random patch from the first denoising step and fall within the second patch) as they primarily contain noise and this degrades the quality of the entire image.
17

---

In Figure 8 for illustration purposes, we keep the higher-resolution at 1536". The Slider here is at position as we want to encourage the algorithm to create artifacts that are easily distinguishable, as stronger guidance results in them being less noticeable. We only use three inference steps here to keep the comparison as close to the base resolution as possible.
D Multi-Scale Duplications
Existing patch diffusion approaches risk introducing artifacts with each denoising patch. This phenomena is more prominent at higher image resolutions as the increase in the image resolution leads to increase in the size of the latent space while the size of the denoising patch still remains constant. Duplications are especially prominent in approaches like Du et al.) [023) where a fixed number of intermediate resolution steps may amplify this issue progressively with every step. We demonstrate this in Figure TO Although we appreciate that using many steps helps refine the image gradually, depending on the generation the steps may have the negative effect of progressively amplifying generated artifact at a lower resolution. For this reason we created Pixelsmith to be flexible in its image generation capabilities. Depending on the required result and text-prompt, we allow generating a Image generation intermediate steps (see also Appendix i treated as hyperparameter in our framework.
image without generating any intermediate resolutions first.
1024
20482
3072
DemoFusion
Ours
Figure 10: Comparison between a state-of-the-art method, DemoFusion Du et al. 2023 and our work, Pixelsmith. This figure shows how artifacts amplified at higher-resolutions in DemoFusion (as denoted by the red arrows). We directly generate a 30722 image from the base without any artifacts. The flexibility of our approach allows for any resolution to be generated right after the base resolution minimizing seeing artifacts at every intermediate step (zoom in to see in better detail).
18

---

E
Multi-Step Generation
1360x768
5440x3072
3400x1920
5440x3072
Figure 11: A comparison between two-step and one-step generations. On the top, we show that the base model first generates the image following which we generate the final resolution we desire. At the bottom, we use an intermediate step for better local image feature refinement. From the amplified region of the generation where we employ the intermediate generation step, we see that all the steps contribute to enhancing the features in the image to create a sharper image at the higher-resolution (zoom in see in better detail).
In Pixelsmith, generating the required resolution directly after the base model helps with removing multi-scale duplications of unwanted artifacts (Appendix D). However, allowing to generate multiple intermediate higher-resolution images before the required resolution also leads to a much better definition of objects present in the image at the higher-resolution. We demonstrate this with an example, as shown Figure [T] We generate a 5440 x 3072 image from base image resolution 1360 x 768 directly, as well as going through the intermediate step of generating 3400 x 1920 resolution image before the final resolution. We amplify the same region in all generated images to show the ad advantage of having an intermediate generation. Both 5440 x 3072 images are highly
19

---

detailed, but comparing the amplified region (zoom in the image to see in more detail), we can clearly see the high frequency components in the image, such as the grass, are lot more well defined even though they appear pixelated regions in the base image. There can however, be cases where this difference details in the images are not needed/noticeable and one step image generation would be the preferable and faster choice in such case.
Super-Resolution Comparison
base model In addition comparing performance with current state-of-the models for higher-resolution generation, we also compare our results with diffusion super-resolution model. Following Scale- Crafter by [e et al. generates images at 1024 resolution, choosing the 4x upscaler leads to creating a lot of compu- tational overhead. Hence, choose the 2x upscaler|Rombach RRombach et al.] (2022a] and evaluate the 2048² image resolution. Table shows that even though the super-resolution model is trained to upscale, still outperform it across all metrics.
2023] compare results with latent upscaler, but
Table 3: Quantitative comparisons with the 2x upscaler trained for the task of super-resolution. The final image resolution for both approaches is 2048 Pixelsmith outperforms the super-resolution model even in zero-shot setting.
Res.
Metric
2x upscaler
Ours
FIDL
63.913
63.116
KID,
0.002
0.002
2048
IS1 CLIP
19.603
24.242
32.250
33.429
Super-Resolution
Ours
Figure 12: Qualitative comparison with the latent upscaler. The fine details generated by framework are richer comparison to the super-resolution model.
20

---

G
Slider Position
generated image depends on many factors, namely, the input text-prompt, the input negative text-prompt, the seed, and the number of inference steps to name few. The generated images can differ across these factors and even across intermediate image resolutions in terms of shape, texture, colour and a plethora of other varying attributes. This is especially seen when using a model that was not trained on the data for the task at hand directly making the task a lot more challenging. Varying sizes of objects in the image such as arbitrary big structures, differ from the smaller ones and miscellaneous textured surfaces in the images are different from smooth ones. To account for these changes, and to control and constraint image generation at higher-resolutions, we introduce The Slider. The Slider helps provide a varying amount of control to user to create image generations depending on how much they want the previously generated scene to influence the next generation.
In real world scenario, this is especially useful as depending on the type of image being generated and the prominence of artifacts that could potentially be introduced in the image, the user can switch between allowing more guidance from the previous resolution, whereas for truly unconstrained image generation, the Slider provides the added benefit of more freedom. We demonstrate the expressive power of The Slider in Figure
We see more bees generated in the image on the left when we position the Slider value at 1. This results in certain artifacts and changes the structure of the image. At the other extreme, setting the max value of the Slider (49) results in preserving the overall structure of the base image, while not introducing any novel aspects into the generated image. Setting the Slider 27 allows the framework to trade-off between image generation and preserving the global structure where it generates more realistic look high frequency components like the hair on the bees' body. Depending on the different settings for the Slider we can constraint the image generation to be more free.
We observe from detail but at the same time, are prone to changing the global structure of the image while creating duplications and artifacts. Higher values tend keep the overall global structure of the image intact, but do not refine the image to create finer details. The Slider is a tunable hyperameter that provides means trade-off between freedom and constraints create the best possible generation of an image from the text-prompt.
experiments that lower values tend generate images that are rich in fine
2304x4080
-27
449
Figure 13: Experimenting with different positions of The Slider. The image on the left contains finer details loses the overall structure, generating a artifacts. The image on the right has a good global structure but lacks any high frequency details due to the Slider constraining the image generation process. The central position, in this case, is the most optimal generation where the overall structure of the image is preserved and the same time, finer details are also introduced such as the individual hair strands on the bees' body. We observed that by generating multiple resolutions in a cascaded framework, each intermediate generation will contribute to creating more detailed final image.
21

---

H
Patch Denoising Masks
Figure 14: Mask visualisation. Half of the pixels in the denoised patch are masked and replaced with values from the image guidance, significantly reducing duplications.
I Additional visualizations
(68
20x1536
40x3072
Figure 15: Example of 16x (5440 3072) images. The amplified cut-outs illustrate how facial features become increasingly well-defined with each generation (zoom in to view in greater detail).
22

---

11022x1024
x16 14296 40966
x1 11024x1024)
x16 (4996x4096)
(788-1986)
11024x 1024)
x266 163344 x 16384)
Figure 16: Examples of images at base resolutions alongside their nigher-resolution counterparts generated using Pixelsmith (zoom to see in better detail).
23

---

X953 (31616*31616)
8888x88
888xx988
088
1988x988
0000 988
1988x988
988x 988
1988*88
Figure 17: An example of a 953x (31616 x 31616) gigapixel image. Each displayed patch is 988 x 988, roughly the size of the base image. The image closely resembles those captured by the Hubble telescope. (zoom in to see in better detail).
24

---

Text-prompts
Figure
lego astronaut, riding a lego horse on Mars
pearly gates of heaven, ethereal, intricate details, angels, cloud, 7th sky, eternal bliss, in an otherworldly universe, detailed. realistic, 8k uhd, high quality
photo realistic, ultra details, natural light ultra detailed portrait of female necromancer, skeleton face volumetric fog, Hyperrealism, breathtaking, ultra realistic, ultra detailed, cyber background, cinematic lighting, highly detailed, breathtaking, photography, stunning environment, wide-angle
Landscape of a dark far away planet, rock and organic soil, glowing trees, UHD, masterpiece, trending artstation, sharp focus, studio photo, intricate details, highly detailed, by Greg Rutkowski
photo the dark sky with visible but very distant swirling galaxies and stars
Figure
cat, close-up
Figure
A village on cliff, realistic transparent sea, fishing boats
Figure
Photo of zebra in forest, black and white
Photo realistic oil painting of Robert De Niro
Figure
photo of Emma Watson, bike
Figure
Darth Maul in a white tuxedo, in forest, photo realistic
Raw photo of warrior looking the camera, close-up portrait, high detail, master- piece
ultimate Japanese style, ink wash. Black and White, japanese Writing in Red and Blue, trending ArtStation, intricate details, masterpiece, best quality, Use Dream Diffusion Secret Prompt, Epic
Figure
Close up photo of the human eye iris
Figure TO
Astronaut on Mars
Figure
masterpiece, best quality, high quality, extremely detailed CG unity 8k wallpaper, scenery, outdoors, sky, cloud, day, no humans, mountain, landscape, water, tree, blue sky, water- fall, cliff, nature, lake, river, cloudy sky, award winning photography, Bokeh, Depth of Field, HDR, bloom. Chromatic Aberration, Photorealistic, extremely detailed, trending on artstation, trending on CGsociety, Intricate, High Detail, dramatic, art by midjourney
Figure B3
25

---

close-up up photo of a bee, macro, 100mm lens, bokeh, depth of field, high quality
Figure 22
samurai iron man, dark black samurai armor, Artgerm, photorealism, Unreal engine 5 highly rendered, glowing red eyes, Tilt-shift 4k, White Balance, High Contrast, Low Saturation, Bracketing Detailed.
macro photography of a sakura blossom flower, epic
photo of cute fluffy cat, close-up, high detail
Figure 13
turian on andromeda, 50mm lens
Figure
portrait photo of beautiful Greek girl, with black pigtails, wearing fluffy white coat, wide mid shot, derelict building, soft lighting, Hasselblad, Voigtländer 50mm lens, in style of Oleg Oprisco
photography in the style of detailed hyperrealism ,cinematic composition, dramatic light, glass with forest in it, surrealism 8k, graphic of enchanted terrarium, 4k highly detailed digital art, 3D render digital art, 3D render stylized, stylized 3D render, highly detailed scene, 3D landscape, 3D landscape, der, surreal 3D render. Dreamlike, mysterious, provocative, symbolic, intricate, detailed, expressivc.hyper
surrealism, sci-fiish landscape, artistic ren-
The gladiator stands after the battle, fear and horror his face, tired and beaten, sand on his face mixed with sweat, an atmosphere of darkness and horror, hyper realistic photo. In post-production, enhance the details, sharpness, and contrast.
aerial photo, amazon forest
mosaic of van Gogh
Figure
photo of the dark sky with visible but very distant swirling galaxies and stars
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All claims in the paper are studied, there are explanations for all of them.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: see Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
There is no theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In Section 5 we detail how we did the experiments. All images can easily generated and given resources the metrics as well.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Yes the code will be open open-sourced and the data to evaluate is already open- source.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis... cc/
public/guides ottp://s/eoooeeeebeessssoooooon.
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https
wetgguddddeeeuuubbueeee ssinoolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training is irrelevant we did not train any model. Details about evaluating are provided (see Section 55.. We cannot share the subset to evaluate as we not hold the rights but an official download link will be provided. Code will be provided as well.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow previous works in the field. We use relevant metrics and rely on qualitative comparisons.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: the used GPU is mentioned. Time of inference is also provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://wwwwg...ccoaaaaa/ccccccca
Answer: [Yes]
Justification: We use a pre-trained diffusion model which should used as every other generative model, with care. We do not fine-tune or change it in our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our work does not change the societal impact of diffusion models as the generation remains the same as the pre- are-trained model.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: We use a pre-trained model without altering the weights.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Everything is cited and the pre-trained model is free to be used for research.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: A pre- trainerind model is used, not realeasing it for the first time
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We use an open sourced pre-trained model
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We an open sourced pre-trained model
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "2C9d8vaj2k", "reviewer": "Reviewer_SvAB", "summary": "This paper introduces a framework for generating high-resolution images from text prompts using pre-trained diffusion models. The key innovations are: A cascading approach that uses lower-resolution generated images as guidance for higher resolutions. A \"Slider\" mechanism to control the balance between following the guidance and allowing novel generation. Patch-based denoising to enable generation of arbitrarily large images on a single GPU. Averaging techniques to reduce artifacts from patch-based generation. The authors demonstrate that Pixelsmith can generate images up to 32,768 x 32,768 pixels on a single GPU, outperforming existing methods in terms of quality and efficiency.", "questions": "See Weaknesses please.", "limitations": "The authors claim they discussed the limitation but I didn't find it in the text. Correct me if I am wrong.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The proposed method enables generation of ultra-high resolution images without additional training, addressing an important limitation of current models.\n\nGood results.\n\nThe patch-based approach allows for generating massive images on consumer GPUs, which is a significant practical advantage.", "weaknesses": "1. Poor presentation. The writing of this paper needs substantial improvement to be published in a venue like NeurIPS. First, the language of the paper needs to be improved. Second, the paper spends a lot of space on unnecessary content. The method in this paper is not difficult, but it is very difficult to understand the method in one reading. In terms of writing, the authors did not succeed in emphasizing the core concept of their proposed method. In addition, the length of the paper is too long. Such a simple method, but the experiments are not introduced until page 8, which should not happen.\n2. The core idea of ​​this paper seems to be to generate once and then upsample the generated image by patch-wise processing. (By the way, Figure 3 is difficult to understand and Eq. 4 is not explained). This method is essentially not much different from other operations that use diffusion for upsampling. So what is the advantage of this method? Why is this method feasible?\n3. The role of FFT Transformation is not well demonstrated. No exploratory experiments are used to demonstrate the motivation and effect of this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive criticism. We welcome any comments that can help improve the quality of our work.\n\n>W1. Poor presentation. The writing of this paper needs substantial improvement to be published in a venue like NeurIPS. First, the language of the paper needs to be improved. Second, the paper spends a lot of space on unnecessary content. The method in this paper is not difficult, but it is very difficult to understand the method in one reading. In terms of writing, the authors did not succeed in emphasizing the core concept of their proposed method. In addition, the length of the paper is too long. Such a simple method, but the experiments are not introduced until page 8, which should not happen.\n\nWe carefully review and change the language to make it easier for readers to follow. The method section has been refined to highlight the core concept effectively. Additionally, we have edited the method figure to enhance the clarity. See 'global rebuttal' and the pdf with the Figure. Could you kindly provide specific examples where the language was not up to the NeurIPS level as it will help us better revise our paper?\nThe \"Related work\" and \"Foundations\" are shortened so that the \"Experiments\" can be introduced at page 7. Given the nature of the paper images take a lot of space (for example Figure 1 takes a full page) and as a result sections are pushed back. \n\n>W2. The core idea of this paper seems to be to generate once and then upsample the generated image by patch-wise processing. (By the way, Figure 3 is difficult to understand and Eq. 4 is not explained). This method is essentially not much different from other operations that use diffusion for upsampling. So what is the advantage of this method? Why is this method feasible?\n\nWe have revised Figure 3 to make it easier for readers to understand. As mentioned in the global response, the revised \"Method\" section and updated Figure 3 are now better aligned, ensuring improved coherence.\nOur method is fundamentally different from diffusion models used for upsampling. While diffusion models trained specifically for super-resolution generate high-resolution images from low-resolution inputs, our approach generates high-resolution images directly from text prompts. Diffusion models for upsampling primarily focus on the input image and largely ignore the text. In contrast, our method relies heavily on the text prompt, which drives the changes. As the patches denoise different areas of the latent space, the content of the text prompt becomes apparent.\nWithout our proposed method, numerous duplications would appear across the final image. Our framework restricts these duplications, and depending on the position of the Slider, some or a lot of new information will appear in the higher resolution that is not present in the lower resolution. For example, in Figure 1 (top right), the low-resolution cut-out box for the necromancer woman resembles a metallic connection without a distinct shape. Our method's patch denoising, considering the text prompt, reveals a skull, which makes sense for a necromancer. The patches attempt to denoise based on the text prompt but are constrained by the proposed key components to avoid, for example, denoising multiple necromancers all over the image. Current methods attempting similar results often end up with artifacts, which is not the case with Pixelsmith, as the Slider’s position can eliminate them.\nIn conclusion, diffusion models for upsampling lack this generative freedom. The downside of current works is the introduction of duplications and strange artifacts, but Pixelsmith provides control (via the Slider’s position) to avoid them.\n\nThis explanation will be reflected in the final version.\n\n>W3. The role of FFT Transformation is not well demonstrated. No exploratory experiments are used to demonstrate the motivation and effect of this method.\n\nThe FFT transformation is used to fuse information between the guidance latent space and the current (higher resolution) latent space. The decision to use the imaginary part is based on empirical results. This is now reflected in our paper as mentioned in the global response. The effect is demonstrated in Table 1.\n\nIn Table 1:\n- The column labeled **\"Re\"** represents the case where we averaged the real parts of both latents and used the imaginary part of the current latent.\n- The column labeled **\"Re&Im\"** indicates that we averaged both the real and imaginary parts of the two latents.\n- The final column shows the proposed method, where we averaged the imaginary parts of the two latents and used the real part of the current latent.\n\nAs shown, the proposed method in the last column performs better.\n\n\n\n>L. The authors claim they discussed the limitation but I didn't find it in the text. Correct me if I am wrong.\n\nPlease refer to \"6 Discussion and Considerations.\" One of the main limitations in generating higher-resolution images is the availability of effective metrics. More research is needed to develop methods that can accurately evaluate images without reducing their resolution to much smaller sizes, which results in the loss of detail. Additionally, there is no penalty for artifacts and duplications commonly found in many recent papers (see Figure 4 and Figure 6).\n\nIn the same section, we discuss Pixelsmith and the trade-off between preserving fine detail and suppressing artifacts. As noted, achieving true higher-resolution detail becomes increasingly challenging as the resolution increases, making it difficult to eliminate artifacts completely.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FrkRBMtFti", "reviewer": "Reviewer_wkBK", "summary": "The paper introduces Pixelsmith, a framework designed to utilize pre-trained diffusion models to enable high-resolution image generation using only a single GPU.\nPatch-based denoising ensures that the entire generation process can be accommodated on a single GPU.\nThe Slider mechanism balances the trade-off between finer details and overall structure by controlling the transition from guided generation to native generation.\nGuided generation operate on the latent fused by one upscaled higher-res image patch and one native-generated higher-res image patch in Fourier space, helping to maintain global structures as claimed by the authors.\nExperimental results show that Pixelsmith not only produces high-quality and diverse images but also improves sampling time and reduces artifacts compared to existing techniques.", "questions": "The aim of achieving high-resolution image generation on a single GPU is commendable, and the proposed method successfully accomplishes this goal. However, the presentation quality is relatively poor, especially with some key parts described incorrectly. I hope the authors can address the questions mentioned in the weaknesses section and resolve my concerns.", "limitations": "The authors have discussed the trade-off between achieving finer details and suppressing artifacts. They have also suggested proposing appropriate metrics for evaluating high-resolution image generation, which would be beneficial for the community.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. Memory Efficiency with Limited Computational Resources: Achieving high-resolution image generation on a single GPU is a good contribution, and the image generation quality is not compromised by this restricted setting.\n2. High-Quality Generation: The proposed Fourier space fusion maintains fine-grained details and global structures, while also preventing some artifacts that occur in other methods.", "weaknesses": "1.\tPaper Structure Issues: The content before the methods section is too lengthy, making the key method and experiments sections comparatively short and harder to fully understand. These are elaborated upon in the following points.\n2.\tMethod Description is Hard to Follow: For instance, the overview in Lines #171-173 does not fully describe all the components involved in the method, requiring readers to review the paper multiple times to understand the entire pipeline. Additionally, the intuition behind combining $\\hat{z}^{iFFT}{t-1}$ and $\\hat{z}^{guid}{t-1}$ is not explained, which is confusing since $\\hat{z}^{iFFT}_{t-1}$ already incorporates information from $\\hat{z}^{guid}_t$.\n3.\tErrors and Mismatches in the Experiment Section: According to Table 1, using the real part is better, which contradicts Line #248. Line #252 describes the wrong column for \"the next inference step\". Also, the term “base model” in Table 2 is unclear—does it refer to the same model as Table 2’s Pixelsmith and Figure 1’s base model?\n4.\tFlaws in Figure Illustrations: In Figure 1, the base model image for “x256 (16384x16384)” is not displayed. Is this base model the same as the one mentioned in Table 1? In Figure 3, the illustration should be more concrete and easier to understand; however, some variable names are too small and there are too many blank areas. The fusion part is also unclear, with no mention of the real part of the Fourier transform and no explicit explanation of $\\mu( . , . )$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive criticism. We welcome any comments that can help improve the quality of our work.\n\n>W1. (full question)\n\nWe have restructured the content based on the suggestions. The Related Work and Foundations sections have been shortened. The Method section has been revised, as detailed in the global response. Additionally, the Experiments section has been expanded to enhance comprehension as follows:\n\n> ### Ablation study\n> We conduct a qualitative examination of the framework on $2048^2$ image resolution. Specifically, we assess the effects of the Slider position, the importance of the imaginary part, the significance of incorporating information from the next inference step using masking and the impact of averaging overlapping patches.\n> \n> **Slider Position (columns SP0, SP24, SP49):** Our findings indicate that the Slider position significantly influences the results. The proposed model with a Slider position of 30 (Table 1, column Proposed) outperforms positions 0, 24, and 49 (Table 1, columns SP0, SP24, and SP49). A position of 0 introduces numerous artifacts, while a position of 49 lacks fine detail. Position 24 is close to the proposed but not optimal for the random subset. Position 30 was chosen randomly and is not cherry-picked. Appendix F demonstrates the effect of the position with a qualitative example.\n> \n> **Imaginary Part (columns Re, Re\\&Im):** The proposed model (Table 1, column Proposed) averages the imaginary parts of the guidance latents and the current latents, then uses the real part of the current latents as described in the Method section (see Figure 3). This setup is chosen based on experimental results demonstrated here. We compare this with averaging the real parts of the two latent spaces and using the imaginary part of the current latents to invert back to the pixel space (Table 1, column Re), as well as averaging both the imaginary and real parts from the two latent spaces (Table 1, column Re&Im).\n> \n> **Masking (column No Mask):** Analysis in column No Mask shows that not using the chess-like mask yields better scores on two metrics. However, as shown in Appendix G, incorporating the mask is crucial for removing artifacts, so we continue to use it despite the metrics.\n> \n> **Averaging (column No Aver.):** Finally, we show that using patch averaging (Table 1, column Proposed) improves scores compared to not using it (Table 1, column No Aver.). This is because patch averaging eliminates patch artifacts, as seen in Appendix C.\n>\n> Table1: A quantitative examination of our framework through ablations.\n>\n>| Metric        | SP0    | SP24   | SP49   | Re     | Re&Im  | No Mask | No Aver. | Proposed |\n>|---------------|--------|--------|--------|--------|--------|--------|----------|----------|\n>(unchanged)\n\n\n>W2. (full question)\n\nWe have revised the Method description to better align with Figure 3 (please see the global response). Additionally, we have updated Figure 3 (please see the attached PDF) to enhance readability and coherence. Regarding the second part of your concern, please refer to Appendix G for a qualitative comparison and the Experiments section for a quantitative comparison. Every time a denoising step finishes, there is a chance for artifacts to appear. The chess-like mask is applied after the denoising, replacing some information with the guidance where no artifacts exist.\n\n\n>W3. Errors and Mismatches in the Experiment Section: According to Table 1, using the real part is better, which contradicts Line \\#248.\n\nIn Lines \\#247-248 we state that \"Using the imaginary part to average the guidance latents and the current latents is more effective than using the real part or...\". This statement is consistent with the results shown in Table 1, where all four metrics demonstrate better performance when using the imaginary part (last column) compared to using the real part (column Re).\n\n\n>W3. Line \\#252 describes the wrong column for \"the next inference step\".\n\nYou are correct, this was an error. Thank you for bringing this to our attention. We have now corrected it.\n\n>W3. Also, the term “base model” in Table 2 is unclear—does it refer to the same model as Table 2’s Pixelsmith and Figure 1’s base model?\n\nThroughout the paper we refer to SDXL as the base model (one of the references is in Line \\#258: \"SDXL, which serves as the base model\"). In the Experiments section, we also referred to our final proposed model as the base model, which caused confusion. This inconsistency has now been corrected by replacing the base model in the Experiments with proposed.\n\n>W4. Flaws in Figure Illustrations: In Figure 1, the base model image for “x256 (16384x16384)” is not displayed.\n\nThe higher resolution images are in scale with those generated by the base model. In the “x256 (16384x16384)” case, the base model image is not missing but hard to see due to the significant difference in resolution. As with the other images in Figure 1, the base model image is located at the bottom-right, overlaid on the higher resolution image. We understand this may confuse readers, so we have added a white box around all base model images to make them more distinguishable. The legend of Figure 1 has been expanded as follows: \"... Pixelsmith and the base model. The higher resolution images are in scale with the images generated by the base model. The base model generations are enclosed in a white frame. Some cut-outs ...\"\n\n>W4. Is this base model the same as the one mentioned in Table 1?\n\nPlease refer to our response to W3.\n\n>W4. In Figure 3, the illustration should be more concrete and easier to understand; however, some variable names are too small and there are too many blank areas. The fusion part is also unclear, with no mention of the real part of the Fourier transform and no explicit explanation of.\n\nWe have revised Figure 3 (please see the attached PDF). For a detailed explanation, please refer to the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "btlldGDAAf", "reviewer": "Reviewer_T7y4", "summary": "This paper proposes a training-free method for diffusion models to sample images at higher resolution with limited GPU memory, which introduces several tricks, such as fourier merging, chess-mask deduplication, and slider control. Experiments show the effectiveness of the proposed method.", "questions": "1. How to determine the patch orders in high-resolution generation, is it random orders or follow some principles.\n2. From Tab. 2, it is not clear why Pixelsmith slower than Scalecrafter at 2048 resolution, while nearly 2x faster at 4096 resolution. Is there any trend that the bigger, the more efficient?\n3. Is the content of the final scaled high resolution output significantly differ from the base output, whether the new objects will be generated?", "limitations": "The authors have adequately discussed the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The fourier merging seems interesting with imaginary part to maintain the global structure;\n2. The chess mask seems effectively eliminates the duplicate artifacts in high resulotion generation;\n3. The ablations are sufficient and comparisons show the superiority of the proposed method.", "weaknesses": "1. Is there any literatures to support that the imaginary part of the fourier transform corresponding to the low-frequency of the signal, or is there any deeper analysis beyond the ablation results in Tab.1 to illustrate this? By the way, it is suggested to provide the visual ablations of this part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive criticism. We welcome any comments that can help improve the quality of our work.\n\n>W1. Is there any literatures to support that the imaginary part of the fourier transform corresponding to the low-frequency of the signal, or is there any deeper analysis beyond the ablation results in Tab.1 to illustrate this?\n\nPlease refer to the global response.\n\n>W1. By the way, it is suggested to provide the visual ablations of this part.\n\nIncluding visual ablations is indeed a good suggestion. They will be part of the final version.\n\n>Q1. How to determine the patch orders in high-resolution generation, is it random orders or follow some principles.\n\nThe patch orders are random, as described in Lines 143 and 183. However, this randomness is controlled. We track which areas have been denoised (since each pixel is denoised only once per timestep) and avoid selecting those areas again. Therefore, while the selection of patches is random, it is constrained to areas that still require denoising. For example, if 90% of the latent space has already been denoised for timestep t, the coordinates for the next patches will be chosen from the remaining 10%.\n\nIf we selected coordinates purely at random without considering the denoised areas, the inference time would increase dramatically, rendering the process infeasible, especially for very high resolutions such as $32768^2$.\n\nThis explanation will be incorporated into the \"Patch Sampling\" subsection for added clarity.\n\n\n>Q2. From Tab. 2, it is not clear why Pixelsmith slower than Scalecrafter at 2048 resolution, while nearly 2x faster at 4096 resolution. Is there any trend that the bigger, the more efficient?\n\nWe extract patches of the same size, specifically $128²$, regardless of the resolution. Pixelsmith requires 130 seconds for $2048^2$ (4194304 pixels) and 549 seconds for $4096^2$ (16777216 pixels). This represents a 4x increase in pixels and a 4.22x increase in inference time. Pixelsmith’s inference time scales linearly with the number of pixels because the denoising UNet consistently uses patches of the same size. In contrast, Scalecrafter scales differently as it does not use patches. Unfortunatelly, Scalecrafter's official code provides scripts for specific resolutions and no further experiments can be conducted to examine how they scale at different resolutions and most importantly there is also the constraints of the required memory. Pixelsmith apart from not being constrained by memory is very flexible. It is posible to generate an image with resolution 4096x1024 using a $1024^2$ base image. The final ratio is not constrained by the base one. To accomplish this we only change the final resolution (ie just one number), no script or otherwise optimization is needed. \n\n>Q3. Is the content of the final scaled high resolution output significantly differ from the base output, whether the new objects will be generated?\n\nSeveral factors influence the comparison between the final image and the base image. Key factors include the Slider position, the number of intermediate steps, the resolution of the final image, and the specific text prompt used. The Slider is designed to offer more control over the generation process. For instance, Figure 11 demonstrates how different Slider positions produce varying results, with position 49 being closest to the base image and position 1 showing excessive repetitions. However, in certain contexts, such as generating the surface of a planet with craters, having more craters can be beneficial. Thus, the final content can vary significantly or minimally, and this can be easily adjusted using the Slider.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tW63Al6phH", "reviewer": "Reviewer_snt9", "summary": "In this paper the authors propose a number of engineering tricks which enable generating at higher resolutions from a pre-trained txt2img diffusion model.\nNotably, the requirements for the proposed method are relatively low.\n\nIn the proposed approach, an image of a standard resolution is generated first.\nAfter that, is it upsampled (in RGB space) to the desired size and used as a guidance for the new, truly high-resolution image which is generated as a sequence of overlapping patches.\nThe guidance mechanism is implemented by mixing the imaginary component of Fourier decomposition of the currently denoised and guidance patches.\nThis procedure takes place not for all the denoising steps, but only up to certain noise level referred to as Slider.\nThe value of Slider controls how similar are the original standard-resolution image and its higher-resolution version.\n\nPerformance of the method is evaluated with commonly used metrics such as FID, KID, IS and CLIP-similarity.\nAlso, a lot of samples are provided for visual inspection.", "questions": "I ask the authors to address the weaknesses listed above during the rebuttal period. In particular, I am interested in the deeper justification of  using imaginary Fourier coefficients than empirical evidence.\n\nPOST-REBUTTAL UPDATE: I keep my initial score. I think that although the method is very simple, this does not mean that the submission is bad. I believe that with improved presentation this paper can be interesting for the practitioners.", "limitations": "No actions needed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "According to the provided metrics and visual results, the methods performs quite well in comparison with recent baselines. \nThe fact that it requires only 7.4 GB of memory makes it very affordable for the community. \nThe idea of mixing in the spectral domain, to the best of my knowledge, is novel enough in context of ultra-high resolution sampling from a pre-trained model. \nExploiting the checkerboard pattern for mixing with guidance is also an interesting engineering trick.", "weaknesses": "1. While the authors several times mention that \"the imaginary part in the frequency space contains most of the low-frequency information of the image\" (line 187) and \"imaginary part provides more structural information than the real part\" (line 249), they do not provide any references for this statement or supporting experiments. To me, this fact does not look evident enough. I am sure this needs more thorough explanation since otherwise it looks just like an empirical trick.\n\n1. From the text, it is unclear which Slider value was used to obtain the numbers from Table 2. Was it the value of 30, as defined in line 244? Also, what exactly is \"using next inference step (NIS)\" (see line 253)? Is this about checkerboard mask mixing?\n\n1. The presentation of Fig. 3 is not easy for understanding, I advise considering its redesign. For example, forward diffusion and taking imaginary part of FFT are denoted in the same way. \n\n1. It is unclear from the text what the guidelines are for selecting the size for the zone of averaging for overlapping patches (line 805).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive criticism. We welcome any comments that can help improve the quality of our work.\n\n>W1. While the authors several times mention that \"the imaginary part in the frequency space contains most of the low-frequency information of the image\" (line 187) and \"imaginary part provides more structural information than the real part\" (line 249), they do not provide any references for this statement or supporting experiments. To me, this fact does not look evident enough. I am sure this needs more thorough explanation since otherwise it looks just like an empirical trick.\n\nPlease refer to the global response.\n\n>W2. From the text, it is unclear which Slider value was used to obtain the numbers from Table 2. Was it the value of 30, as defined in line 244?\n\nYes, the position of the Slider was 30. This is also indicated by the metrics in Table 1 and Table 2 which are identical. Nevertheless, we understand that it should be specifically mentioned in the Comparison section, as such Line 264 is expanded as follows \"... refer to Appendix B. The Slider position is set to 30.\".\nWe note here that the position is not cherry picked but chosen randomly. Different Slider positions will yield different results (as shown in Table 1), and there may be other positions that could potentially perform even better.\n\n>W1. Also, what exactly is \"using next inference step (NIS)\" (see line 253)? Is this about checkerboard mask mixing?\n\nYes, this is correct. The \"next inference step\" mentioned in the Ablation study refers to the use of the chess-like mask. As noted in line 253, \"... in Appendix G, incorporating the next inference ...\". Appendix G provides further explanation of this process. We recognize that this may have been confusing, so the issue has been addressed as follows:\n* The line 241 is changed to \"... information from the next inference step using masking and the impact ...\"\n* The paragraph starting with line 252 is changed to \" **Masking (column No Mask):** Analysis in column No Mask shows that not using the chess-like mask yields better scores on two metrics. However, as shown in Appendix G, incorporating the mask is crucial for removing artifacts, so we continue to use it despite the metrics.\" \nPlease refer to our reply to reveiwer wkBK for the fully revised Ablation study subsection.\n\n>W3. The presentation of Fig. 3 is not easy for understanding, I advise considering its redesign. For example, forward diffusion and taking imaginary part of FFT are denoted in the same way. \n\nWe have revised Figure 3 to enhance clarity. Please refer to the attached PDF for the updated version.\n\n>W4. It is unclear from the text what the guidelines are for selecting the size for the zone of averaging for overlapping patches (line 805).\n\nThis is a set number (not a hyperparameter) that we set to 10 and never change. From our experiments, the exact number is not critical, which is why line 805 refers to a \"few pixels.\" We have set this number to 10, and it remains fixed, as variations like 15 or 20 pixels do not produce noticeable differences. This can also be seen in the provided code (smoothed_time_mask = create_gradient_border(time_mask, gradient_width=10)). We understand that specifying the exact value is preferable, so line 804 has been updated to read: \"... a tolerance of 10 pixels in the overlap ...\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers (Rvs) for the valuable feedback and the opportunity to improve our work.

## Acknowledged Strengths

**Results:** All 4 Rvs agree that the paper achieves good results compared to current works. It's worth emphasizing that these works are published at the most prestigious conferences, underscoring the significant impact and competitiveness of our results

**Contribution:** Rv SvAB highlights the importance of our work, noting that it addresses an "important limitation of current models". This recognition emphasizes our paper's relevance in advancing the field and tackling key challenges. Further, Rvs SvAB, wkBK, and snt9 acknowledge the importance of our ability to use just 1 GPU regardless of the final resolution, identifying this as a significant contribution. We would like to add that our framework not only is memory efficient but the only one to scale a pre-trained diffusion model up to $32768^2$ ($\approx$1.1 gigapixel). Most current works show results up to $4096^2$ and a very few of them scale up to $8192^2$ which means that our model is able to generate $\times16$ more pixels than any current work

**Originality:**  Rv snt9 commends our method as both novel and interesting

**Ablations:**  Rv t7y4 finds that no further ablations are needed and that the introduced parts in the method are effective and interesting. We note that the rest of the Rvs also do not request extra experiments showing the thoroughness of our evaluation

## Key Concerns

**Fourier transformations:** Some Rvs have noted issues with lines 187 and 248 concerning the justification of using the imaginary part of the guidance latents and why we did not use the real instead. We observed that removing some of the data when fusing the two resolutions of the image helped to improve results and seemed to preserve low-frequency information well. The ablation studies compared Re+Im, Im only, and Re only. Im only performed best empirically. We thank the Rvs for highlighting this, as our observations were poorly formulated. We will change all references to reflect that it is based on empirical observations

**Presentation:**  Some Rvs were concerned about the presentation. We have revised the organization to enhance clarity. The Related Work and Foundations have been reduced and the Method has been rewritten (see below a preview due to character limitations). Additionally, Fig. 3 has been revised to align more closely with the Method, ensuring a seamless understanding (f.i. in Fig. 3 the "3. Image guidance" aligns with "Image guidance" in Overview)

>## Framework
>In this section, we describe the workflow of Pixelsmith, detailing how the framework adapts pre-trained text-to-image LDMs to generate images with higher resolutions on a single GPU (see Fig. 3). In order to generate ultra high resolution images without artifacts, we introduce these key components: the Slider (see Slider), patch averaging (see Patch averaging) and masking (see Masking).
>
>### Overview
>
>**Text-to-image generation:** First, given a conditional prompt $c$, ... (line 174-176) SDXL.
>
>**Upsampling process:** After the image generation, we apply an upsampling algorithm ... (lines 176-179)
>
>**Image guidance:** Once the guidance image is encoded in the VAE's latent space $z_0^{guid}=\mathcal{E_\theta}(x^{guid})$, we can easily sample each latent variable of the diffusion process through the forward diffusion process $z^{guid}_t \sim q(z^{guid}_t|z^{guid}_0)$. 
>
>**Image generation:** The generative process starts from $z_T \sim \mathcal{N}(0,I)$, which has the same dimensions as $z^{guid}_T$. At each step, a random patch is cropped as described in Section 3.2:
>
>(Eq. 4)
>
>where $\mathcal{C}^{i,j}$ crops the latent variables $z_t,z_t^{guid}$ at the coordinates $i,j$ for the patch sampling described in Section 3.2.
>
>The Slider’s position (see Slider), indicated by a blue line in Fig. 3, determines whether the guidance mechanism (see Guidance mechanism) or unguided patch denoising will be applied. In the unguided mode, each patch is based solely on the previous one, similar to a conventional patch denoising process. The Slider allows control over whether a generated image will be slightly or significantly altered compared to the previous resolution.
>After the denoising process has ended, the latents $z_0$ are decoded and the higher resolution image is generated. Using a cascade upsampling approach, this generated image can be upsampled again, repeating the process to achieve an even higher resolution image.
>
>### Cascade upsampling
>(unchanged)
>
>## Guidance mechanism
>The guidance mechanism fuses the $\hat{z}^{guid}_t$, $\hat{z}_t$ and $\hat{z}^{guid}\_{t-1}$ random patches to generate the $\hat{z}\_{t-1}$ patch. First, the $\hat{z}^{guid}_t$ and $\hat{z}_t$ patches are transformed to the Fourier space using a Fast Fourier Transformation ($\mathcal{FFT}$), where their imaginary parts $\mathcal{I}m$ are averaged:
>
>(Eq. 5)
>
>The imaginary part is then combined with the real part of the $\hat{z}_t$ patch to form $\hat{z}^{FFT}_t = \mathcal{R}e(\hat{z}_t) + i \hat{z}^{im}_t$	
 , which is then transformed back to the spatial domain:
>
>(Eq. 6) 
>
>The output is then used as the condition for the reverse diffusion process
>
>(Eq. 7)
>
>To prevent further prompt duplications across the entire latent, we use masking.
>
>### Masking
>
>We combine the sampled $\hat{z}^{iFFT}\_{t-1}$ with the image guidance $\hat{z}^{guid}\_{t-1}$ using a chess-like mask $\Lambda$
>
>(Eq. 8)
>
>### Patch averaging
>
>Due to overlapping patches, visible distinctions can sometimes be noticed at the borders of the patches (Appendix C). To eliminate these, we create a zone where the patches meet and take the average of both patches to achieve a seamless denoised result.
>
>### Slider
>(unchanged)

Masking, Patch averaging and Slider will also include Appendices G, C and F

Meta-review: This paper presents a practical approach to generating ultra-high-resolution images from pretrained text-to-image foundation models. The reviewers find that it addresses a significant limitation of previous work, and the presented paper is the first one to achieve gigapixel image generation on a single GPU, which is a substantial improvement over prior work. The method combines a few different technical designs (e.g., FFT features) for better generation quality. On the other hand, the presentation of the paper needs improvement, and the AC hopes the authors can further revise the paper to incorporate the suggestions of the reviewers. The AC believes that the paper has technical merits with strong experimental results, and the major concerns are addressed. Thus, the AC recommends the acceptance of the paper. The decision of this paper has been discussed with the SAC.
Decision: Accept (poster)