Is One GPU Enough? Pushing Image Generation at Higher-Resolutions with Foundation Models.

Authors: athanasios  tragakis
marco  aversa
chaitanya  kaul
roderick  murray-smith
daniele  faccio
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 02:59:47

Abstract: In this work, we introduce Pixelsmith, a zero-shot text-to-image generative framework to sample images at higher resolutions with a single GPU. We are the first to show that it is possible to scale the output of a pre-trained diffusion model by a factor of 1000, opening the road to gigapixel image generation at no extra cost. Our cascading method uses the image generated at the lowest resolution as baseline to sample at higher resolutions. For the guidance, we introduce the Slider, a mechanism that fuses the overall structure contained in the first-generated image with enhanced fine details. At each inference step, we denoise patches rather than the entire latent space, minimizing memory demands so that a single GPU can handle the process, regardless of the image's resolution. Our experimental results show that this method not only achieves higher quality and diversity compared to existing techniques but also reduces sampling time and ablation artifacts.

Main paper: 
 introduction: 
 Recent advances in diffusion models (DMs) have revolutionized the field of high-fidelity image generation. Foundational works like Ho et al. [2020] , Sohl-Dickstein et al. [2015] , Song and Ermon [2019] established the groundwork, leading to significant breakthroughs demonstrated by Dhariwal and Nichol [2021] . These models have evolved rapidly, with innovations such as new sampling techniques Lu et al. [2022] , Song et al. [2020] and capabilities for inpainting Lugmayr et al. [2022] and image editing Brooks et al. [2023] , Mokady et al. [2023] , Nichol et al. [2021] . However, even though DMs have shown remarkable results on high-fidelity image generation, scaling them to high-resolutions is still an open challenge. The introduction of the latent diffusion model (LDM) Rombach et al. [2022b] has made high-resolution image synthesis more accessible. However, most pre-trained DMs based on the LDM are limited to generating images with a maximum resolution of Trained models are those that are specifically designed for multi-resolution generation. Recent models like Matryoshka Gu et al. [2023] can generate various resolutions up to 1024 2 through a progressive training schedule. However, scaling beyond this is limited. CogView3 Zheng et al. [2024b] , based on Relay Teng et al. [2023] , upsamples from a base resolution of 512 2 , though higher-resolutions like 4096 2 remain a future goal. Fine-tuning methods such as DiffFit Xie et al. [2023] are costly, requiring 51 V100 GPU days. ASD Zheng et al. [2024a] introduces a memory-efficient sampling method capable of theoretically generating images up to 18432 2 resolution. Patch-DM Ding et al. [2023] , by training on 64 2 patches, can generate resolutions like 1024×512. LEGO Zheng et al. [2023] and Inf-DiT Yang et al. [2024] also rely on training. 
adapted models: 
 Adapted models, on the other hand, modify pre-trained models to generate higher-resolutions without additional training. MultiDiffusion Bar-Tal et al. [2023] offers controllable generation through multiple processes but is slow and prone to structural errors Du et al. [2023] , He et al. [2023] , Zheng et al. [2024a] . ScaleCrafter He et al. [2023] improves resolution by altering the convolution kernel dilation but faces memory limitations. DemoFusion Du et al. [2023] adapts MultiDiffusion Figure 1 : Examples of generated images using Pixelsmith. The proposed framework generates images on higher-resolutions than the pre-trained model without any fine-tuning. Images at different resolutions are shown with cut-out areas for both Pixelsmith and the base model. The higherresolution images are in scale with the images generated by the base model. Only the lower resolution version of the gigapixel image has been resized for a better visualisation. Some cut-outs of the gigapixel generation have resolution close to the base model which is 1024 2 and it can be seen that the images are comparable in aesthetics showing that our framework is capable of true gigapixel generations (zoom in to see in better detail).for constant memory use, though it is time-consuming. ElasticDiffusion Haji-Ali et al. [2023] and SyncDiffusion Lee et al. [2023] aim for specific high-resolution goals, but object repetition remains a problem at large scales Jin et al. [2024] . Recently, HiDiffusion Zhang et al. [2023] introduced modifications to the UNet architecture to prevent object duplication; however, it is limited to a maximum resolution of 4096 2 due to high GPU memory requirements. AccDiffusion Lin et al. [2024] employs patch-content-aware prompts and adjusts the attention masks within the UNet to suppress artifacts, but these changes result in blurry images. Similarly, Fouriscale Huang et al. [2024] is another model that alters the UNet but exhibits issues at higher-resolutions, particularly at 4096 2 . Current approaches excel in some areas but fall short in others. To enable high-resolution generation on a single consumer GPU, models must be adapted to avoid retraining costs. Efficient memory usage is crucial to prevent the need for expensive, high-memory GPUs, and the process must be fast and artifact-free. We propose a flexible framework that addresses these challenges based on text conditions.Figure 2 : Overview of the patch denoising process proposed by DiffInfinite: The top row represents the latent space, while the bottom row tracks the timesteps for each pixel. Each pixel should be denoised only once per timestep, so when overlapping occurs, already denoised pixels revert to their previous values from the prior timestep. After denoising, these reverted pixels are restored to their original denoised state from the current timestep.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: see Section 6Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
diffusion models: 
 DMs Ho et al. [2020] , Sohl-Dickstein et al. [2015] are probabilistic generative models that first add noise to a distribution during diffusion and then learn to remove this noise during denoising. This way, during training, a Gaussian probability distribution is learnt and during inference sampling from the Gaussian leads to the data probability distribution. Executing this process in the latent space Rombach et al. [2022b] is more resource efficient allowing for faster training and inference times. In a Latent Diffusion Model, let z 0 ≡ E θ (x) represent the clean training data x ∼ q(x) mapped to the latent space by the VQ-VAE encoder E θ . Beginning from z 0 , at each timestep t ∈ {1, . . . , T }, noisy latents z 1 , . . . , z T ∼ q(z t | z t-1 ) := N (z t ; √ 1 -β t z t-1 , β t I) are sampled according to a noise schedule function based on the timestep β t ∈ [0, 1]. During training, the model learns to denoise these latent variables using a conditional prompt c . At inference time, starting from a random latent z T ∼ N (0, I) , the model generates by iteratively samplingz 0 ∼ p θ (z 0 | z T , c) = T i=1 p θ (z t-1 | z t , c).The default LDM denoising process samples the entire latent space at each timestep, which becomes resource-intensive as the resolution increases. Methods like Li et al. [2024] distribute this load across multiple GPUs, but this requires expensive hardware. To address this, we adapted and refined the DiffInfinite sampling method Aversa et al. [2024] for text-to-image DMs, enabling ultra-highresolution generation efficiently on a single GPU.At each timestep, random patches are selected for denoising, and this process is repeated until the entire latent space is denoised. The randomness is controlled for efficiency. We track which pixels have been denoised at each timestep, and select areas where pixels have not been denoised yet. This sampling process avoids excessive inference times, especially at very high-resolutions like 32768 2 . In our experiments, we sample images at higher-resolution using fixed 128 2 patches in the latent space. To handle overlapping patches, pixel values are temporarily reverted when already denoised pixels are encountered, ensuring correct processing. After the current patch is denoised, the pixels that were reverted are restored to their denoised values. Only the first denoising of each pixel is retained for each timestep, preventing redundant denoising (see Fig. 2 ).Despite its advantages, the DiffInfinite sampling method still relies on segmentation masks to condition each patch, providing rich spatial information. However, in text-to-image DMs, where a global text prompt conditions the entire latent space, this method is insufficient on its own. Applying the same text prompt to each patch results in repetitive content and poor-quality generations. To overcome this, we introduce a guiding mechanism that incorporates structural information between patches alongside the text prompt, improving the quality and diversity of the generated images. guidance latents control the image generation. The framework fuses guidance latents (green patches) with high-resolution latents (purple patches) using the Fast Fourier Transformation (FFT). The phases are averaged and combined with the amplitude, then transformed back via the inverse FFT (iFFT). A chess-like mask integrates information from the successive guidance step (orange), resulting in fully processed patches (cyan). Right of Slider (Pure Generation): the generation relies only on the prompt. Higher-Resolution Comparison: while the base model upscales the bust with disfigured hands, the proposed method enhances details, corrects distortions, and prevents new artifacts.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Answer: [Yes] Justification: Yes the code will be open-sourced and the data to evaluate is already opensource.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
problem statement: 
 Currently, pre-trained text-to-image LDMs generate a latent variable z 0 ∼ p θ (z 0 |z T , c) with a fixed size using a diffusion process, which is then decoded into pixel space using the decoder D θ of a VAE. In this section, we show how to leverage foundational generative models to sample arbitrarily large images based on a given prompt. Without requiring additional training or fine-tuning, the LDM, trained on images with size H × W × 3, is adapted to generate images with size mH × nW × 3, where m, n ≥ 1 are the scaling factors for achieving higher-resolution.We present Pixelsmith, a framework with a flexible number of cascaded steps (Section 4.3) designed to generate images at ultra-high-resolution on a single GPU. The implementation of the Slider (Section 4.4) provides control over the generation process using patches (Section 4.2), making it resource-efficient regardless of the resolution.In this section, we describe the workflow of Pixelsmith, detailing how the framework adapts pretrained text-to-image LDMs to generate images with higher-resolutions on a single GPU (see Fig. 3 ). In order to generate ultra-high-resolution images without artifacts, we introduce these key components: the Slider (see Sec. 4.4), patch averaging (see Sec. 4.5) and masking (see 4.6).Text-to-image generation First, given a conditional prompt c, we use SDXL to sample the latent variable z0 and decode it into pixel space as x = D θ (z 0 ), generating a 1024 2 resolution image.Upsampling process After the image generation, we apply an upsampling algorithm to increase the image resolution (see Sec. 4.3 for details). In our case, we used Lanczos interpolation Lanczos [1950] to scale up to the desired resolution. However, upsampling leads to a blurred output and lack of additional content. The upsampled image, x guid , will serve as guidance for our generative process.Once the guidance image is encoded in the VAE's latent space z guid 0 = E θ (x guid ), we can easily sample each latent variable of the diffusion process through the forward diffusion process z guid t ∼ q(z guid t |z guid 0 ).The generative process starts from z T ∼ N (0, I), which has the same dimensions as z guid T . At each step, a random patch is cropped as described in Section 3.2:EQUATIONwhere C i,j : R 4,h,w → R 4,p h ,pw is a cropping function that extracts patches of size (p h , p w ) from the latent variables z t and z guid t of size (h, w) at the coordinates (i, j). For simplicity, we will refer to the cropped latent patch ẑ(i,j) t as ẑt and ẑguid,(i,j) t as ẑguid t throughout this discussion.The Slider's position (see Sec. 4.4 for details), indicated by a blue line in Fig. 3 , determines whether the guidance mechanism (see Sec. 4.4.1 for details) or unguided patch denoising will be applied. In the unguided mode, each patch is based solely on the previous one and the text condition, similar to a conventional patch denoising process. The Slider allows control over whether a generated image will be slightly or significantly altered compared to the previous resolution.After the denoising process has ended, the latents z 0 are decoded and the higher-resolution image is generated. Using a cascade upsampling approach, the generated image can be upsampled again, repeating the process to achieve an even higher-resolution image.Most of existing higher-resolution generative methods rely on cascade sampling approaches Denton et al. [2015] , Ho et al. [2022] , Menick and Kalchbrenner [2018] . Our approach, however, differs from previous works like Du et al. [2023] , Guo et al. [2024] in two significant ways.First, we perform upsampling directly in the pixel space rather than in the latent space. Manipulating the latent space distribution with transformations like upsampling can introduce distortions that the scheduler has not accounted for Chang et al. [2024] , leading to degraded image quality Hwang et al. [2024] . Second, our flexible method enables higher-resolution image generation in two generative steps: first by producing the base image, then by enhancing it to the desired resolution. For instance, Du et al. [2023] requires passing through intermediate resolutions (1024 2 , 2048 2 , 3072 2 ) to achieve a 4096 2 resolution, with even higher-resolutions necessitating additional steps. This results in prolonged waiting times and multi-scale duplications, as each step may produce duplicates at different resolutions (Appendix D).In contrast, our framework can generate any resolution directly from the base one, making it feasible to scale up to a 32768 2 image directly from the base resolution of 1024 2 . However, intermediate steps can sometimes yield better small-sized details (Appendix E). Our flexible approach supports both two-step generation and cascade generation, allowing users to select their preferred sampling method depending on the use case. We highlighted the artifacts introduced by generating at higher scales. These artifacts demonstrate the challenges of maintaining coherence and accuracy when scaling up the resolution without additional guidance.Currently, higher-resolution generative models often encounter issues such as duplications and atypical anatomical features or structures He et al. [2023] . To mitigate these artifacts, we introduce the Slider (indicated by a blue line in Figure 3 ). The Slider is a parameter that determines at which denoising step there is the transition between the guidance mechanism introduced in Sec. 4.4.1 and traditional unguided generation. The Slider can either constrain the denoising process to eliminate undesirable outputs or allow for less constrained generation to add details.When set to zero, the generation is entirely unconstrained. This results in an image that might resemble the 1024 2 base image but will likely contain multiple duplicates. At t = T , the generation is highly guided, leading to an image that is almost identical to the previous resolution. The optimal value for the Slider varies depending on the base image. Adjusting the Slider manually is necessary to achieve the best output. This manual adjustment allows for fine-tuning the balance between constrained and unconstrained generation, ensuring that the resulting image maintains the desired quality and level of detail. The ideal Slider setting is influenced by factors such as image complexity, the presence of fine details, and the desired resolution. Experimentation and iterative adjustments are often required to find the most effective value for each specific image and resolution.An example demonstrating the impact of the Slider's position can be found in Appendix G.The guidance mechanism combines the random patches ẑguid t , ẑt , and ẑguid t-1 to generate the updated patch ẑt-1 . First, the patches ẑguid t and ẑt are transformed into the Fourier space using the Fast Fourier Transform (FF T ), where their phase components ϕ t and ϕ guid t are averaged:FF T (ẑ t ) = A t e iϕt FF T ẑguid t = A guid t e iϕ guid t ⇒ ϕ t = arg e iϕt + e iϕ guid t (2)Here, A t and A guid t represent the amplitudes, while ϕ t and ϕ guid t are the phases. 2 The averaged phase ϕ t is then combined with the amplitude A t from ẑt to form a new Fourier representation ẑFFT t = A t e iϕ t . While the amplitude A t preserves the intensity and contrast information, the phase averaging process ensures that the resulting phase contains structural characteristics from both 2 Directly averaging phases, as in ϕ 1 +ϕ 2 2 , can lead to issues due to the periodic nature of sinusoidal functions. For example, averaging ϕ = 0 and ϕ = 2π -ϵ should result in a phase near zero, not π. To correctly average phases, we compute the arg(•) = arctan(y/x) of the sum of the complex numbers' phase components.the generated and guiding images. This modified Fourier representation is transformed back to the spatial domain using the inverse Fast Fourier Transform ẑiFFT t = iFF T ẑFFT t . The output ẑiFFT t is then used as the condition for the reverse diffusion process to generate the next patch ẑiFFTt-1 ∼ p θ ẑiFFT t-1 | ẑiFFT t , c. This solution suppresses local artifacts and helps maintain lowfrequency structural consistency between the reverse diffusion steps, reducing the likelihood of significantly altering the global properties of the generated image. However, the guidance mechanism alone still generates some long-range discrepancies across the image due to prompt sharing between the patches. To mitigate these artifacts and prevent prompt duplications across the latent space, we introduce a masking method(see Section 4.6).Overlapping patches can sometimes cause visible differences at their borders. To address this, we introduce a transition zone where the values of overlapping patches at timestep t are averaged, producing a smooth and seamless denoised output. In Appendix C, we illustrate the artifacts caused by patch overlap and show how averaging effectively removes them.One of the main reasons patch-based image generation in diffusion models Bar-Tal et al. [2023] suffers from artifacts at higher-resolutions is the use of the overall image text prompt for each patch during the denoising process. This leads to duplicate structures forming in the latent space. We partially addressed the spatial coherence with the guidance mechanism in Section 4.4.1. To further improve results, we combine the sampled ẑiF F T t with the image guidance ẑguid t using a chess-like mask Λ (Equation 3).EQUATIONIn Fig. 4 , we show the comparison between a higher-resolution image generated with and without masking with the image guidance. The image on the right, which is not constrained by pixels from ẑguid t , exhibits more freedom in generation and, as a result, introduces notable artifacts. For example, clouds transform into waves, and the blue sky morphs into the sea. Additionally, a small, seemingly artificial lake or river appears in the middle of the village, situated unnaturally close to the beach. 
experiments: 
 Pixelsmith is tested on a single RTX 3090 GPU, with all tested resolutions requiring 8.4 GB of memory. Performance is evaluated on the LAION-5B dataset Schuhmann et al. [2022] by randomly sampling 1,000 image and text prompt pairs. The metrics used for evaluation are Fréchet Inception Distance (FID) Heusel et al. [2017] , Kernel Inception Distance (KID) Bińkowski et al. [2018] , Inception Score (IS) Salimans et al. [2016] , and CLIP Score Radford et al. [2021] , with the FID metric computed using the clean-FID approach Parmar et al. [2022] (for further comparisons, see Appendix B). Additionally, we compare our results with a super-resolution model in Appendix F.We conducted a quantitative examination of our framework at a resolution of 2048 × 2048 pixels, focusing on key factors that influence its performance: the Slider position, the role of amplitude and phase in the latent space, the importance of masking during guidance, and the impact of averaging overlapping patches. Our results are shown in Table 1 .Slider position Our findings highlight the critical role of the Slider position in the quality of the generated images. Setting the Slider to position 30 (proposed method) provides an optimal trade-off between guidance and generation, outperforming other positions like 1 (SP1, no guidance), 24 (SP24, mid-point), and 49 (SP49, full guidance). A Slider position of 1 introduces numerous artifacts due to insufficient guidance, while a position of 49 lacks fine detail because it relies too heavily on lower-resolution. Position 24 has been chosen as mid-point in the diffusion process, it is closer to the optimal value but does not yield the best results across our dataset. Amplitude and phase In exploring the role of amplitude and phase within the latent space, we experimented with different configurations. The proposed method averages the phase of the guidance latents and the current latents while using the amplitude of the current latents, as detailed in the Methods section (see Figure 3 ). We compared it with other setups: one that averages the amplitude of both latent spaces while using the phase of the current latents (denoted as A in Table 1 ), and another that averages both amplitude and phase from both latent spaces (denoted as A&ϕ). The proposed method demonstrated superior performance in maintaining image quality and preserving fine details. Averaging the two phases provides guidance through structural information, while averaging the amplitudes only conveys the intensity of the frequencies. Averaging both offers information on both structure and intensity which overly influences the final image, leading to a lack of fine details expected in a higher-resolution output.Masking Masking emerges as a significant factor in enhancing image quality. The quantitative metrics indicate that the chess-like mask leads to slightly better scores compared to using no mask (as seen in the No Mask column of Table 1 compared to the proposed one). Visual inspections reveals that incorporating the mask is crucial for removing artifacts (see Figure 4 ).Overlapping patches Lastly, we assessed the impact of averaging overlapping patches. Implementing patch averaging improved the performance compared to not using it, as evidenced by the comparison between the proposed method and the No Averaging column in Table 1 . Although, despite the small difference in the metrics, averaging overlapping patches effectively eliminates artifacts that occur at patch borders due to overlapping regions, resulting in smoother and more coherent images (see Appendix C). This discrepancy underscores the limitations of metrics like FID in detecting high-resolution artifacts, highlighting the necessity of qualitative assessments. We compare Pixelsmith to state-of-the-art frameworks in the higher-resolution image generation task, as well as to SDXL, which serves as the base model for our adaptation. To ensure a fair comparison, we include only models with native resolutions comparable to ours, excluding those trained at resolutions smaller than 1024 2 . While Pixelsmith outperforms other methods in most metrics (see Table 2 ), its real advantage lies in its ability to adapt to the unique characteristics of each individual image generation. This flexibility enables precise control to emphasize details and reduce artifacts. In Figure 5 , we qualitatively show how Pixelsmith maintain the image structure by adding fine-details while other models introduce duplications or artifacts. There, we highlighted with red arrows the duplications and with purple arrows spurious artifacts. Additional comparisons can be found in Appendix B.  
discussion and considerations: 
 Pixelsmith can generate images at arbitrarily high-resolutions, but as the resolution increases, adding more generative details without introducing artifacts becomes difficult due to the smaller denoising patch size relative to the latent space. For example, a 32768 2 image maps to a 4096 2 latent space with a 128 2 denoising patch. The higher the resolution to generate in a single step, the higher the Slider value is required to suppress artifacts. However, increasing the Slider value reduces the finer detail generation and return an image which closely resembles the previous resolution, limiting the true quality of the higher-resolution image. Generating an image through a cascade generation approach allows for using a lower Slider value at each stage, providing more details and fewer artifacts compared to single-step generation, as the lower Slider settings can be more effective in this incremental process. Additionally, appropriate metrics for evaluating high-resolution images are lacking, and further research is necessary to improve quantitative evaluation methods, as highlighted in previous studies Jin et al. [2024] as well.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The training is irrelevant as we did not train any model. Details about evaluating are provided (see Section 5). We cannot share the subset to evaluate as we do not hold the rights but an official download link will be provided. Code will be provided as well.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusions: 
 In this work, we introduced Pixelsmith, a text-to-image generative framework that leverages a pretrained foundation model to generate higher-resolution images. Our flexible method uses a base image generated at the lowest resolution, combined with the Slider mechanism, to ensure both structural coherence and fine detail enhancement in the resulting images. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We follow previous works in the field. We use relevant metrics and rely on qualitative comparisons.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: the used GPU is mentioned. Time of inference is also provided.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We use a pre-trained diffusion model which should be used as every other generative model, with care. We do not fine-tune or change it in our work.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Justification: Our work does not change the societal impact of diffusion models as the generation remains the same as the pre-trained model.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [No] Justification: We use a pre-trained model without altering the weights. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Everything is cited and the pre-trained model is free to be used for research. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [No] Justification: A pre-trained model is used, we are not realeasing it for the first time Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We show the qualitative improvements Pixelsmith provides to the base model through an example here. We observe that Pixelsmith improves the generation at higher image generation resolutions.In Figure 6 , we can see that relatively small and larger areas in the image are also improved after each iteration of the image generation. The final image resembles the actress more in terms of facial attributes and has more finer details in terms of multiple physical characteristics.Figure 6: We show an example of a two-step image generation process here. The base model provides a 1024 2 image which our method then uses to iteratively generate a 2048 2 and a 4096 2 image. It is evident from this example that the higher-resolutions overall have increasingly improved facial characteristics, demonstrating the effectiveness of our framework (zoom in to see in better detail). Patch denoising may introduce artifacts as seen in Figure 8 . This occurs because we sample overlapping patches from the latent space, which can result in parts of the same area being denoised differently across patches. We address this issue by averaging along the edges where the patches meet. Figure 9 demonstrates our approach visually. Here, the image on the left represents the first noised latents. During the first denoising step (the next image), we sample a random patch (shown in blue) for the denoising process. After every denoising step, our algorithm checks for overlap between patches to perform averaging to remove any potential artifacts that may occur. The second denoising step (the third image), demonstrates a random patch sampled (in green) which overlaps with the patch in the previous denoising step. The third image shows an area of overlap where artifacts can be potentially generated. To alleviate this issue, we then take a tolerance of 10 pixels in the overlap but within the bounds of the first (blue) patch, and take an average across that region (as shown in yellow in the final image in the figure). From our experiments, we note that averaging across the edges of the patches, forces that region to create a smooth structure instead of horizontal and vertical lines (as shown in the second image in Figure 8 ). In order to prevent degrading the quality of the final generated image, we do not include pixels exclusively from the patch of the second denoising step (pixels that are immediately adjacent to the edge of the random patch from the first denoising step and fall within the second patch) as they primarily contain noise and this degrades the quality of the entire image.In Figure 8 , for illustration purposes, we keep the higher-resolution at 1536 2 . The Slider here is at position 0 as we want to encourage the algorithm to create artifacts that are easily distinguishable, as a stronger guidance results in them being less noticeable. We only use three inference steps here to keep the comparison as close to the base resolution as possible.Existing patch diffusion approaches risk introducing artifacts with each denoising patch. This phenomena is more prominent at higher image resolutions as the increase in the image resolution leads to an increase in the size of the latent space while the size of the denoising patch still remains constant. Duplications are especially prominent in approaches like Du et al. [2023] where a fixed number of intermediate resolution steps may amplify this issue progressively with every step. We demonstrate this in Figure 10 . Although we appreciate that using many steps helps refine the image gradually, depending on the generation the steps may have the negative effect of progressively amplifying a generated artifact at a lower resolution. For this reason we created Pixelsmith to be flexible in its image generation capabilities. Depending on the required result and text-prompt, we allow generating a higher-resolution image without generating any intermediate resolutions first. Image generation at intermediate steps (see also Appendix E is treated as a hyperparameter in our framework. In Pixelsmith, generating the required resolution directly after the base model helps with removing multi-scale duplications of unwanted artifacts (Appendix D). However, allowing to generate multiple intermediate higher-resolution images before the required resolution also leads to a much better definition of objects present in the image at the higher-resolution. We demonstrate this with an example, as shown in Figure 11 . We generate a 5440 × 3072 image from a base image of resolution 1360 × 768 directly, as well as going through the intermediate step of generating a 3400 × 1920 resolution image before the final resolution. We amplify the same region in all generated images to show the advantage of having an intermediate generation. Both 5440 × 3072 images are highly detailed, but comparing the amplified region (zoom in the image to see in more detail), we can clearly see the high frequency components in the image, such as the grass, are a lot more well defined even though they appear as pixelated regions in the base image. There can however, be cases where this difference in details in the images are not needed/noticeable and a one step image generation would be the preferable and faster choice in such a case.In addition to comparing our performance with current state-of-the-art models for higher-resolution generation, we also compare our results with a diffusion super-resolution model. Following Scale-Crafter by He et al. [2023] , we compare our results with a latent upscaler, but as our base model generates images at a 1024 2 resolution, choosing the 4× upscaler leads to creating a lot of computational overhead. Hence, we choose the 2× upscaler Rombach et al. [2022a] and evaluate on the 2048 2 image resolution. Table 3 , shows that even though the super-resolution model is trained to upscale, we still outperform it across all metrics. A generated image depends on many factors, namely, the input text-prompt, the input negative text-prompt, the seed, and the number of inference steps to name a few. The generated images can differ across these factors and even across intermediate image resolutions in terms of shape, texture, colour and a plethora of other varying attributes. This is especially seen when using a model that was not trained on the data for the task at hand directly -making the task a lot more challenging.Varying sizes of objects in the image such as arbitrary big structures, differ from the smaller ones and miscellaneous textured surfaces in the images are different from smooth ones. To account for these changes, and to control and constraint image generation at higher-resolutions, we introduce The Slider. The Slider helps provide a varying amount of control to a user to create image generations depending on how much they want the previously generated scene to influence the next generation.In a real world scenario, this is especially useful as depending on the type of image being generated and the prominence of artifacts that could potentially be introduced in the image, the user can switch between allowing more guidance from the previous resolution, whereas for truly unconstrained image generation, the Slider provides the added benefit of more freedom. We demonstrate the expressive power of The Slider in Figure 13 .We see more bees generated in the image on the left when we position the Slider value at 1. This results in certain artifacts and changes the structure of the image. At the other extreme, setting the max value of the Slider ( 49) results in preserving the overall structure of the base image, while not introducing any novel aspects into the generated image. Setting the Slider to 27 allows the framework to trade-off between image generation and preserving the global structure where it generates more realistic look high frequency components like the hair on the bees' body. Depending on the different settings for the Slider we can constraint the image generation to be more free.We observe from our experiments that lower values tend to generate images that are rich in fine detail but at the same time, are prone to changing the global structure of the image while creating duplications and artifacts. Higher values tend to keep the overall global structure of the image intact, but do not refine the image to create finer details. The Slider is a tunable hyperameter that provides a means to trade-off between freedom and constraints to create the best possible generation of an image from the text-prompt. Figure 16 : Examples of images at base resolutions alongside their higher-resolution counterparts generated using Pixelsmith (zoom in to see in better detail). • lego astronaut, riding a lego horse on Mars• pearly gates of heaven, ethereal, intricate details, angels, cloud, 7th sky, eternal bliss, in an otherworldly universe, detailed, realistic, 8k uhd, high quality• photo realistic, ultra details, natural light ultra detailed portrait of a female necromancer, skeleton face volumetric fog, Hyperrealism, breathtaking, ultra realistic, ultra detailed, cyber background, cinematic lighting, highly detailed, breathtaking, photography, stunning environment, wide-angle• Landscape of a dark far away planet, rock and organic soil, glowing trees, UHD, masterpiece, trending on artstation, sharp focus, studio photo, intricate details, highly detailed, by Greg Rutkowski• photo of the dark sky with visible but very distant swirling galaxies and stars • Darth Maul in a white tuxedo, in a forest, photo realistic• Raw photo of a roman warrior looking at the camera, close-up portrait, high detail, masterpiece• ultimate Japanese style, ink wash. Black and White, japanese Writing in Red and Blue, trending on ArtStation , intricate details, masterpiece, best quality, Use Dream Diffusion Secret Prompt, Epic • masterpiece, best quality, high quality, extremely detailed CG unity 8k wallpaper, scenery, outdoors, sky, cloud, day, no humans, mountain, landscape, water, tree, blue sky, waterfall, cliff, nature, lake, river, cloudy sky, award winning photography, Bokeh, Depth of Field, HDR, bloom, Chromatic Aberration, Photorealistic, extremely detailed, trending on artstation, trending on CGsociety, Intricate, High Detail, dramatic, art by midjourney Figure 13 :• close-up photo of a bee, macro, 100mm lens, bokeh, depth of field, high quality Figure 12 :• samurai iron man, dark black samurai armor, Artgerm, photorealism, Unreal engine 5 highly rendered, glowing red eyes, Tilt-shift 4k,White Balance, High Contrast, Low Saturation, Bracketing Detailed. • macro photography of a sakura blossom flower, epic • photo of a cute fluffy cat, close-up, high detail Figure 15: • a turian on andromeda, 50mm lens Figure 16 • portrait photo of a beautiful Greek girl, with black pigtails, wearing fluffy white coat, wide mid shot, derelict building, soft lighting, Hasselblad, Voigtländer 50mm lens, in style of Oleg Oprisco • photography in the style of detailed hyperrealism ,cinematic composition, dramatic light, a glass with a forest in it, surrealism 8k, graphic of enchanted terrarium, 4k highly detailed digital art, 3D render digital art, 3D render stylized, stylized 3D render, highly detailed scene, 3D landscape, 3D landscape, 4 k surrealism, sci-fiish landscape, 3 D artistic render, surreal 3D render. Dreamlike, mysterious, provocative, symbolic, intricate, detailed, expressive,hyper detailed,intricate,poster,artstation • The gladiator stands after the battle, fear and horror on his face, tired and beaten, sand on his face mixed with sweat, an atmosphere of darkness and horror, hyper realistic photo. In post-production, enhance the details, sharpness, and contrast. • aerial photo, amazon forest • mosaic of van Gogh Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper presents a practical approach to generating ultra-high-resolution images from pretrained text-to-image foundation models. The reviewers find that it addresses a significant limitation of previous work, and the presented paper is the first one to achieve gigapixel image generation on a single GPU, which is a substantial improvement over prior work. The method combines a few different technical designs (e.g., FFT features) for better generation quality. On the other hand, the presentation of the paper needs improvement, and the AC hopes the authors can further revise the paper to incorporate the suggestions of the reviewers. The AC believes that the paper has technical merits with strong experimental results, and the major concerns are addressed. Thus, the AC recommends the acceptance of the paper. The decision of this paper has been discussed with the SAC.

Reviews: review: paper_summary: paper_summary: This paper introduces a framework for generating high-resolution images from text prompts using pre-trained diffusion models. The key innovations are: A cascading approach that uses lower-resolution generated images as guidance for higher resolutions. A "Slider" mechanism to control the balance between following the guidance and allowing novel generation. Patch-based denoising to enable generation of arbitrarily large images on a single GPU. Averaging techniques to reduce artifacts from patch-based generation. The authors demonstrate that Pixelsmith can generate images up to 32,768 x 32,768 pixels on a single GPU, outperforming existing methods in terms of quality and efficiency.
strength_weakness: strength_weakness: 1. Poor presentation. The writing of this paper needs substantial improvement to be published in a venue like NeurIPS. First, the language of the paper needs to be improved. Second, the paper spends a lot of space on unnecessary content. The method in this paper is not difficult, but it is very difficult to understand the method in one reading. In terms of writing, the authors did not succeed in emphasizing the core concept of their proposed method. In addition, the length of the paper is too long. Such a simple method, but the experiments are not introduced until page 8, which should not happen.
2. The core idea of ​​this paper seems to be to generate once and then upsample the generated image by patch-wise processing. (By the way, Figure 3 is difficult to understand and Eq. 4 is not explained). This method is essentially not much different from other operations that use diffusion for upsampling. So what is the advantage of this method? Why is this method feasible?
3. The role of FFT Transformation is not well demonstrated. No exploratory experiments are used to demonstrate the motivation and effect of this method.
questions: questions: See Weaknesses please.
limitations: limitations: The authors claim they discussed the limitation but I didn't find it in the text. Correct me if I am wrong.
paper_summary: paper_summary: The paper introduces Pixelsmith, a framework designed to utilize pre-trained diffusion models to enable high-resolution image generation using only a single GPU.
Patch-based denoising ensures that the entire generation process can be accommodated on a single GPU.
The Slider mechanism balances the trade-off between finer details and overall structure by controlling the transition from guided generation to native generation.
Guided generation operate on the latent fused by one upscaled higher-res image patch and one native-generated higher-res image patch in Fourier space, helping to maintain global structures as claimed by the authors.
Experimental results show that Pixelsmith not only produces high-quality and diverse images but also improves sampling time and reduces artifacts compared to existing techniques.
strength_weakness: strength_weakness: 1.	Paper Structure Issues: The content before the methods section is too lengthy, making the key method and experiments sections comparatively short and harder to fully understand. These are elaborated upon in the following points.
2.	Method Description is Hard to Follow: For instance, the overview in Lines #171-173 does not fully describe all the components involved in the method, requiring readers to review the paper multiple times to understand the entire pipeline. Additionally, the intuition behind combining $\hat{z}^{iFFT}{t-1}$ and $\hat{z}^{guid}{t-1}$ is not explained, which is confusing since $\hat{z}^{iFFT}_{t-1}$ already incorporates information from $\hat{z}^{guid}_t$.
3.	Errors and Mismatches in the Experiment Section: According to Table 1, using the real part is better, which contradicts Line #248. Line #252 describes the wrong column for "the next inference step". Also, the term “base model” in Table 2 is unclear—does it refer to the same model as Table 2’s Pixelsmith and Figure 1’s base model?
4.	Flaws in Figure Illustrations: In Figure 1, the base model image for “x256 (16384x16384)” is not displayed. Is this base model the same as the one mentioned in Table 1? In Figure 3, the illustration should be more concrete and easier to understand; however, some variable names are too small and there are too many blank areas. The fusion part is also unclear, with no mention of the real part of the Fourier transform and no explicit explanation of $\mu( . , . )$.
questions: questions: The aim of achieving high-resolution image generation on a single GPU is commendable, and the proposed method successfully accomplishes this goal. However, the presentation quality is relatively poor, especially with some key parts described incorrectly. I hope the authors can address the questions mentioned in the weaknesses section and resolve my concerns.
limitations: limitations: The authors have discussed the trade-off between achieving finer details and suppressing artifacts. They have also suggested proposing appropriate metrics for evaluating high-resolution image generation, which would be beneficial for the community.
paper_summary: paper_summary: This paper proposes a training-free method for diffusion models to sample images at higher resolution with limited GPU memory, which introduces several tricks, such as fourier merging, chess-mask deduplication, and slider control. Experiments show the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. Is there any literatures to support that the imaginary part of the fourier transform corresponding to the low-frequency of the signal, or is there any deeper analysis beyond the ablation results in Tab.1 to illustrate this? By the way, it is suggested to provide the visual ablations of this part.
questions: questions: 1. How to determine the patch orders in high-resolution generation, is it random orders or follow some principles.
2. From Tab. 2, it is not clear why Pixelsmith slower than Scalecrafter at 2048 resolution, while nearly 2x faster at 4096 resolution. Is there any trend that the bigger, the more efficient?
3. Is the content of the final scaled high resolution output significantly differ from the base output, whether the new objects will be generated?
limitations: limitations: The authors have adequately discussed the limitations of their work.
paper_summary: paper_summary: In this paper the authors propose a number of engineering tricks which enable generating at higher resolutions from a pre-trained txt2img diffusion model.
Notably, the requirements for the proposed method are relatively low.

In the proposed approach, an image of a standard resolution is generated first.
After that, is it upsampled (in RGB space) to the desired size and used as a guidance for the new, truly high-resolution image which is generated as a sequence of overlapping patches.
The guidance mechanism is implemented by mixing the imaginary component of Fourier decomposition of the currently denoised and guidance patches.
This procedure takes place not for all the denoising steps, but only up to certain noise level referred to as Slider.
The value of Slider controls how similar are the original standard-resolution image and its higher-resolution version.

Performance of the method is evaluated with commonly used metrics such as FID, KID, IS and CLIP-similarity.
Also, a lot of samples are provided for visual inspection.
strength_weakness: strength_weakness: 1. While the authors several times mention that "the imaginary part in the frequency space contains most of the low-frequency information of the image" (line 187) and "imaginary part provides more structural information than the real part" (line 249), they do not provide any references for this statement or supporting experiments. To me, this fact does not look evident enough. I am sure this needs more thorough explanation since otherwise it looks just like an empirical trick.

1. From the text, it is unclear which Slider value was used to obtain the numbers from Table 2. Was it the value of 30, as defined in line 244? Also, what exactly is "using next inference step (NIS)" (see line 253)? Is this about checkerboard mask mixing?

1. The presentation of Fig. 3 is not easy for understanding, I advise considering its redesign. For example, forward diffusion and taking imaginary part of FFT are denoted in the same way. 

1. It is unclear from the text what the guidelines are for selecting the size for the zone of averaging for overlapping patches (line 805).
questions: questions: I ask the authors to address the weaknesses listed above during the rebuttal period. In particular, I am interested in the deeper justification of  using imaginary Fourier coefficients than empirical evidence.

POST-REBUTTAL UPDATE: I keep my initial score. I think that although the method is very simple, this does not mean that the submission is bad. I believe that with improved presentation this paper can be interesting for the practitioners.
limitations: limitations: No actions needed.

Comments: title: Official Comment by Reviewer T7y4
Response to the rebuttal
comment: Thank you for the comment! We appreciate the opportunity to continue the discussion. 
> The author has partially addressed my concerns.

Can you please tell us which concerns have not been addressed so we can attend to them as well?
> However, I still think that the presentation of the paper is lacking at this stage.

In our original reply, we kindly asked for specific examples where the language was not up to the NeurIPS level, as it is important to us to have this constructive feedback. Could we please extend the original request here and ask for specifics on where the paper is lacking as well? We have revised the entire paper and followed all suggestions from all reviewers. It would greatly benefit our work to know which parts are lacking, as we understand that this is the main (remaining) issue with the review. Examples of language before the revision (referring to W1) and the presentation after all the implemented changes would help our work, as we believe in the NeurIPS review process and the best practices to help promote research.  
> However, since I cannot see the revised paper, I cannot judge whether the final presentation meets the requirements of NeurIPS.

As we cannot upload the full paper, I can kindly point you to the Method section, which is in the global response, and the Experiments section, which is a reply to reviewer wkBK. The Related Work and Foundations sections that were changed can be found below. The rest of the paper remains unchanged, so the full paper is now shown in our replies.
> 
> # Related Work
> Pre-trained DMs are following a ... each new version [30]. It is clear that there is demand for increasingly higher resolution generation.
> Currently, generating images ... application.
> ## Trained models
> (unchanged)
> ## Adapted models
> (unchanged)
> # Foundations
> ## Diffusion models
> DMs [15,42] are probabilistic generative models that first add noise to a distribution during diffusion and then learn to remove this noise during denoising. During training, a Gaussian probability distribution is learned, and during inference, sampling from the Gaussian leads to the data probability distribution. Executing this process in the latent space [37] is more resource-efficient, allowing for faster training and inference times.
> In formal terms, for a Latent Diffusion Model, if $z_0$ represents the data point in the latent space, given $z_0 \sim q(z_0)$ and $q$ being the diffusion process, then for timesteps t $\in$ $\{1,T\}$ $z_1,\ldots,z_T$ noisy latents, with variance $\beta_t \in (0,1)$, are produced, defining a joint distribution conditioned on $z_0$:
> 
>(unchanged equation 1)
> 
>(unchanged equation 2)
> 
> The training estimates an isotropic Gaussian distribution for $z_T$. During the denoising process we sample from $z_T \sim \mathcal{N}(0,{I})$ and remove the added noise using a neural network:
> 
>(unchanged equation 3)
>  ## Patch sampling
> The default denoising process of an LDM involves sampling the entire latent space at each timestep. While this approach works for lower resolutions, it becomes increasingly resource-intensive as the resolution increases.
> Instead, we modify the default process to denoise patches of a fixed dimension $128^{2}$ as introduced by DiffInfinite [1]. At each timestep, random patches are selected for denoising, and this process is repeated until the entire latent space is denoised (see Appendix Patch sampling). The DiffInfinite process relies on segmentation masks to condition each individual patch, providing rich spatial information. In a text-to-image DM, where the text-prompt is global for the entire latent space, using this method means that each patch is denoised with the same condition—the text-prompt. This leads to multiple repetitions of the condition and results in poor-quality generations. To address this, we implement a series of key components that enable scaling a pre-trained DM to resolutions never achieved before.

The above changes result in considerably smaller pre-Method sections, allowing the new, better-explained Method section to fit and enhance readability. The previous full text of this section will become a new appendix, titled "Patch sampling," with the addition of our explanation to question 2 of Rv T7y4. Nothing else changes or is added, so this appendix will not be new to you.
Thank you for the comment! We appreciate the opportunity to continue the discussion. 
>Most of my questions have been addressed, however, the reason why the imaginary part of the fourier transform corresponds to the low-frequency of the signal is still not illustrated well with deeper analysis, and depend on the ablation experiments, which may undermining the potential inspiration

As stated in our global response, all the references have been changed to reflect the concerns raised. For example, in Lines 187 and 248, where it was previously mentioned that the imaginary part of the Fourier transform corresponds to the low-frequency information of the signal, this has now been removed. In our revised paper, we do not state a connection between the imaginary part and the low-frequency information and we base our choice of the imaginary part on the results of the ablation experiments.
Thanks for the authors rebuttal. Most of my questions have been addressed, however, the reason why the imaginary part of the fourier transform corresponds to the low-frequency of the signal is still not illustrated well with deeper analysis, and depend on the ablation experiments, which may undermining the potential inspiration, therefore, I will not raise my score.
I have read the author's response, as well as the comments and discussions with other reviewers. The author has partially addressed my concerns. However, I still think that the presentation of the paper is lacking at this stage. I will improve my score. However, since I cannot see the revised paper, I cannot judge whether the final presentation meets the requirements of NeurIPS.
I would like to thank the authors for their feedback.