Noise Re-sampling for High Fidelity Image Generation

Authors: Hao Wang, Weihua Chen, Chenming Li, Wenjian Huang, Jingkai Zhou, Fan Wang, Jianguo Zhang
Primary area: generative models
Keywords: Generative models, Diffusion based models

Abstract:
Latent diffusion models (LDMs) have emerged as powerful tools for generating diverse and realistic samples across domains. However, their efficacy in capturing intricate details and small-scale objects remains a challenge. 
Our investigation reveals that VAE compression induces errors in the latent space and limits the generation quality. Furthermore, LDMs trained on fixed-resolution images struggle to produce high-resolution outputs without distortions, making simple resolution increases ineffective.
In this paper, we propose a novel **noise re-sampling** strategy that enables multi-scale generation of LDMs, allowing LDMs to "zoom in" and improve generation quality of local regions. By increasing the sampling rates from the noise perspective in the latent space, we effectively bypass the constraints imposed by VAE compression, thus preserving crucial high-frequency information. Our approach, a simple yet effective plugin for current LDMs, enhances the quality of image generation in local regions while maintaining overall structural consistency and providing fine-grained control over the scale of generation in latent diffusion models.
Through extensive experimentation and evaluation, we demonstrate the efficacy of our method in enhancing the generation quality across various LDM architectures. Our approach surpasses existing methods, including stable diffusion (SD) models, SD-based super-resolution methods and high-resolution adaptation methods, in generating high-fidelity samples of complex objects.

Decision: reject
Reviewer scores: 5, 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729590494788

thread_type:
official_review

<thread_object>
creator: reviewer_92ch

<content>
summary:
This paper proposes a noise re-sampling strategy and VAE-based upsampling to address the lossy compression problem of VAE in latent diffusion models, which are almost essential in the text-to-image field. The authors observe distortions in small objects generated by latent diffusion models like Stable Diffusion 1, 2, and 3, and propose increasing the sampling rates of local regions. The proposed noise re-sampling method crops the local region of the latent image generated by the diffusion model, upscales it using the proposed VAE-based upsampling method to match the size learned by the diffusion model, and then refines the upsampled crop before attaching it to the original image. The paper compares this method with approaches like Stable SR and Multi-Diffusion, demonstrating superior qualitative and quantitative performance.

strengths:
- This paper highlights the issue that VAE compression errors are more prominent in lower-resolution images, a problem that the generative AI community has not paid close enough attention to.

- The proposed method is training-free, making it easy to apply with minimal effort to any newly developed latent diffusion model.

weaknesses:
- The most fundamental way to address the VAE's lossy compression issue is by using a better VAE. For instance, OpenAI's consistency decoder [1] replaces the decoder of Stable Diffusion 1's VAE with a new diffusion model and releases model weights that can rapidly decode latents into images via consistency distillation [2]. Samples from the consistency decoder demonstrate strong reconstruction performance, particularly on low-resolution (256x256) images where reconstruction errors, like those mentioned in this paper, often occur. This paper should also include a comparison with the consistency decoder.

- Another approach is to ensure the diffusion model generates better latents. Recent research on diffusion guidance [3] suggests that strong classifier-free guidance at middle noise levels enhances image detail. The proposed noise re-sampling method also requires experiments with various guidance scales.

- Some implementation details are missing. The local patch chosen for upsampling should ideally be a small object prone to distortion, but how is this patch selected? What is the exact upsampling ratio used in experiments? When denoising the local patch with the diffusion model, what text prompt is used?

- The proposed VAE-based upsampling seems counterintuitive because it relies on the same VAE that the paper argues to suffer lossy compression issues. This suggests that the upsampled patch might contain significant artifacts.

- The paper lacks of qualitative results of SD3 and SDXL.

[1] Open AI, https://github.com/openai/consistencydecoder

[2] Song et al., Consistency Models, ICML 2023.

[3] Kynkäänniemi et al., Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models, 2024.

questions:
Stable Diffusion 3 is known for its ability to generate text well. Can noise re-sampling correct errors when text generation goes wrong?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729980478860

thread_type:
official_review

<thread_object>
creator: reviewer_vj6f

<content>
summary:
The paper proposes a Noise Re-sampling strategy to enhance high-fidelity image generation in Latent Diffusion Models (LDMs). This method selectively increases sampling rates in detailed local regions. Additionally, a VAE-based upscaling technique ensures consistency across local and global regions by upscaling latent spaces in the image domain. Experiments shows that the approach outperforms existing super-resolution and high-resolution adaptation methods on SD backbones.

strengths:
1. The paper introduces the concept of Noise Resampling, which is somewhat novel to me in the context of latent diffusion.
2. The paper studies the error in VAE reconstruction and attempts to address it with noise resampling. It's somewhat valuable.
3. The paper is well organized.

weaknesses:
1. The proposed solution is straightforward. To address the low-quality fine-grained details in SD generation, the method adds noises to generated patches then conduct the backward denoising process again. Although some methods such as VAE upscaling has been proposed to improve this method, this idea is straightforward without many insights into the diffusion process. 

2. The proposed method requires massive denoising and VAE upsampling over different resolutions of the generation. This would introduce significant computational costs. Unfortunately, I didn't see any discussions on method efficiency in the paper.

3. The performance improvement on global metrics, such as FID and KID, is trivial.

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730557886780

thread_type:
official_review

<thread_object>
creator: reviewer_9gds

<content>
summary:
This work focuses on the noise sampling rate/resolution in LDMs, and proposes a local reference re-sampling strategy to improve the high-resolution image generation performance. 
The proposed method uses the normal global noise-based denoised image as the reference, to further guide another branch which is focused on local region. 
Meanwhile, the method deploys a training-free mixed upsampling strategy to provide corresponding local latent noise aiming at generating patches with better quality.
Extensive experiments show the proposed method achieves good performance, especially for complex objects and other local details.

strengths:
1. The idea is quite simple but overall I think it is a good paper. Instead of directly increasing the sampling rate, using VAE to perform local space re-sampling in the image space is quite interesting, as it avoids the need for a higher-resolution dataset and re-training. Figure 5 also demonstrates that this method indeed captures more high-frequency information.

2. The proposed re-sampling strategy is straightforward and can be plug-and-play for LDMs.

3. The author's exploration of the impact of VAE compression on the generation quality of LDMs aligns with intuition, and this paper reminds me of a work IDM [1] that uses cascaded Diffusion Models for continuous super-resolution

5. The authors provide a demo code which is worth encouraging. I've tried it and it works fine.

[1] Gao S, Liu X, Zeng B, et al. Implicit diffusion models for continuous super-resolution[C]. CVPR 2023.

weaknesses:
1. Figure 4 is kind of ambiguous. 

     (1): If I understand correctly, the blue boxes in subfigures (a) and (b) should be the same size. I also suggest indicating the size of each box for clarity.

     (2): I suggest removing the "Direct latent space upsampling" branch below subfigure (c), or replacing it with the noise re-sampling branch corresponding to Figure 5 (d). Additionally, the title for subfigure (c) appears twice.

2. The statement in lines 367-369, "The final re-sampled clean latent code can be directly pasted back to the corresponding local region of the original output after decoding" is ambiguous. Based on Figure 4 (a), it should be decoded first, and then pasted back to the output image.

3. I am a bit confused by the statement in line 395, "This reference ensures that the denoising process aligns with the overall image". Since the local reference image only selects a portion of the patch without including information from the surrounding areas, how does it ensure continuity between the patch edges and the rest of the image?

4. I suggest conducting an ablation study on the selection of  𝑁 and the size of the local patch 𝑠 to explore the impact of different combinations on performance and efficiency.

5. How to determine the ratio 𝛼_𝑡 of latent upscaling and VAE-based upscaling in Equation 8? Does it also gradually decrease as 𝑡 increases?

6. I think the Figure 3 (d) in line 264 should be Figure 3 (b).

7. Some typos: for example, "to generated" should be "to generate" in line 299

8. There are some minor errors in the reference. For example, Self-Cascade [1] in ECCV is not correctly referenced and is repeated in lines 044 and 048. Also ScaleCrafter [2] is published in ICLR23.

[1] Guo L, He Y, Chen H, et al. Make a cheap scaling: A self-cascade diffusion model for higher-resolution adaptation. ECCV 2024.
[2] He Y, Yang S, Chen H, et al. Scalecrafter: Tuning-free higher-resolution visual generation with diffusion models[C]. ICLR 2023.

questions:
Please see the Weaknesses part.

Overall, I'm satisfied with the paper's quality and am willing to improve my score if the authors can address my questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730683548857

thread_type:
official_review

<thread_object>
creator: reviewer_4z4z

<content>
summary:
The paper proposes an interesting noise re-sampling technique aimed at improving the fidelity of image generation in Latent Diffusion Models (LDMs), particularly for small, detailed objects that are challenging to render accurately. Traditional LDMs suffer from detail loss due to Variational Autoencoder (VAE) compression, especially in high-frequency regions critical for intricate textures. The authors address this by locally increasing the sampling rate in noise space, allowing LDMs to generate high-resolution details without altering global structures. Additionally, they introduce a VAE-based upscaling method to ensure consistency between upsampled local patches and the global image. The approach demonstrates superiority over baseline models, including Stable Diffusion and super-resolution methods, by enhancing image detail and fidelity across multiple scales.

strengths:
- The identified issue in latent diffusion models is interesting and requires attention for better image generation.  
- The perspective from signal processing is interesting, and provides a feasible direction for optimization with diffusion models.  
-  From experimental evaluations, the authors demonstrated the generalizability of their approach for different models and in all cases, they further boosted the performance of the the backbone model.

weaknesses:
Firstly, the paper writing is poor. It's a little hard to capture the overall idea of this paper. A few more concrete comments include: 
- The whole method section makes it hard to capture the high-level idea. I am not sure if I understand this paper correctly, does the whole logic of the proposed method be as follows: Identify and decode local patches --> Upsampling in image space --> Re-encoding with added noise and denoising --> Merging with global image?   
- Following the previous point, the whole method section should be revised with a clear problem formulation (i.e. what are the input and outputs, and what are the goals), a straightforward model pipeline, and consistent and well-explained notations. 
- Quite a few texts are redundant. The authors repeatedly discuss the motivation of their approach in the introduction, method, and experimental evaluations.   
- Table 1 is hard to read. Does SD1.5, SD2 mean backbone models, and for all the compared models, they are built upon SD?  

Next, the novelty of this paper is limited. The proposed method is more like an engineering-wise post-processing to enhance the fine-grained details of the diffusion models. There is no tech contribution from this paper, and the signal processing perspective does not fully align with the proposed method. Even if it is, the perspective itself is more like a way to explain why diffusion models miss fine-grained details rather than a contribution of this paper. 

Other questions:
- high-frequent components or fine-grained details? In the paper, the authors take a unique perspective on signal processing, and reveal that VAE has a higher error on high-frequent objects. However, they essentially propose a resampling method to restore fine-grained details. Taking Figure 6 as an example, the proposed method did better restore the details, but the high-frequent components (boundaries) are not clearly better. In the method section, the ``resampling'' is simply crop-and-resize. In other words, the proposed method itself is not quite aligned with signal processing.  
- How about the inference time? The proposed method runs two rounds of denoising and at least two rounds of VAE encoding and decoding, will the inference time doubled, and how to further optimize the inference time?  
- In real practice, how would the proposed method work, without manually selecting resampling regions?

questions:
See the previous section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730875669251

thread_type:
official_review

<thread_object>
creator: reviewer_c8p3

<content>
summary:
The paper proposes to use noise re-sampling to generate high-fidelity images. The method takes place in the inference stage of the diffusion model where 1) an image is sampled first using the original diffusion process 2) a patch is cropped and upsampled and then sent to the encoder to get the latent features 3) a random Gaussian noise is added to the latent and denoise back to get clean image 4) paste back to the image. The paper shows comparisons on multiple images especially human images which demonstrates the effectiveness of the method.

strengths:
1. The method is simple and does not need any training.
2. The results show improvements in the cropped regions in the generated images.

weaknesses:
1. How would the method affect the inference speed? As the proposed method requires some more operations than the vanilla generation process, it would be better to understand the efficiency of the method as well.
2. The method contains randomness as it involves the addition of the noise and the random selection of the cropped region, how would the method's robustness be? Will it keep outputting high-quality images regardless of the randomness?
3. Could the authors also present some results on SD 3, the paper only shows some original results in Figure 1 but lacks the results output by the proposed method.

questions:
See the weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732624878436

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Message from Authors to Reviewers

comment:
Dear Reviewers,

Thank you for your thoughtful comments and valuable feedback on our manuscript.

We have carefully revised the paper and addressed each of your concerns in detail. We believe that we have sufficiently addressed all the issues raised during this discussion period. If you have any further questions or require additional clarification, please do not hesitate to reach out.

We would be grateful if you could reconsider your evaluation and, if our revisions meet your expectations, kindly adjust your score accordingly.

Thank you once again for your time and consideration.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733298604284

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
Dear reviewers, 

We would like to express our sincere gratitude to all reviewers for their thoughtful and constructive feedback. We are pleased that the reviewers have acknowledged our contribution in addressing the limitations of current Latent Diffusion Models (LDMs) and for recognizing the significance of our novel approach. Specifically, our work **reveals the inherent constraints of LDMs in generating complex objects and fine-grained details**, and proposes **a solution that enhances the model's ability to handle these challenges by investigating the properties of latent space and the limitations of current interpolation methods.**

During the rebuttal period, we have carefully addressed the concerns raised by the reviewers regarding the efficiency, motivation, and contributions of our method. We are pleased to report that we successfully addressed reviewer 4z4Z and reviewer 93CH’s concerns, whose feedback was invaluable in refining our manuscript. We sincerely appreciate their constructive input. Additionally, we would like to thank reviewer 9GdS for initially acknowledging our contributions. In response, we have carefully revised the manuscript and provided detailed responses to the constructive feedback from reviewer 9GdS. The responses regarding the robustness, consistency, and hyperparameters of our method were also addressed in responses to reviewers 4z4Z and 93CH and have been approved by them.

We understand that time is limited, and some reviewers may not be available to provide additional responses. Nonetheless, we believe we have provided sufficient responses to all concerns raised. Specifically, we have addressed and justified the concerns raised by reviewer C8P3 and reviewer VJ6F regarding the robustness, efficiency, and effectiveness of our approach. These concerns, also mentioned by reviewers 4z4Z, 93CH, and 9GdS, have been addressed in our responses and acknowledged by them.

We sincerely appreciate the time and effort all reviewers have put into evaluating our work. Thank you again for your thoughtful and valuable contributions to this work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a noise re-sampling strategy and VAE-based upsampling to address the limitations of latent diffusion models, particularly in generating fine-grained details. However, reviewers noted several weaknesses, including the paper's presentation issues, with redundant and unclear sections, and a lack of clarity in implementation details, such as patch selection and upsampling parameters. Additionally, the novelty of the method was questioned, as it was seen as a straightforward engineering solution with sensitivity to upsampling ratios and marginal improvements on global metrics like FID and KID.