ClearSR: Latent Low-Resolution Image Embeddings Help Diffusion-Based Real-World Super Resolution Models See Clearer

Authors: Yuhao Wan, Peng-Tao Jiang, Qibin Hou, Hao Zhang, Jinwei Chen, Ming-Ming Cheng, Bo Li
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Diffusion Model, Super-Resolution, Real-World Image Super-Resolution

Abstract:
We present ClearSR, a new method that can better take advantage of latent low-resolution image (LR) embeddings for diffusion-based real-world image super-resolution (Real-ISR). Previous Real-ISR models mostly focus on how to activate more generative priors of text-to-image diffusion models to make the output high-resolution (HR) images look better. However, since these methods rely too much on the generative priors, the content of the output images is often inconsistent with the input LR ones. To mitigate the above issue, in this work, we explore using latent LR embeddings to constrain the control signals from ControlNet, and extract LR information at both detail and structure levels. We show that the proper use of latent LR embeddings can produce higher-quality control signals, which enables the super-resolution results to be more consistent with the LR image and leads to clearer visual results. In addition, we also show that latent LR embeddings can be used to control the inference stage, allowing for the improvement of fidelity and generation ability simultaneously. Experiments demonstrate that our model can achieve better performance across multiple metrics on several test sets and generate more consistent SR results with LR images than existing methods. Our code will be made publicly available.

Decision: reject
Reviewer scores: 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730193647335

thread_type:
official_review

<thread_object>
creator: reviewer_dufd

<content>
summary:
This paper introduces ClearSR, a novel method for real-world image super-resolution (Real-ISR) using pretrained T2I diffusion models. ClearSR leverages LR embeddings to constrain ControlNet's control signals,  extracting LR information at detail and structure levels. The authors design DPM and SPM modules, which enhance image details and maintain structural integrity, respectively. Additionally, they propose an LSA strategy during inference to balance fidelity and generative capabilities. Extensive experiments demonstrate that ClearSR outperforms existing methods across multiple benchmarks and metrics.

strengths:
A key challenge in RealSR tasks using powerful T2I models is generating fine details while maintaining fidelity, which presents a trade-off. ClearSR explore this by using a pre-trained VAE encoder as an initial feature extractor for LR images to preserve fidelity as much as possible, and designing DPM and SPM to handle specific control tasks. Additionally, ClearSR observed that the added realistic details largely come from the final inference steps. Therefore, it introduced the LLA mechanism to move away from the LR latent space in the final stages, enhancing generative capability and improving model flexibility.

weaknesses:
1. Lacks more detailed comparisons, such as inference time, parameter count, and computational cost.
2. Missing some key details, like the number of inference steps, and Figure 10 doesn't provide the names of the comparison methods.
3. While the motivation is good, the novelty of the solution seems relatively weak.

questions:
[1] Selection of α and β Parameters:

a. How were the values for α and β in the LSA strategy chosen? Did you perform a systematic parameter search or optimization? Are these parameters required to be tuned for different datasets or image types, and is there a way to automate their selection?

[2] Implementation of LoRA Layers

How does the choice of LoRA rank (set to 16) impact model performance, and was this rank value optimized experimentally?


[3] Something about classfier-free guidance, cfg

During the inference stage, by adjusting the CFG value, RealSR methods based on pre-trained T2I diffusion models can also balance fidelity and perception. The authors did not report the CFG settings during inference, such as the CFG value and negative prompt. Additionally, the proposed LSA control method needs to be compared in detail with the CFG control method to highlight the differences.

If the main concerns are well addressed, I will consider increasing the score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730360724957

thread_type:
official_review

<thread_object>
creator: reviewer_4hyn

<content>
summary:
This paper introduces ClearSR, a novel approach designed to enhance the utilization of LR image information in SR tasks. The DPM and SPM modules are designed, enabling the extraction of more LR details and structural information. The method also demonstrates that latent LR embeddings can be used to adjust the latent space during inference, improving both fidelity and generative quality. ClearSR outperforms existing SR models across multiple metrics on various test datasets, producing SR results with rich generated details while maintaining consistency with the LR images.

strengths:
1. This paper proposes two modules to extract more LR details for structural and detail preservation.

2. In the inference stage, this paper proposes an LSA strategy, which performs different directional adjustments towards LR embeddings in the latent space in the earlier and later steps. This idea is reasonable and interesting.

3. The results look good, the writing is well, and the paper is easy to follow.

weaknesses:
1. This paper introduces two modules (DPM and SPM) to enhance the utilization of LR image information, but these increase model parameters and inference time compared to ControlNet. However, algorithmic complexity is not discussed.

2. The description in Line 190 is confusing; PASD does not use the CLIP image encoder to extract LR features.

3. The explanation of image-level feature $\textbf{p}$ in Figure 3 is unclear. How is $\textbf{p}$ integrated into SD Unet, and what is its role in the framework?

4. DPM and SPM are designed to extract LR information at detail and structure levels, both of which should contribute to fidelity. However, Table 2 suggests that SPM improves fidelity, while window-based cross-attention layers in DPM weaken fidelity. More explanation is required.

questions:
1. The authors need to compare the complexity of ClearSR with that of the other methods, including the model parameter counts, inference time, and inference timestep.

2. The authors should double-check the understanding of PASD in Line 190.

3.  The authors should add a clearer description of the image-level feature $\textbf{p}$ in Figure 3. How is $\textbf{p}$ integrated into SD Unet, and what is its role in the framework?

4. The authors should explain more clearly in Table 2. Why does the SPM improve fidelity, while window-based cross-attention layers in DPM weaken fidelity? In addition, the ablation study that includes a model without DPM should also be provided for a more complete picture of each module's contribution.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730494091178

thread_type:
official_review

<thread_object>
creator: reviewer_wngp

<content>
summary:
This paper proposes a prior-based controlnet-like approach for image super-resolution. The motivation is to refine the conditional feature to improve the fidelity of the SR output while avoid the obvious degradation of generation ability. The proposed approach aims to achieve this goal from both the architecture design by introducing additional modules as well as cross-attention layers and the inference strategy by introducing proper guidance at difference inference steps. There are also some observations to support the design.

strengths:
+ The motivation is clear and there are also some observations to provide the insights for the design of the approach.
+ The evaluation shows reasonable improvement of the proposed approach.
+ The paper is easy to follow.

weaknesses:
- The additional modules introduced in this paper may also increase the cost of training and inference. Some evaluation on the complexity should be provided.
- The proposed Latent Space Adjustment strategy is somewhat tricky. How to choose ideal hyperparameters can be tough and case-by-case. Moreover, when the degradation is severe, adding LR guidance into the inference may leads to blurry outputs.
-Some strong baselines are missing ,e.g., SUPIR.

questions:
My main concerns are as follows:

1. The author claims that ControlNet cannot preserve the LR information well in Figure 2. Is it because that ControlNet adds noise to the LR conditional during training and inference? Does the proposed approach also follows this setting as ControlNet? The authors should explicitly state whether they follow the same noise addition process as ControlNet, and if not, to explain how their approach differs.

2. The additional modules introduced in this paper may also increase the cost of training and inference. Some evaluation on the complexity should be provided, e.g., parameters, flops and inference time. The authors may consider provide some numerical comparison with existing baselines.

3. The proposed Latent Space Adjustment strategy is somewhat tricky. How to choose ideal hyperparameters can be tough and case-by-case. Moreover, when the degradation is severe, adding LR guidance into the inference may leads to blurry outputs. The authors should consider providing guidelines or heuristics for choosing hyperparameters, and discussing how their method performs under severe degradation conditions and the quality of the guidance under such cases.

4. SUPIR has more powerful generative ability then the baselines in the paper. The authors may want to explain why SUPIR was not included as a baseline, or to consider adding it to their comparisons if feasible.

5. Why choosing window cross-attention rather than full-attention and how to decide the window size? The authors should provide empirical or theoretical justification for using window cross-attention, and explain how they determined the optimal window size.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730525233290

thread_type:
official_review

<thread_object>
creator: reviewer_h8y1

<content>
summary:
This paper propose new diffusion-based method, named ClearSR, which can use the LR latent embedding to guide diffusion to generate better results. In particular, the author designs two modules to effectively use the information of LR embedding and propose a adjust strategy to balance the fidelity and detail of SR results.

strengths:
This paper is clear in describing its contributions and methodology. 
The author analyzed the relationship between image fidelity and model generation capabilities, and attempted to propose a solution strategy.
The experimental arrangement is relatively reasonable, and the ablation study can prove the effectiveness of the strategies proposed by the author.

weaknesses:
Some descriptions in the paper may lead to confusion. The authors classify detail information as high-frequency information and structural information as low-frequency information. However, edges can also represent structure and are actually considered high-frequency information. The authors should use more appropriate terminology to avoid ambiguity.

To balance the fidelity and details of results, the author propose Latent Space Adjustment (LSA) strategy. However, the experimental results do not clearly demonstrate that the proposed method performs better in terms of fidelity (PSNR, SSIM, LPIPS, etc). In addition, similar approaches have also appeared in DiffBIR and PASD, and the author should provide a thorough comparison with the strategies proposed by these other methods.

questions:
The motivation is clear. However, there are some concerns regarding the proposed approach. Specifically, the LR latent embedding, which is the output of the VAE encoder, has a size of 4x64x64, while the input image is 3x512x512. Compared to the original image, the LR embedding loses a significant amount of spatial information. Therefore, the LR latent embedding may not be suitable for supplementing detail and structural information. 

Figure 2 shows that the proposed method has a low KL divergence value between the control signal and the low-resolution latent embedding. This suggests that the authors have introduced two modules to achieve a similar distribution between the LR latent embedding and the control signal. So why not use the LR latent embedding directly? Furthermore, from past work (DiffBIR, PASD, SeeSR), we know that the role of the control branch is primarily to remove degradation and bring it closer to the HR distribution. However, the method proposed by the authors results in the distribution of the control branch outputs being closer to the distribution of LR latent embedding, which is puzzling.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes ClearSR for image super-resolution. The main contribution of this work is the better use of the embeddings of the LR input image. Specifically, two modules, DPM and GSPM, are proposed to better encode information from the LR embedding, leading to enhanced output quality. 

This paper is well motivated, highlighting the fidelity problem of existing super-resolution methods. In addition, the proposed method demonstrates decent performance in the evaluation datasets.

Despite the above strengths, the effectiveness of the proposed modules are not fully verified, some of which are shared by the reviewers. In particular, 

1. The theoretical or intuitive explanation of the modules' effectiveness are not convincing. Why does attention help preserving details, and why are convolution blocks good at structure preservation? These claims are not carefully verified.

2. The comparison to `using only LR latent` is missing. This ablation is essential as it is a direct proof of the proposed modules leading to a better restoration. Since existing works have different settings and designs, the comparison with them could not lead to the conclusion that the propose model is effective.

3. The improvements over existing works are not conclusive. As also demonstrated by the authors, CFG, window size, text prompts, and other factors also lead to non-negligible quality differences. Given the inconclusive metrics (e.g., in Table 1), it is hard to claim that the proposed modules lead to positive effects. More ablations are needed to demonstrate the contributions in this work.

Based on the above considerations, the AC would recommend a rejection.