Reti-Diff: Illumination Degradation Image Restoration with Retinex-based Latent Diffusion Model

Authors: Chunming He, Chengyu Fang, Yulun Zhang, Longxiang Tang, Jinfa Huang, Kai Li, Zhenhua Guo, Xiu Li, Sina Farsiu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Illumination degradation image restoration, Latent diffusion model, Retinex theory

Abstract:
Illumination degradation image restoration (IDIR) techniques aim to improve the visibility of degraded images and mitigate the adverse effects of deteriorated illumination. Among these algorithms, diffusion-based models (DM) have shown promising performance but are often burdened by heavy computational demands and pixel misalignment issues when predicting the image-level distribution. To tackle these problems, we propose to leverage DM within a compact latent space to generate concise guidance priors and introduce a novel solution called Reti-Diff for the IDIR task. Specifically, Reti-Diff comprises two significant components: the Retinex-based latent DM (RLDM) and the Retinex-guided transformer (RGformer). RLDM is designed to acquire Retinex knowledge, extracting reflectance and illumination priors to facilitate detailed reconstruction and illumination correction. RGformer subsequently utilizes these compact priors to guide the decomposition of image features into their respective reflectance and illumination components. Following this, RGformer further enhances and consolidates these decomposed features, resulting in the production of refined images with consistent content and robustness to handle complex degradation scenarios. Extensive experiments demonstrate that Reti-Diff outperforms existing methods on three IDIR tasks, as well as downstream applications.

Decision: accept (spotlight)
Reviewer scores: 6, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1729949556710

thread_type:
official_review

<thread_object>
creator: reviewer_72tr

<content>
summary:
This paper focuses on designing a latent diffusion model that utilizes the Retinex priors to address the issue of illumination degradation. To be detailed, the authors developed a module for estimating the Retinex priors from the low-quality and high-quality images and a Transformer backbone for the image enhancement. These components work in synergy to restore images. The authors provide extensive experiments to demonstrate the effectiveness and advancement of their method.

strengths:
1. The coupling of latent diffusion model with the Retinex priors and Transformer is intriguing. And exploring the robustness of network with the Retinex theory in illumination degradation scenarios is meaningful.
2. The proposed method, compared to existing diffusion models in image enhancement, demonstrates relatively high computational efficiency and lower resource consumption while maintaining strong performance.
3. The experimental validation in the paper is thorough, particularly regarding downstream tasks and evaluations across multiple datasets.
4. The paper is well-organized, clearly written, and easy to understand.

weaknesses:
1. The paper utilizes diffusion models to get the latent Retinex priors, but further analysis is needed to explore the relationship between diffusion models and the designed latent Retinex priors. (insights)
2. The mechanisms by which the Retinex priors operate need to be clarified. Ensuring the consistency of the Retinex theory in the latent space is a question worthy of investigation. Further deepening the results-oriented experimental conclusions in the mechanistic level would improve the quality of study.
3. The design of the method results in a significant reliance on the modeling of Retinex theory. However, there seems to be little discussion on the reliability of Retinex decomposition.

questions:
1. What’s the motivation of using the latent diffusion model to generate the Retinex priors? What are its advantages? Would it be more efficient and reliable to use a single Transformer or CNN network for it?
2. DiffIR [1] achieves image restoration using a similar prior. What would be the difference if the entire image enhancement process were simply treated as the prior, compared to using the Retinex-based model as the priors? Demonstrating the necessity of the latent Retinex priors would be more valuable.
3. About the Retinex priors, how does it perform in terms of prior feature errors when exchanging features between the ground truth and low-light images? In addition to the qualitative and quantitative evaluations of the enhancement results, it would be beneficial to conduct an assessment at the prior level as well.
4. It is recommended to perform a convergence analysis of using the latent Retinex priors. For example, showcasing the changes in loss or evaluation metrics on the training and validation sets would be beneficial.
5. Given the introduction of the Retinex priors, does the proposed method heavily depend on the reliability of Retinex decomposition? Since this is an ill-posed problem, it would be helpful to provide an explanation or at least a few qualitative results of the Retinex decomposition to support this claim.
6. How does the method perform in extreme conditions, specifically when the Retinex decomposition fails? (Extremely bright light and strong shadows) 

Although there are some concerns, they do not detract from the fact that this is a relatively well-researched paper. I am inclined to provide a positive opinion in this stage.

[1] Xia, Bin, et al. "Diffir: Efficient diffusion model for image restoration." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711940058

thread_type:
official_review

<thread_object>
creator: reviewer_nsrn

<content>
summary:
This paper introduces the Retinex-based LDM solution, Reti-Diff, for illumination degradation image restoration (IDIR) tasks, aiming to generate visual fidelity results while decreasing computational burdens. In specific, Reti-Diff proposes RLDM to acquire Retinex priors and then leverage RGformer to guide the image restoration process with the extracted priors, resulting in the production of refined images with consistent content and robustness to handle complex degradation scenarios. Extensive experiments demonstrate that Reti-Diff outperforms existing methods on three IDIR tasks, as well as downstream applications.

strengths:
1. This paper proposes RLDM to extract Retinex priors, and RGformer to integrate the priors, ensuring robustness and generalization in complex illumination degradation scenarios.
2. This paper is well organized.
3. Experiments on four IDIR tasks verify the superiority, efficiency, and generalizability of the method and demonstrate that Retinex priors can serve as a plug-and-play strategy to improve the quality of existing methods.

weaknesses:
1. More related works are expected to be discussed, for example, low-light image enhancement via clip-fourier guided wavelet diffusion[1]. 
2. Why do the authors choose to use a cross-attention mechanism to model the Retinex theory?
3. The authors are encouraged to highlight the motivation for extracting Retinex priors and explain why using a diffusion model to extract the priors. 
4. Additionally, in the experiment, the authors are expected to verify if the Retinex priors perform better than the sole RGB prior and if the diffusion model can serve as a better predictor than the common network structure.
5. In Phase 2, the authors first train the RLDM before conducting joint training with RGFormer. In the ablation study, the authors provide results without the joint training stage, but they do not compare the results of removing the independent RLDM training stage. Would it yield better results if RLDM, RPE, and RGFormer were trained directly together?
6. Some typo errors are expected to be fixed, for example, ZR and ZL in Line 90 in the Supp.

[1] Xue M, He J, He Y, et al. Low-light image enhancement via clip-fourier guided wavelet diffusion[J]. arXiv preprint arXiv:2401.03788, 2024.

questions:
Can the authors analyze the reason for the failure cases? Will this be because of the failure of one of the Retinex priors or because of the failure of RGformer to integrate the priors?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730745541581

thread_type:
official_review

<thread_object>
creator: reviewer_uvsx

<content>
summary:
The paper introduces Reti-Diff, a novel latent diffusion model-based framework tailored for Illumination Degradation Image Restoration (IDIR). It leverages Retinex theory to improve image quality by using two primary components: the Retinex-based Latent Diffusion Model (RLDM) and the Retinex-guided Transformer (RGformer).

strengths:
1. The paper combined the Retinex model and data-driven methods since the first effort of RetinexNet. 
2. Experiments are sufficient.
3. Code is a plus.

weaknesses:
1. For the low-light enhancement task, I want to see more results on LIME, NPE, MEF, DICM and VV. Since these datasets do not have ground truth, then is more fair to justify the effectiveness. Any visual results?

2. Missing citations of real-world low-light enhancement methods,
[1] Unsupervised Night Image Enhancement: When Layer Decomposition Meets Light-Effects Suppression
[2] Enhancing Visibility in Nighttime Haze Images Using Guided APSF and Gradient Adaptive Convolution
 
3. LDM is slow in inference. 
The paper lacks a discussion on the computational complexity and runtime efficiency of the proposed model.

questions:
1. How does the proposed method handle edge cases such as extreme noise or heavy color distortions, which may not follow typical low-light degradation patterns?

2. Is there a recommended strategy for tuning the noise variance parameters in the latent diffusion process for optimal performance across varied image qualities?

3. How does Reti-Diff perform in scenarios with extreme lighting conditions, such as overexposed or underexposed images, which may challenge the reliability of Retinex-based priors?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730982121851

thread_type:
official_review

<thread_object>
creator: reviewer_p8wx

<content>
summary:
The paper proposes to use diffusion models in conjunction with a transformer model to perform image restoration of poorly lit images. The proposed method is evaluated in 4 different settings and shown the merits.

strengths:
+ The paper looks at an important problem of illumination restoration in images which is quite relevant for downstream tasks.

weaknesses:
- The use of intrinsic image decomposition through Retinex is a dated idea. There are several better methods available that have neither been explored nor used. https://www.elenagarces.es/projects/SurveyIntrinsicImages/ 

- The work lacks novelty as it uses diffusion to decompose the image and a transformer to reconstruct it. The method has hardly any contribution. 

- Though the evaluation of the method is done for several conditions, the process is not built correctly with any conviction, which is a downer. 

- The method uses a couple of downstream applications to demonstrate the merits: detection and segmentation. The tasks are very easy to generalize, given the power of deep networks without illumination enhancement. The work could not convince the reader about the merit of this approach.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732043144442

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the rebuttal by authors

comment:
We extend our sincere gratitude to all the reviewers (**R1**-**p8Wx**, **R2**-**uVsX**, **R3**-**NsRN**, and **R4**-**72tr**) for their insightful and considerate reviews, which help us to emphasize the contributions of our approach. We are pleased to hear that the reviewers approved the well-structured presentation of our research (**R3, R4**), the novelty of our work (**R2, R3, R4**), and its commendable performance (**R2, R3, R4**).

We are delighted to see reviewers confirm our contributions to the field of illumination degradation image restoration. These encompass our pioneering LDM-based framework, **Reti-Diff**, the plug-and-play Retinex prior estimator, **RLDM**, and the prior-guided image restoration network, **RGformer**.

In direct response to your thoughtful comments, we have methodically addressed each point in our individual responses, and we provide a summary here:

- We reorganized the paper to enhance clarity, highlighting the motivation and contributions while correcting writing errors.
- We added more qualitative and quantitative experiments, along with efficiency analyses, to further underscore our superior performance.
- We explored the potential of our approach under extremely degraded scenarios, demonstrating the generalizability of Reti-Diff.
- We provided additional ablation studies to validate the robustness and effectiveness of the various modules and strategies within Reti-Diff.

Once again, we deeply appreciate the reviewers' valuable suggestions. We have updated the paper accordingly and will release our code for community use and further study. We sincerely hope to **continue the discussion with the reviewers to ensure all concerns have been fully addressed**. If any aspects of our work remain unclear, we kindly invite further feedback to help us improve.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732078098467

thread_type:
official_comment

<thread_object>
creator: area_chair_m2kb

<content>
comment:
Dear reviewers, do the authors' reply address your concerns? This is a reminder that November 13 to November 26 at 11:59pm AoE: Reviewers and authors can exchange responses with each other as often as they wish. Thanks!
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Summary:
The paper introduces Reti-Diff, a novel framework leveraging Retinex-based Latent Diffusion Models (RLDM) and Retinex-guided Transformers (RGformer) for illumination degradation image restoration (IDIR). The method estimates compact Retinex priors to enhance degraded images, achieving state-of-the-art results on multiple IDIR tasks and downstream applications like detection and segmentation, while maintaining computational efficiency.

(b) Strengths:
Innovative Approach: Combines Retinex theory, latent diffusion models, and transformers for robust illumination restoration.
Comprehensive Experiments: Validates effectiveness across multiple IDIR tasks and challenging conditions.
Efficiency: Achieves high performance with reduced computational cost.
Clear Presentation: Well-organized, easy to understand, and supported by accessible code.
(c) Weaknesses:
Reliance on Retinex Theory: Heavily depends on Retinex decomposition, with limited discussion on its reliability and robustness.
Gaps in Related Work: Lacks recent citations and explanation for design choices like cross-attention and training strategies.
(d) Recommendation:
The paper offers significant contributions to IDIR tasks, demonstrating innovation and strong empirical results. Despite minor concerns, it provides a solid foundation for future research.

Decision: Accept.