Reti-Diff: Illumination Degradation Image Restoration with Retinex-based Latent Diffusion Model

## Abstract

Abstract Illumination degradation image restoration (IDIR) techniques aim to improve the visibility of degraded images and mitigate the adverse effects of deteriorated illumination. Among these algorithms, diffusion model (DM)-based methods have shown promising performance but are often burdened by heavy computational demands and pixel misalignment issues when predicting the image-level distribution. To tackle these problems, we propose to leverage DM within a compact latent space to generate concise guidance priors and introduce a novel solution called Reti-Diff for the IDIR task. Reti-Diff comprises two key components: the Retinex-based latent DM (RLDM) and the Retinex-guided transformer (RGformer). To ensure detailed reconstruction and illumination correction, RLDM is empowered to acquire Retinex knowledge and extract reflectance and illumination priors. These priors are subsequently utilized by RGformer to guide the decomposition of image features into their respective reflectance and illumination components. Following this, RGformer further enhances and consolidates the decomposed features, resulting in the production of refined images with consistent content and robustness to handle complex degradation scenarios. Extensive experiments show that Reti-Diff outperforms existing methods on three IDIR tasks, as well as downstream applications. The code will be released.

###### Keywords:

## 1 Introduction

Illumination degradation image restoration (IDIR) seeks to enhance the visibility and contrast of degraded images while mitigating the adverse effects of deteriorated illumination, *e.g*., indefinite noise and variable color deviation. IDIR has been investigated in various domains, including low-light image enhancement [], underwater image enhancement [], and backlit image enhancement [].
By addressing illumination degradation, the enhanced images are expected to exhibit improved visual quality, making them more suitable for decision-making or subsequent tasks like nighttime object detection and segmentation.

Traditional IDIR approaches [, ] primarily rely on manually crafted enhancement techniques with limited generalization capabilities. Leveraging the robust feature extraction capabilities of convolutional neural networks and transformers, a series of deep learning-based methods [, ] have been proposed and have achieved remarkable success in the IDIR domain. However,
as depicted in Fig. 1,
they still face challenges in complex illumination degradation scenarios due to their constrained generative capacity.

To overcome these challenges, deep generative models, like generative adversarial networks [] and variational autoencoder [],
have gained popularity in the IDIR task for their generative abilities. Recently, the diffusion model (DM) [] has been introduced to the IDIR field for high-quality image restoration.
However, existing DM-based methods, *e.g*., Diff-Retinex [] and GSAD [], apply DM directly to image-level generation, leading to two main challenges: (1) These methods incur high computational costs, as predicting the image-level distribution requires a large number of inference steps. (2) The enhanced results may exhibit pixel misalignment with the original clean image in terms of restored details and local consistency.

To tackle the above problems, we propose introducing the latent diffusion model (LDM) to solve the IDIR problem. By applying DM in the low-dimensional compact latent space, we effectively alleviate the computational burden. Additionally, we incorporate LDM into transformers to prevent pixel misalignment in the generated image, which is often observed in existing deep generative models.
Unlike existing LDM-based methods that solely use the priors extracted from the RGB domain, our method, tailored to the specific characteristics of IDIR tasks, empowers LDMs to extract Retinex information from both the reflectance and illumination domains.
This adaptation allows our method to generate high-fidelity Retinex priors directly from low-quality input images. By doing so, this approach enables us to simultaneously enhance image details using the reflectance prior and correct color distortions with the illumination prior, resulting in visually appealing results with favorable downstream tasks.

With this inspiration, we present Reti-Diff, the first LDM-based solution to tackle the IDIR problem. Reti-Diff, depicted in Fig. 2, consists of two primary components: the Retinex-based LDM (RLDM) and the Retinex-guided transformer (RGformer). Initially, RLDM is employed to generate Retinex priors, which are then integrated into RGformer to produce visually appealing results. To ensure the generation of high-quality priors, we propose a two-phase training approach, wherein Reti-Diff undergoes initial pretraining followed by subsequent RLDM optimization.

Figure: Figure 1: Our Reti-Diff outperforms cutting-edge techniques on three IDIR tasks and the low-light object detection task, where CLIP, Diff-Re, and SNR are short for CLIP-LIT [], Diff-Retinex [], and SNR-Net [].
Refer to caption: x1.png

In phase I, we introduce a Retinex prior extraction (RPE) module to compress the ground-truth image into the highly compact Retinex priors, namely the reflectance prior and the illumination prior. These priors are then sent to RGformer to guide feature decomposition and the generation of reflectance and illumination features. Afterward, RGformer employs the Retinex-guided multi-head cross attention (RG-MCA) and dynamic feature aggregation (DFA) module to refine and aggregate the decomposed features, ultimately producing enhanced images with coherent content and ensuring robustness and generalization in extreme degradation scenarios. In phase II, we train RLDM in reflectance and illumination domains to estimate Retinex priors from the low-quality image, with the constraint of consistency with those extracted by RPE from the ground-truth image. Therefore, the extracted Retinex priors can guide the RGformer in detail enhancement and illumination correction, resulting in visually appealing results with favorable downstream performance.

Our contributions are summarized as follows:

- $\bullet$
We propose a novel DM-based framework, Reti-Diff, for the IDIR task. To the best of our knowledge, this is the first application of the latent diffusion model to tackle the IDIR problem.
- $\bullet$
We propose to let RLDM learn Retinex knowledge and generate high-quality reflectance and illumination priors from the low-quality input, which serve as critical guidance in detail enhancement and illumination correction.
- $\bullet$
We propose RGformer to integrate extracted Retinex priors to decompose features into reflectance and illumination components and then utilize RG-MCA and DFA to refine and aggregate the decomposed features, ensuring robustness and generalization in complex illumination degradation scenarios.
- $\bullet$
Extensive experiments on three IDIR tasks verify our superiority to existing methods in terms of image quality and favorability in downstream applications, including low-light object detection and segmentation.

## 2 Related Works

Illumination Degradation Image Restoration.
Early IDIR methods mainly include three approaches: histogram equalization (HE) [], gamma correction (GC) [], and Retinex theory []. HE-based and GC-based methods focus on directly amplifying the low
contrast regions
but overlook illumination factors. Retinex-based variants [, ] propose the development of priors to constrain the solution space for reflectance and illumination maps. However, these methods still rely on hand-crafted priors, limiting their ability to generalize effectively.
With the rapid development of deep learning, approaches based on CNNs and transformers [, , ] have achieved remarkable success in IDIR. For instance, LLNet [] proposed a sparse denoising structure to enhance illumination and suppress noise. DIE [] integrated Retinex cues into a learning-based structure, presenting a one-stage Retinex-based solution for color correction. To enhance generative capacity, Diff-Retinex [] and GSAD [] introduced DM to the IDIR field by directly applying it to image-level generation. However, they entail significant computational costs and may lead to pixel misalignment with the original input, particularly concerning restored image details and local consistency.

Diffusion Models.
Diffusion models (DMs) have demonstrated considerable success in various domains, including density estimation [] and data generation []. Such a probabilistic generative model adopts a parameterized Markov chain to optimize the lower variational bound on the likelihood function, enabling them to generate target distributions with greater accuracy than other generative models, *i.e*., GAN and VAE. Recently, DMs have been introduced to solve the IDIR problem [, ]. However, when directly applied to image-level generation, these approaches introduce computational burdens and pixel misalignment. To overcome this, we propose employing LDM to estimate priors within a low-dimensional latent space. We then integrate these priors into the transformer-based framework, thus addressing the above problems. Besides, unlike existing LDM-based methods [, ] that solely rely on priors extracted from the RGB domain, our method, tailored to the specific characteristics of IDIR tasks, empowers LDMs to extract Retinex information from both the reflectance and illumination domains.
This adaptation allows our method to generate high-fidelity Retinex priors directly from low-quality input images. By doing so, this novel approach enables us to simultaneously enhance image details using the reflectance prior and correct color distortions with the illumination prior, resulting in visually appealing results with favorable downstream tasks.

## 3 Methodology

In this paper, we propose Reti-Diff, the pioneering method based on Latent Diffusion Models (LDM) for IDIR tasks. Reti-Diff is specifically tailored to address the challenges inherent in IDIR tasks by leveraging Retinex priors extracted from both the illumination and reflectance domains to guide the restoration process. This innovative approach utilizes the extracted Retinex prior representation as dynamic modulation parameters, facilitating simultaneous enhancement of restoration details through the reflectance prior and correction of color distortion via the illumination prior. This ensures the generation of visually compelling results while positively impacting downstream tasks.

Figure: Figure 2: Framework of Reti-Diff. We first pretrain Reti-Diff to ensure the robust learning of RLDM and then optimize RLDM to generate high-quality Retinex priors, which guide RGformer in detail enhancement and illumination correction. In (a), we omit the auxiliary decoder $D_{a}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}} }}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{ \vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{ \scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}})$ for simplicity. In (c), we only give the example by using RLDM to extract the reflectance prior and the illumination prior can be extracted similarly.
Refer to caption: x5.png

As shown in Fig. 2, our Reti-Diff comprises two parts: the Retinex-guided transformer (RGformer) and the Retinex-based latent diffusion model (RLDM).
To ensure the generation of high-quality priors, Reti-Diff undergoes a two-phase training strategy, involving the initial pretraining of Reti-Diff and the subsequent optimization of RLDM. In this section, we provide an in-depth explanation of the two-phase training approach and elucidate the entire restoration process.

### 3.1 Pretrain Reti-Diff

We first pretrain Reti-Diff to encode the clear image, termed ground truth, into compact priors with Retinex prior extraction (RPE) module and use the extracted Retinex priors to guide RGformer for restoration.

Retinex prior extraction module.
Given the low-quality (LQ) image $\mathbf{I}_{LQ}\in\mathbb{R}^{H\times W\times 3}$ and its corresponding ground truth $\mathbf{I}_{GT}\in\mathbb{R}^{H\times W\times 3}$, we initially decompose them into the reflectance image $\mathbf{R}\in\mathbb{R}^{H\times W\times 3}$ and the illumination map $\mathbf{L}\in\mathbb{R}^{H\times W}$ according to Retinex theory:

$$ $\mathbf{I}_{LQ}=\mathbf{R}_{LQ}\odot\mathbf{L}_{LQ},\mathbf{I}_{GT}=\mathbf{R} _{GT}\odot\mathbf{L}_{GT},$ (1) $$

where $\odot$ denotes the Hadamard product. Following Retformer [], We use a pretrained decomposing network $D(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{
\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{
\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox
{0.5}{$\scriptscriptstyle\bullet$}}}}})$ to decompose $\mathbf{I}_{LQ}$ and $\mathbf{I}_{GT}$. Then we concatenate
the corresponding components of ground truth and LQ image and use the RPE module $\text{RPE}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle
\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{
\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{
\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}})$ to encode them into Retinex priors $\mathbf{Z}_{\mathbf{R}}\in\mathbb{R}^{3C^{\prime}}$, $\mathbf{Z}_{\mathbf{L}}\in\mathbb{R}^{C^{\prime}}$:

$$ $\displaystyle\mathbf{Z}_{\mathbf{R}}=\text{RPE}(\text{down}(\text{conca}( \mathbf{R}_{GT},\mathbf{R}_{LQ}))),$ (2) $\displaystyle\mathbf{Z}_{\mathbf{L}}=\text{RPE}(\text{down}(\text{conca}( \mathbf{L}_{GT},\mathbf{L}_{LQ}))),$ $$

where $\text{conca}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle
\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{
\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{
\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}})$ is concatenation and $\text{down}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle
\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{
\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{
\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}})$ is downsampling that is operated by PixelUnshuffle. We then send Retinex priors, $\mathbf{Z}_{\mathbf{R}}$ and $\mathbf{Z}_{\mathbf{L}}$, to RGformer to serve as dynamic modulation parameters for detail restoration and color correction.

Retinex-guided transformer.
RGformer mainly consists of two parts in each block, i.e., Retinex-guided multi-head cross attention (RG-MCA) and dynamic feature aggregation (DFA) module. In RG-MCA, we first split the input feature $\mathbf{F}\in\mathbb{R}^{\tilde{H}\times\tilde{W}\times\tilde{C}}$ into two parts $\mathbf{F}_{1}\in\mathbb{R}^{\tilde{H}\times\tilde{W}\times(3\tilde{C}/4)}$ and $\mathbf{F}_{2}\in\mathbb{R}^{\tilde{H}\times\tilde{W}\times(\tilde{C}/4)}$ along the channel dimension. Afterwards, we integrated $\mathbf{Z}_{\mathbf{R}}$ and $\mathbf{Z}_{\mathbf{L}}$ as the corresponding dynamic modulation parameters to generate reflectance-guided feature $\mathbf{F}_{\mathbf{R}}\in\mathbb{R}^{\tilde{H}\times\tilde{W}\times(3\tilde{C
}/4)}$ and illumination-guided feature $\mathbf{F}_{\mathbf{L}}\in\mathbb{R}^{\tilde{H}\times\tilde{W}\times(\tilde{C}
/4)}$:

$$ $\displaystyle\mathbf{F}_{\mathbf{R}}=\text{Li}_{1}(\mathbf{Z}_{\mathbf{R}}) \odot\text{Norm}(\mathbf{F}_{1})+\text{Li}_{2}(\mathbf{Z}_{\mathbf{R}}),$ (3) $\displaystyle\mathbf{F}_{\mathbf{L}}=\text{Li}_{1}(\mathbf{Z}_{\mathbf{L}}) \odot\text{Norm}(\mathbf{F}_{2})+\text{Li}_{2}(\mathbf{Z}_{\mathbf{L}}),$ $$

where $\text{Norm}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle
\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{
\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{
\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}})$ is layer normalization. $\text{Li}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle
\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{
\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{
\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}})$ means linear layer.
Afterward, we aggregate global spatial information by projecting $\mathbf{F}_{\mathbf{R}}$ into query $\mathbf{Q}=\mathbf{W}_{Q}\mathbf{F}_{\mathbf{R}}$ and key $\mathbf{K}=\mathbf{W}_{K}\mathbf{F}_{\mathbf{L}}$ and transforming $\mathbf{F}_{\mathbf{L}}$ into value $\mathbf{V}=\mathbf{W}_{V}\mathbf{F}_{\mathbf{L}}$, where $\mathbf{W}$ is the combination of a $1\times 1$ point-wise convolution and a $3\times 3$ depth-wise convolution. We then perform cross-attention and get the output feature $\tilde{\mathbf{F}}$:

$$ $\tilde{\mathbf{F}}=\mathbf{F}+\text{SoftMax}\left(\mathbf{Q}\mathbf{K}^{T}/ \sqrt{\tilde{C}}\right)\cdot\mathbf{V}.$ (4) $$

By doing so, RG-MCA introduces explicit guidance to fully exploit Retinex knowledge at the feature level and use cross attention mechanism to implicitly model the Retinex theory and refine the decomposed features, which helps to restore missing details and correct color distortion.

Then we employ DFA for local feature aggregation. Apart from the $1\times 1$ convolution and $3\times 3$ depth-wise convolution used for information fusion, DFA also adopts GELU, termed $\text{GELU}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle
\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{
\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{
\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}})$, to ensure the flexibility of aggregation []. Thus, given $\tilde{\mathbf{F}}$ and $\mathbf{Z}$, where $\mathbf{Z}=\text{conca}(\mathbf{Z}_{\mathbf{R}},\mathbf{Z}_{\mathbf{L}})$, the output feature $\hat{\mathbf{F}}$ is

$$ $\displaystyle\hat{\mathbf{F}}=\tilde{\mathbf{F}}+\text{GELU}(\mathbf{W}_{1} \mathbf{F}^{\prime})\odot\mathbf{W}_{2}\mathbf{F}^{\prime},$ (5) $\displaystyle\mathbf{F}^{\prime}=\text{Li}_{1}(\mathbf{Z})\odot\text{Norm}( \tilde{\mathbf{F}})+\text{Li}_{2}(\mathbf{Z}),$ $$

Optimization. To facilitate the extraction of Retinex priors, the RPE module and RGformer are jointly trained by a reconstruction loss with $L_{1}$ norm $\|\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{
\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{
\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox
{0.5}{$\scriptscriptstyle\bullet$}}}}}\|_{1}$:

$$ $L_{Rec}=\|\mathbf{I}_{GT}-\mathbf{I}_{HQ}\|_{1},$ (6) $$

where $\mathbf{I}_{HQ}$ is the enhanced result.
In addition, to ensure that the separated features within RG-MCA effectively capture reflectance and illumination knowledge, we provide an auxiliary decoder $D_{a}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}
}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{
\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{
\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}})$ with the same structure as that in []. $D_{a}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}
}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{
\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{
\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}})$ takes $\tilde{\mathbf{F}}$ as input and outputs the reconstructed reflectance image $\mathbf{R}_{Re}$ and illumination map $\mathbf{L}_{Re}$. For efficiency, we only apply $D_{a}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}
}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{
\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{
\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}})$ for the first transformer block in encoder to get $\mathbf{R}_{Re}^{I}$ and $\mathbf{L}_{Re}^{I}$ and for the last transformer block in decoder to get $\mathbf{R}_{Re}^{L}$ and $\mathbf{L}_{Re}^{L}$. $D_{a}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}
}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{
\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{
\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}})$ is supervised by a Retinex loss $L_{R}$:

$$ $\displaystyle L_{R}=\|\mathbf{R}_{LQ}-\mathbf{R}_{Re}^{I}\|_{1}+\|\mathbf{L}_{ LQ}-\mathbf{L}_{Re}^{I}\|_{1}+\|\mathbf{R}_{GT}-\mathbf{R}_{Re}^{L}\|_{1}+\| \mathbf{L}_{GT}-\mathbf{L}_{Re}^{L}\|_{1},$ (7) $$

Eq. 7 serves to maintain crucial Retinex information throughout the network.
Hence, the integration of Eq. 7 not only promotes the assimilation of Retinex theory by the split features but also amplifies the overall restoration capacity.

In Phase I, the final loss $L_{P1}$ is defined as follows:

$$ $L_{P1}=L_{Rec}+\lambda_{1}L_{R},$ (8) $$

where $\lambda_{1}$ is a hyperparameter and $\lambda_{1}=1$.

### 3.2 Retinex-based Latent Diffusion Model

In Phase II, we train the RLDM to predict Retinex priors from the low-quality input, which are expected to be consistent with that extracted by RPE from the ground-truth image. Unlike conventional LDMs trained on the RGB domain, we introduce two RLDMs with a Siamese structure and train them on distinct domains: the reflectance domain and the illumination domain. This approach, grounded in Retinex theory, equips our RLDM to generate a more generative reflectance prior $\hat{\mathbf{Z}}_{\mathbf{R}}$ to enhance image details, and a more harmonized illumination prior $\hat{\mathbf{Z}}_{\mathbf{L}}$ for color correction. Note that RLDM is constructed upon the conditional denoising diffusion probabilistic models, with both a forward diffusion process and a reverse denoising process.
To simplify, we provide a detailed derivation for $\hat{\mathbf{Z}}_{\mathbf{R}}$ herein, while that of $\hat{\mathbf{Z}}_{\mathbf{L}}$ can be found in the appendix.

Diffusion process.
In the diffusion process, we first use the pretrained RPE to extract the reflectance prior $\mathbf{Z}_{\mathbf{R}}$, which is treated as the starting point of the forward Markov process, i.e., $\mathbf{Z}_{\mathbf{R}}=\mathbf{Z}_{\mathbf{R}}^{0}$. We then gradually add Gaussian noise to $\mathbf{Z}_{\mathbf{R}}$ by $T$ iterations and each iteration can be defined as:

$$ $q\left(\mathbf{Z}_{\mathbf{R}}^{t}|\mathbf{Z}_{\mathbf{R}}^{t-1}\right)= \mathcal{N}\left(\mathbf{Z}_{\mathbf{R}}^{t};\sqrt{1-\beta^{t}}\mathbf{Z}_{ \mathbf{R}}^{t-1},\beta^{t}\mathbf{I}\right),$ (9) $$

where $t=1,\cdots,T$. $\mathbf{Z}_{\mathbf{R}}^{t}$ denotes the noisy prior at time step $t$, $\beta^{t}$ is the predefined factor that controls the noise variance, and $\mathcal{N}$ is the Gaussian distribution. Following [], Eq. 9 can be simplified as follows:

$$ $q\left(\mathbf{Z}_{\mathbf{R}}^{t}|\mathbf{Z}_{\mathbf{R}}^{0}\right)=\mathcal {N}\left(\mathbf{Z}_{\mathbf{R}}^{t};\sqrt{\bar{\alpha}^{t}}\mathbf{Z}_{ \mathbf{R}}^{0},(1-\bar{\alpha}^{t})\mathbf{I}\right),$ (10) $$

where $\alpha^{t}=1-\beta^{t}$ and $\bar{{\alpha}}^{t}=\prod_{i=1}^{t}\alpha^{i}$.

Reverse process.
In the reverse process, RLDM aims to extract the reflectance prior from pure Gaussian noise. Thus, RLDM samples a Gaussian random noise map $\mathbf{Z}_{\mathbf{R}}^{T}$ and then gradually denoise it to run backward from $\mathbf{Z}_{\mathbf{R}}^{T}$ to $\mathbf{Z}_{\mathbf{R}}^{0}$:

$$ $p\!\left(\mathbf{Z}_{\mathbf{R}}^{t-1}|\mathbf{Z}_{\mathbf{R}}^{t},\mathbf{Z}_ {\mathbf{R}}^{0}\!\right)\!=\!\mathcal{N}\!\left(\mathbf{Z}_{\mathbf{R}}^{t-1} ;\boldsymbol{\mu}^{t}(\mathbf{Z}_{\mathbf{R}}^{t},\mathbf{Z}_{\mathbf{R}}^{0}) ,(\boldsymbol{\sigma}^{t})^{2}\mathbf{I}\!\right),$ (11) $$

where mean $\boldsymbol{\mu}^{t}(\mathbf{Z}_{\mathbf{R}}^{t},\mathbf{Z}_{\mathbf{R}}^{0})=
\frac{1}{\sqrt{\alpha^{t}}}(\mathbf{Z}_{\mathbf{R}}^{t}-\frac{1-\alpha^{t}}{
\sqrt{1-\bar{\alpha}^{t}}}\boldsymbol{\epsilon})$ and variance $(\boldsymbol{\sigma}^{t})^{2}=\frac{1-\bar{\alpha}^{t-1}}{1-\bar{\alpha}^{t}}
\beta^{t}$. $\boldsymbol{\epsilon}$ denotes the noise in $\mathbf{Z}_{\mathbf{R}}^{t}$ and is the only uncertain variable. Following previous practice [], we employ a denoising network $\boldsymbol{\epsilon}_{\theta}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}
{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle
\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{
\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}})$ to estimate $\theta$. To operate in the latent space, we further introduce another RPE module $\widetilde{\text{RPE}}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$
\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle
\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{
\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}})$ to extract the conditional reflectance vector $\mathbf{V}_{\mathbf{R}}\in\mathbb{R}^{3C^{\prime}}$ from the reflectance image $\mathbf{R}_{LQ}$ of the LQ image, i.e., $\mathbf{V}_{\mathbf{R}}=\widetilde{\text{RPE}}(\text{down}(\mathbf{R}_{LQ}))$. Therefore, the denoising network can be represented by $\boldsymbol{\epsilon}_{\theta}\left(\mathbf{Z}_{\mathbf{R}}^{t},\mathbf{V}_{
\mathbf{R}},t\right)$. By setting the variance to $1-\alpha^{t}$, we get

$$ $\small\mathbf{Z}_{\mathbf{R}}^{t-1}\!=\!\frac{1}{\sqrt{\alpha^{t}}}(\mathbf{Z} _{\mathbf{R}}^{t}\!-\!\frac{1-\alpha^{t}}{\sqrt{1-\bar{\alpha}^{t}}} \boldsymbol{\epsilon}_{\theta}(\mathbf{Z}_{\mathbf{R}}^{t},\mathbf{V}_{\mathbf {R}},t))\!+\!\sqrt{1-\alpha^{t}}\boldsymbol{\epsilon}^{t},$ (12) $$

where $\boldsymbol{\epsilon}^{t}\sim\mathcal{N}(0,\mathbf{I})$. By using Eq. 12 for $T$ iterations, we can get the predicted prior $\hat{\mathbf{Z}}_{\mathbf{R}}$ and use it to guide RGformer for image restoration. Because the size of the predicted prior $\hat{\mathbf{Z}}_{\mathbf{R}}\in\mathbb{R}^{3C^{\prime}}$ is much smaller than the original reflectance image $\mathbf{R}_{LQ}\in\mathbb{R}^{H\times W\times C}$, RLDM needs much less iterations than those image-level diffusion models []. Thus, we run the complete $T$ iterations for the prior generation rather than randomly selecting one time step.

Optimization. Given the predicted priors $\hat{\mathbf{Z}}_{\mathbf{R}}$ and $\hat{\mathbf{Z}}_{\mathbf{L}}$, generated by two Siamese RLDMs with specific weights, we propose the diffusion loss to supervise them:

$$ $L_{Dif}=\|\mathbf{Z}_{\mathbf{R}}-\hat{\mathbf{Z}}_{\mathbf{R}}\|_{1}+\| \mathbf{Z}_{\mathbf{L}}-\hat{\mathbf{Z}}_{\mathbf{L}}\|_{1}.$ (13) $$

For restoration quality, we propose joint training RPE, RGformer, and RLDM. Thus, the loss in Phase II is formulated as follows:

$$ $L_{P2}=L_{Dif}+\lambda_{2}L_{Rec}+\lambda_{3}L_{R},$ (14) $$

where $\lambda_{2}$ and $\lambda_{3}$ are two hyper-parameters and are set as 1 in this paper.

### 3.3 Inference

In the inference phase, given the LQ input $\mathbf{I}_{LQ}$, Reti-Diff first uses $\widetilde{\text{RPE}}$ to extract the conditional vectors $\mathbf{V}_{\mathbf{R}}$ and $\mathbf{V}_{\mathbf{L}}$, and then generates predicted Retinex priors $\hat{\mathbf{Z}}_{\mathbf{R}}$ and $\hat{\mathbf{Z}}_{\mathbf{L}}$ with two RLDMs. Under the guidance of the Retinex priors, RGformer generates the restored HQ image $\mathbf{I}_{HQ}$. Benefiting from our Retinex-based diffusion framework, $\mathbf{I}_{HQ}$ enjoys richer texture details and more harmonized illumination, thereby further facilitating downstream tasks.

**Table 1: Results on the low-light image enhancement task. The best two results are in red and blue fonts, respectively.**
|  |  | LOL-v1 | LOL-v2-real | LOL-v2-synthetic | SID |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | Sources | PSNR $\uparrow$ | SSIM $\uparrow$ | FID $\downarrow$ | BIQE $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | FID $\downarrow$ | BIQE $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | FID $\downarrow$ | BIQE $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | FID $\downarrow$ | BIQE $\downarrow$ |
| MIRNet [] | ECCV20 | 24.14 | 0.835 | 71.16 | 47.75 | 20.02 | 0.820 | 82.25 | 41.18 | 21.94 | 0.876 | 40.18 | 36.29 | 20.84 | 0.605 | 81.37 | 40.63 |
| EnGAN [] | TIP21 | 17.48 | 0.656 | 153.98 | 35.82 | 18.23 | 0.617 | 173.28 | 51.06 | 16.57 | 0.734 | 93.66 | 45.59 | 17.23 | 0.543 | 77.52 | 33.47 |
| RUAS [] | CVPR21 | 18.23 | 0.723 | 127.60 | 45.17 | 18.27 | 0.723 | 151.62 | 34.73 | 16.55 | 0.652 | 91.60 | 46.38 | 18.44 | 0.581 | 72.18 | 45.02 |
| IPT [] | CVPR21 | 16.27 | 0.504 | 158.83 | 29.35 | 19.80 | 0.813 | 97.24 | 31.17 | 18.30 | 0.811 | 76.79 | 42.15 | 20.53 | 0.618 | 70.58 | 36.71 |
| URetinex [] | CVPR22 | 21.33 | 0.835 | 85.59 | 30.37 | 20.44 | 0.806 | 76.74 | 28.85 | 24.73 | 0.897 | 33.25 | 33.46 | 22.09 | 0.633 | 71.58 | 38.44 |
| UFormer [] | CVPR22 | 16.36 | 0.771 | 166.69 | 41.06 | 18.82 | 0.771 | 164.41 | 40.36 | 19.66 | 0.871 | 58.69 | 39.75 | 18.54 | 0.577 | 100.14 | 42.13 |
| Restormer [] | CVPR22 | 22.43 | 0.823 | 78.75 | 33.18 | 19.94 | 0.827 | 114.35 | 37.27 | 21.41 | 0.830 | 46.89 | 35.06 | 22.27 | 0.649 | 75.47 | 32.49 |
| SNR-Net [] | CVPR22 | 24.61 | 0.842 | 66.47 | 28.73 | 21.48 | 0.849 | 68.56 | 28.83 | 24.14 | 0.928 | 30.52 | 33.47 | 22.87 | 0.625 | 74.78 | 30.08 |
| SMG [] | CVPR23 | 24.82 | 0.838 | 69.47 | 30.15 | 22.62 | 0.857 | 71.76 | 30.32 | 25.62 | 0.905 | 23.36 | 29.35 | 23.18 | 0.644 | 77.58 | 31.50 |
| PyDiff [] | IJCAI23 | 21.15 | 0.857 | 49.47 | 21.13 | — | — | — | — | — | — | — | — | — | — | — | — |
| Retformer [] | ICCV23 | 25.16 | 0.845 | 72.38 | 26.68 | 22.80 | 0.840 | 79.58 | 34.39 | 25.67 | 0.930 | 22.78 | 30.26 | 24.44 | 0.680 | 82.64 | 35.04 |
| Diff-Retinex [] | ICCV23 | 21.98 | 0.852 | 51.33 | 19.62 | 20.17 | 0.826 | 46.67 | 24.18 | 24.30 | 0.921 | 28.74 | 26.35 | 23.62 | 0.665 | 58.93 | 31.17 |
| MRQ [] | ICCV23 | 25.24 | 0.855 | 53.32 | 22.73 | 22.37 | 0.854 | 68.89 | 33.61 | 25.54 | 0.940 | 20.86 | 25.09 | 24.62 | 0.683 | 61.09 | 27.81 |
| IAGC [] | ICCV23 | 24.53 | 0.842 | 59.73 | 25.50 | 22.20 | 0.863 | 70.34 | 31.70 | 25.58 | 0.941 | 21.38 | 30.32 | 24.80 | 0.688 | 63.72 | 29.53 |
| DiffIR [] | ICCV23 | 23.15 | 0.828 | 70.13 | 26.38 | 21.15 | 0.816 | 72.33 | 29.15 | 24.76 | 0.921 | 28.87 | 27.74 | 23.17 | 0.640 | 78.80 | 30.56 |
| CUE [] | ICCV23 | 21.86 | 0.841 | 69.83 | 27.15 | 21.19 | 0.829 | 67.05 | 28.83 | 24.41 | 0.917 | 31.33 | 33.83 | 23.25 | 0.652 | 77.38 | 28.85 |
| GSAD [] | NIPS23 | 23.23 | 0.852 | 51.64 | 19.96 | 20.19 | 0.847 | 46.77 | 28.85 | 24.22 | 0.927 | 19.24 | 25.76 | — | — | — | — |
| Reti-Diff (Ours) | — | 25.35 | 0.866 | 49.14 | 17.75 | 22.97 | 0.858 | 43.18 | 23.66 | 27.53 | 0.951 | 13.26 | 15.77 | 25.53 | 0.692 | 51.66 | 25.58 |

Figure: Figure 3: Visual results on the low-light image enhancement task.
Refer to caption: x6.png

## 4 Experiment

### 4.1 Experimental Setup

Our Reti-Diff is implemented in PyTorch on four RTX3090TI GPUs and is optimized by Adam with momentum terms $(0.9,0.999)$. In phases I and II, we train the network for 300K iterations and the learning rate is initially set as $2\times 10^{-4}$ and gradually reduced to $1\times 10^{-6}$ with the cosine annealing []. Following [], random rotation and flips are used for augmentation.
Reti-Diff mainly comprises RLDM and RGformer. For RLDM, the channel number $C^{\prime}$ is set as 64.
The total time step $T$ is set to 4 and the hyperparameters $\beta^{1:T}$ linearly increase from $\beta^{1}=0.1$ to $\beta^{T}=0.99$.
RGformer adopts a 4-level cascade encoder-decoder structure. We set the number of transformer blocks, the attention heads, the channel number as $[3,3,3,3]$, $[1,2,4,8]$, $[64,128,256,512]$ from level 1 to 4.

Figure: Figure 4: Visual results on the underwater image enhancement task.
Refer to caption: x7.png

Figure: Figure 5: Visual results on the backlit image enhancement task.
Refer to caption: x8.png

### 4.2 Comparative Evaluation

**Table 2: Results on the underwater image enhancement task.**
|  |  | UIEB | LSUI |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | Sources | PSNR $\uparrow$ | SSIM $\uparrow$ | UCIQE $\uparrow$ | UIQM $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | UCIQE $\uparrow$ | UIQM $\uparrow$ |
| FUGAN [] | IRAL20 | 17.41 | 0.842 | 0.527 | 2.614 | 22.16 | 0.837 | 0.576 | 2.667 |
| EnGAN [] | TIP21 | 17.73 | 0.833 | 0.529 | 2.465 | 19.30 | 0.851 | 0.587 | 2.817 |
| Ucolor [] | TIP21 | 20.78 | 0.868 | 0.537 | 3.049 | 22.91 | 0.886 | 0.594 | 2.735 |
| S-uwnet [] | AAAI21 | 18.28 | 0.855 | 0.544 | 2.942 | 20.89 | 0.875 | 0.582 | 2.746 |
| PUIE [] | ECCV22 | 21.38 | 0.882 | 0.566 | 3.021 | 23.70 | 0.902 | 0.605 | 2.974 |
| U-shape [] | TIP23 | 22.91 | 0.905 | 0.592 | 2.896 | 24.16 | 0.917 | 0.603 | 3.022 |
| PUGAN [] | TIP23 | 23.05 | 0.897 | 0.608 | 2.902 | 25.06 | 0.916 | 0.629 | 3.106 |
| ADP [] | IJCV23 | 22.90 | 0.892 | 0.621 | 3.005 | 24.28 | 0.913 | 0.626 | 3.075 |
| NU2Net [] | AAAI23 | 22.38 | 0.903 | 0.587 | 2.936 | 25.07 | 0.908 | 0.615 | 3.112 |
| Reti-Diff (Ours) | — | 24.12 | 0.910 | 0.631 | 3.088 | 28.10 | 0.929 | 0.646 | 3.208 |

**Table 4: Results on the real-world illumination degradation image restoration task.**
| Methods | Sources | DICM | LIME | MEF | NPE | VV |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PI $\downarrow$ | NIQE $\downarrow$ | PI $\downarrow$ | NIQE $\downarrow$ | PI $\downarrow$ | NIQE $\downarrow$ | PI $\downarrow$ | NIQE $\downarrow$ | PI $\downarrow$ | NIQE $\downarrow$ |  |  |
| EnGAN [] | TIP21 | 4.173 | 4.064 | 3.669 | 4.593 | 4.015 | 4.705 | 3.226 | 3.993 | 3.386 | 4.047 |
| KinD++ [] | IJCV21 | 3.835 | 3.898 | 3.785 | 4.908 | 4.016 | 4.557 | 3.179 | 3.915 | 3.773 | 3.822 |
| SNR-Net [] | CVPR22 | 3.585 | 4.715 | 3.753 | 5.937 | 3.677 | 6.449 | 3.278 | 6.446 | 3.503 | 9.506 |
| DCC-Net [] | CVPR22 | 3.630 | 3.709 | 3.312 | 4.425 | 3.424 | 4.598 | 2.878 | 3.706 | 3.615 | 3.286 |
| UHDFor [] | ICLR23 | 3.684 | 4.575 | 4.124 | 4.430 | 3.813 | 4.231 | 3.135 | 3.867 | 3.319 | 4.330 |
| PairLIE [] | CVPR23 | 3.685 | 4.034 | 3.387 | 4.587 | 4.133 | 4.065 | 3.726 | 4.187 | 3.334 | 3.574 |
| GDP [] | CVPR23 | 3.552 | 4.358 | 4.115 | 4.891 | 3.694 | 4.609 | 3.097 | 4.032 | 3.431 | 4.683 |
| GSAD [] | NIPS23 | — | 3.465 | — | 4.517 | — | 3.815 | — | 3.806 | — | 3.355 |
| Reti-Diff (Ours) | — | 2.351 | 3.255 | 2.837 | 3.693 | 3.308 | 3.792 | 2.599 | 3.384 | 3.341 | 3.000 |

Low-light Image Enhancement. We conduct a comprehensive evaluation on four datasets: LOL-v1 [], LOL-v2-real [], LOL-v2-syn [], and SID []. We adhere to the training manner outlined in []. Our assessment involves four metrics: PSNR, SSIM, FID [], and BIQE [].
Note that larger PSNR and SSIM scores, as well as smaller FID and BIQE scores, denote superior performance. We compare our approach against 17 cutting-edge enhancement techniques and report the results in Tab. 1.
As depicted in Tab. 1, our method emerges as

**Table 5: Efficiency analysis in diffusion model-based methods.**
|  | Diff-Retinex [] | PyDiff [] | GSAD [] | Ours |
| --- | --- | --- | --- | --- |
| Parameter (M) | 56.88 | 97.89 | 17.17 | 26.11 |
| MACs (G) | 396.32 | 459.69 | 1340.63 | 156.55 |
| FPS | 4.25 | 3.63 | 2.33 | 12.27 |

the top performer across all datasets and significantly outperforms the second-best
method (Diff-Retinex) by $13.2\%$. These results underscore the superiority of our Reti-Diff. Fig. 3 presents qualitative results, showcasing our capacity to generate enhanced images with corrected illumination and enhanced texture, even in extremely challenging conditions. In contrast, existing methods struggle to achieve the same level of performance such as the boundaries of power lines, color harmonization of lakes, and detailed textures of wooded areas. Besides, we also compare the efficiency of the diffusion model-based methods. As presented in Tab. 5, despite having the second smallest parameter count, our Reti-Diff has the lowest MACs, highest FPS, and superior performance (see Tab. 1). This efficiency can be attributed to our utilization of the diffusion model within a low-dimensional compact latent space.
For fairness, results from the compared methods are generated by their provided models under the same settings with no GT-mean strategy.

Underwater Image Enhancement. We extend our evaluation to encompass two widely-used underwater image enhancement datasets: UIEB [] and LSUI []. In addition to PSNR and SSIM, we employ two metrics specifically tailored for underwater images, namely UCIQE [] and UIQM [], to assess the performance of the ten enhancement approaches. In all cases, higher values indicate superior performance. The results are presented in Tab. 3. As showcased in Tab. 3, our method achieved the highest overall performance and outperformed the second-best method (PUGAN) by $4.48\%$. A qualitative analysis is presented in Fig. 4, illustrating our method’s ability to correct underwater color aberrations and highlight fine texture details.

Backlit Image Enhancement.
Following CLIP-LIT [], we select the BAID [] dataset for training the network with an image size of $256\times 256$. In addition to PSNR and SSIM, our evaluation incorporates LPIPS [] and FID [] as metrics for evaluation, where lower LPIPS and FID denote superior performance. The evaluation results are reported in Tab. 3. As demonstrated in Tab. 3, our method excels in all metrics and generally outperformed the second-best method (CLIP-LIT) by $6.03\%$. Furthermore, a visual comparison in Fig. 5 provides additional evidence of our superiority in detail reconstruction and color correction.

Real-world Illumination Degradation Image Restoration. We also explore the applicability of our method in real-world IDIR tasks. Following the practice of CIDNet [], we selected five commonly-used real-world datasets, *i.e*., DICM [], LIME [], MEF [], NPE [], and VV(^1^11[https://sites.google.com/site/vonikakis/datasets](https://sites.google.com/site/vonikakis/datasets)), which only have the low-quality images without paired high-quality ground-truth. Therefore, akin to [], we leverage models pretrained on the LOL-v2-syn dataset for inference and select PI [] and NIQE [] as evaluation metrics. In both metrics, lower scores indicate better results. As presented in Tab. 4, our method achieves optimal results and surpasses the second-based method (DCC-Net []) by $13.39\%$. This verifies the generalizability of our Reti-Diff in addressing unknown degradation scenarios. Note that all the methods abandon the GT-mean strategy.

**(a) Break down ablation.**
| Datasets | Metrics | RLDM | RGformer | Train | — |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| w/o RLDM | w/o DFA | w/o RG-MCA | w/o $D_{a}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}<br>}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{<br>\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{<br>\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}})$ | w/o joint | Ours |  |  |
| L-v2-r | PSNR | 21.25 | 22.26 | 21.73 | 22.58 | 22.83 | 22.97 |
| SSIM | 0.822 | 0.840 | 0.840 | 0.847 | 0.853 | 0.858 |  |
| L-v2-s | PSNR | 25.38 | 26.49 | 25.92 | 26.80 | 27.18 | 27.53 |
| SSIM | 0.918 | 0.925 | 0.913 | 0.944 | 0.947 | 0.951 |  |

Figure: Figure 6: Ablation study of the number of iterations in RLDM on LOL-v2-syn.
Refer to caption: x9.png

### 4.3 Ablation Study

We conduct ablation studies on the low-light image enhancement task with the L-v2-r and L-v2-s datasets, which are short for LOL-v2-real and LOL-v2-syn.

Effect of RLDM.
As illustrated in Tab. 5(a), we ablate RLDM by directly removing RLDM or retraining RLDM in the RGB domain, *i.e*., w/o Retinex, rather than in the reflectance and illumination domain (RGformer is guided by one RGB prior instead of the Retinex priors in this time). The two modifications result in significant drops in performance. This outcome underscores the critical role of RLDM in enhancing the restoration process.
Furthermore, to assess the generalizability of RLDM, we conducted additional experiments by replacing our RGformer with two transformer-based frameworks, namely Res (Restormer []) and Ret (Retformer []). Note that the training settings are kept consistent with our Reti-Diff. The results are presented in Tab. 7. Tab. 7 reveals that RLDM significantly improves the performance of both frameworks, where “Gain” is the average gain of PSNR and SSIM. This demonstrates that our RLDM serves as a plug-and-play module with strong generalization capabilities.

Effect of RGformer. We conduct an analysis to assess the impact of our RGformer, and the results are presented in Tab. 5(a). In this study, we systematically removed critical components, such as DFA, RG-MCA, and the auxiliary decoder $D_{a}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}
}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{
\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{
\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}})$, from the model architecture. The outcomes of this ablation study clearly indicate that the performance deteriorates when these components are removed, highlighting their essential role in the system. Additionally, in Tab. 5(a), we conduct an evaluation to affirm the significance of joint training in our approach. This analysis reinforces the importance of the joint training process.

Ablations on iteration number.
The number of iterations in the diffusion
model plays a crucial role in determining the method’s efficiency. To explore this, we conducted experiments with different iteration numbers for Reti-Diff, specifically $T$ values selected from the set $\{1,2,4,8,16,32\}$. We adjusted $\beta^{t}$ as defined in Eq. 9 accordingly. The results in terms of PSNR for different iterations, as shown in Fig. 6, illustrate that Reti-Diff exhibits rapid convergence and generates stable guidance priors with just $4$ iterations. This efficiency is attributed to our application of the diffusion model within the compact latent space.

**Table 8: User study.**
| Methods | L-v1 | L-v2-r | L-v2-s | SID | Mean |
| --- | --- | --- | --- | --- | --- |
| KinD [] | 2.31 | 2.25 | 2.46 | 2.33 | 2.34 |
| EnGAN [] | 2.63 | 1.69 | 2.23 | 1.24 | 1.95 |
| RUAS [] | 3.57 | 3.06 | 3.01 | 2.23 | 2.97 |
| Restormer [] | 3.26 | 3.32 | 3.41 | 2.53 | 3.13 |
| Uretinex [] | 3.82 | 3.98 | 3.70 | 3.28 | 3.70 |
| SNR-Net [] | 3.76 | 4.12 | 3.58 | 3.42 | 3.72 |
| CUE [] | 3.62 | 3.81 | 3.28 | 3.09 | 3.45 |
| Retformer [] | 3.35 | 4.02 | 3.71 | 3.35 | 3.61 |
| Ours | 4.05 | 4.33 | 3.92 | 3.75 | 4.01 |

Figure: Figure 7: Results on the low-light object detection task.
Refer to caption: x11.png

**Table 10: Low-light semantic segmentation, where images are darkened by [].**
| Methods (IoU) | Bicycle | Boat | Bottle | Bus | Car | Cat | Chair | Dog | Horse | People | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline | 43.5 | 36.3 | 48.6 | 70.5 | 67.3 | 46.6 | 11.2 | 42.4 | 56.7 | 57.8 | 48.1 |
| RetinexNet [] | 48.6 | 41.7 | 51.7 | 77.6 | 68.3 | 52.7 | 15.8 | 46.3 | 60.2 | 62.3 | 52.5 |
| KinD [] | 51.3 | 40.2 | 53.2 | 76.8 | 69.4 | 50.8 | 14.6 | 47.3 | 60.3 | 60.9 | 52.5 |
| MIRNet [] | 50.3 | 42.9 | 47.4 | 73.6 | 62.7 | 50.4 | 15.8 | 46.3 | 61.0 | 63.3 | 51.4 |
| RUAS [] | 53.0 | 37.3 | 50.4 | 71.3 | 72.3 | 47.6 | 15.9 | 50.8 | 63.6 | 60.8 | 52.3 |
| Restormer [] | 53.8 | 43.8 | 51.4 | 68.7 | 66.8 | 52.6 | 21.6 | 54.8 | 59.8 | 63.3 | 53.7 |
| SCI [] | 54.5 | 46.3 | 57.2 | 78.4 | 73.3 | 49.1 | 22.8 | 49.0 | 62.1 | 66.9 | 56.0 |
| SNR-Net [] | 57.7 | 48.6 | 59.5 | 81.3 | 74.8 | 50.2 | 24.4 | 50.7 | 64.3 | 68.7 | 58.0 |
| Retformer [] | 50.9 | 47.7 | 58.6 | 77.2 | 68.1 | 53.2 | 17.4 | 52.0 | 61.3 | 71.5 | 55.8 |
| Ours | 59.8 | 51.5 | 62.1 | 85.5 | 76.6 | 57.7 | 28.9 | 56.3 | 66.2 | 73.4 | 61.8 |

Figure: Figure 8: Results on the low-light semantic segmentation task.
Refer to caption: x12.png

### 4.4 User Study and Downstream Tasks

User Study. We conduct a user study to assess the subjective visual perception of low-light image enhancement. In this study, 29 human subjects are invited to assign scores to the enhanced results based on four criteria: (1) The presence of underexposed or overexposed regions. (2) The existence of color distortion. (3) The occurrence of undesired noise or artifacts. (4) The inclusion of essential structural details. Participants rate the results on a scale from 1 (worst) to 5 (best). Each low-light image is presented alongside its enhanced results, with the names of the enhancement methods concealed.
The scores are reported in Tab. 9, where our method receives the highest scores across all four datasets. This highlights our effectiveness in generating visually appealing results.

Low-light Object Detection. The enhanced images are expected to have better downstream performance than the original ones. We first verify this on low-light object detection. Following [], all compared methods are performed on ExDark [] with YOLOv3, which is retrained from scratch with their corresponding enhanced results. As shown in Tab. 9, our Reti-Diff exhibits a substantial advantage over existing methods
and the performance of our method surpasses that of the second-best method, Retformer [], by $4.72\%$,
which verifies
our efficacy in facilitating high-level vision understanding.

Low-light Image Segmentation. We extend our experiment to segmentation tasks, *i.e*., semantic segmentation and concealed object segmentation. Following the practice in detection, we also retrain the segmentor for each method.This means that each method’s enhanced results are segmented by the corresponding segmentor with specific weights. We argue this could better exploit the potential of image enhancement methods as a degraded data restoration module.

For semantic segmentation, following [],
we apply image darkening to samples from the VOC [] dataset according to []. We then employ Mask2Former []
to perform segmentation on the enhanced results of these darkened images.
We select Intersection over Union (IoU) for evaluation, and the results are presented in Tab. 11. As shown in Tab. 11, our method achieves the highest performance across all classes, surpassing the second-best method by
$7.53\%$.

We further venture into concealed object segmentation (COS) on two widely-used datasets, COD10K [] and NC4K [], which represents a challenging segmentation task aimed at delineating objects with inherent background similarity. We also apply image darkening [] and enlist the cutting-edge COS segmentor, FEDER [], to perform segmentation on the enhanced results. We evaluate the results using four metrics: mean absolute error $(M)$, adaptive F-measure $(F_{\beta})$, mean E-measure $(E_{\phi})$, and structure measure $(S_{\alpha})$, which are presented in Tab. 11. As depicted in Tab. 11, our method exhibits superior performance compared to the second-best method, SNR-Net, with a margin of $2.16\%$ on average. Note that it is a notable improvement in COS. Collectively, the exceptional results achieved in these two segmentation tasks substantiate our proficiency in recovering image-level illumination degraded information.

Figure: Figure 10: Failure cases. Our results show blurred texture details in the dashed boxes.
Refer to caption: x14.png

## 5 Discussions

Our Reti-Diff is the first LDM-based solution specifically designed for the IDIR task, setting it apart from existing LDM-based methods applied in other tasks. To illustrate the distinctions, we compare it with a general enhancement method, DiffIR []: (1) Motivation. Reti-Diff targets enhancing details and correcting degraded illumination. Thus, we enable RLDM to learn Retinex knowledge and generate Retinex priors from the low-quality input. We contend that relying solely on priors extracted from the RGB domain struggles to fully represent valuable texture details and correct illumination cues, leading to suboptimal restoration performance. To verify this, we substitute our RLDM for the LDM structure used in DiffIR. In LOL-v2-syn, we observe that the PSNR rises from 24.76 to 26.14 and the SSIM increases from 0.921 to 0.933.
(2) Implementation. Apart from proposing RLDM to extract Retinex priors, we further modify the structure of RGformer to implicitly model the Retinex theory at the feature level and introduce an auxiliary decoder to reconstruct the decomposed Retinex components to the RGB domain.
(3) Performance. As shown in Tab. 1, our Reti-Diff significantly outperforms DiffIR [] by $20.6\%$ on average.

## 6 Limitations and Future Work

As shown in Fig. 10, our Reti-Diff encounters challenges in simultaneously recovering illumination and restoring texture details when the low-quality inputs exhibit severe illumination degradation. This issue persists across existing methods and remains unresolved. We attribute this to the loss of texture information during illumination recovery. To address this limitation in future research, we propose excavating texture priors from other domains, *e.g*., the frequency domain. These priors can complement the reflectance priors extracted from the RGB domain, enhancing the preservation of critical texture features.
Additionally, we consider the use of multimodal data [] to aid in improving image reconstruction performance, such as using infrared images [, , ] to aid in low-light visible image enhancement. Besides, We will explore whether our approach is downstream task-friendly with more segmentation algorithms [, , ].
We also aim to extend our approach to tackle IDIR problems afflicted by other types of degradation, such as haze and motion blur, using some domain adaptation strategies [, ]. These endeavors will further advance the capabilities and applicability of Reti-Diff in real-world scenarios.

## 7 Conclusions

To balance generation capability and computational efficiency,
our approach adopts DM within a compact latent space to generate guidance priors. Specifically, we introduce RLDM to extract Retinex priors, which are subsequently supplied to RGformer for feature decomposition. This process ensures precise detailed reconstruction and effective illumination correction. RGformer then refines and aggregates the decomposed features, enhancing the robustness in handling complex degradation scenarios. Our approach is extensively validated through experiments, establishing the clear superiority of the proposed Reti-Diff.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: See above.
soundness: 3
strengths: + The paper looks at an important problem of illumination restoration in images which is quite relevant for downstream tasks.
summary: The paper proposes to use diffusion models in conjunction with a transformer model to perform image restoration of poorly lit images. The proposed method is evaluated in 4 different settings and shown the merits.
weaknesses: - The use of intrinsic image decomposition through Retinex is a dated idea. There are several better methods available that have neither been explored nor used. https://www.elenagarces.es/projects/SurveyIntrinsicImages/ 

- The work lacks novelty as it uses diffusion to decompose the image and a transformer to reconstruct it. The method has hardly any contribution. 

- Though the evaluation of the method is done for several conditions, the process is not built correctly with any conviction, which is a downer. 

- The method uses a couple of downstream applications to demonstrate the merits: detection and segmentation. The tasks are very easy to generalize, given the power of deep networks without illumination enhancement. The work could not convince the reader about the merit of this approach.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How does the proposed method handle edge cases such as extreme noise or heavy color distortions, which may not follow typical low-light degradation patterns?

2. Is there a recommended strategy for tuning the noise variance parameters in the latent diffusion process for optimal performance across varied image qualities?

3. How does Reti-Diff perform in scenarios with extreme lighting conditions, such as overexposed or underexposed images, which may challenge the reliability of Retinex-based priors?
soundness: 3
strengths: 1. The paper combined the Retinex model and data-driven methods since the first effort of RetinexNet. 
2. Experiments are sufficient.
3. Code is a plus.
summary: The paper introduces Reti-Diff, a novel latent diffusion model-based framework tailored for Illumination Degradation Image Restoration (IDIR). It leverages Retinex theory to improve image quality by using two primary components: the Retinex-based Latent Diffusion Model (RLDM) and the Retinex-guided Transformer (RGformer).
weaknesses: 1. For the low-light enhancement task, I want to see more results on LIME, NPE, MEF, DICM and VV. Since these datasets do not have ground truth, then is more fair to justify the effectiveness. Any visual results?

2. Missing citations of real-world low-light enhancement methods,
[1] Unsupervised Night Image Enhancement: When Layer Decomposition Meets Light-Effects Suppression
[2] Enhancing Visibility in Nighttime Haze Images Using Guided APSF and Gradient Adaptive Convolution
 
3. LDM is slow in inference. 
The paper lacks a discussion on the computational complexity and runtime efficiency of the proposed model.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: Can the authors analyze the reason for the failure cases? Will this be because of the failure of one of the Retinex priors or because of the failure of RGformer to integrate the priors?
soundness: 3
strengths: 1. This paper proposes RLDM to extract Retinex priors, and RGformer to integrate the priors, ensuring robustness and generalization in complex illumination degradation scenarios.
2. This paper is well organized.
3. Experiments on four IDIR tasks verify the superiority, efficiency, and generalizability of the method and demonstrate that Retinex priors can serve as a plug-and-play strategy to improve the quality of existing methods.
summary: This paper introduces the Retinex-based LDM solution, Reti-Diff, for illumination degradation image restoration (IDIR) tasks, aiming to generate visual fidelity results while decreasing computational burdens. In specific, Reti-Diff proposes RLDM to acquire Retinex priors and then leverage RGformer to guide the image restoration process with the extracted priors, resulting in the production of refined images with consistent content and robustness to handle complex degradation scenarios. Extensive experiments demonstrate that Reti-Diff outperforms existing methods on three IDIR tasks, as well as downstream applications.
weaknesses: 1. More related works are expected to be discussed, for example, low-light image enhancement via clip-fourier guided wavelet diffusion[1]. 
2. Why do the authors choose to use a cross-attention mechanism to model the Retinex theory?
3. The authors are encouraged to highlight the motivation for extracting Retinex priors and explain why using a diffusion model to extract the priors. 
4. Additionally, in the experiment, the authors are expected to verify if the Retinex priors perform better than the sole RGB prior and if the diffusion model can serve as a better predictor than the common network structure.
5. In Phase 2, the authors first train the RLDM before conducting joint training with RGFormer. In the ablation study, the authors provide results without the joint training stage, but they do not compare the results of removing the independent RLDM training stage. Would it yield better results if RLDM, RPE, and RGFormer were trained directly together?
6. Some typo errors are expected to be fixed, for example, ZR and ZL in Line 90 in the Supp.

[1] Xue M, He J, He Y, et al. Low-light image enhancement via clip-fourier guided wavelet diffusion[J]. arXiv preprint arXiv:2401.03788, 2024.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. What’s the motivation of using the latent diffusion model to generate the Retinex priors? What are its advantages? Would it be more efficient and reliable to use a single Transformer or CNN network for it?
2. DiffIR [1] achieves image restoration using a similar prior. What would be the difference if the entire image enhancement process were simply treated as the prior, compared to using the Retinex-based model as the priors? Demonstrating the necessity of the latent Retinex priors would be more valuable.
3. About the Retinex priors, how does it perform in terms of prior feature errors when exchanging features between the ground truth and low-light images? In addition to the qualitative and quantitative evaluations of the enhancement results, it would be beneficial to conduct an assessment at the prior level as well.
4. It is recommended to perform a convergence analysis of using the latent Retinex priors. For example, showcasing the changes in loss or evaluation metrics on the training and validation sets would be beneficial.
5. Given the introduction of the Retinex priors, does the proposed method heavily depend on the reliability of Retinex decomposition? Since this is an ill-posed problem, it would be helpful to provide an explanation or at least a few qualitative results of the Retinex decomposition to support this claim.
6. How does the method perform in extreme conditions, specifically when the Retinex decomposition fails? (Extremely bright light and strong shadows) 

Although there are some concerns, they do not detract from the fact that this is a relatively well-researched paper. I am inclined to provide a positive opinion in this stage.

[1] Xia, Bin, et al. "Diffir: Efficient diffusion model for image restoration." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
soundness: 3
strengths: 1. The coupling of latent diffusion model with the Retinex priors and Transformer is intriguing. And exploring the robustness of network with the Retinex theory in illumination degradation scenarios is meaningful.
2. The proposed method, compared to existing diffusion models in image enhancement, demonstrates relatively high computational efficiency and lower resource consumption while maintaining strong performance.
3. The experimental validation in the paper is thorough, particularly regarding downstream tasks and evaluations across multiple datasets.
4. The paper is well-organized, clearly written, and easy to understand.
summary: This paper focuses on designing a latent diffusion model that utilizes the Retinex priors to address the issue of illumination degradation. To be detailed, the authors developed a module for estimating the Retinex priors from the low-quality and high-quality images and a Transformer backbone for the image enhancement. These components work in synergy to restore images. The authors provide extensive experiments to demonstrate the effectiveness and advancement of their method.
weaknesses: 1. The paper utilizes diffusion models to get the latent Retinex priors, but further analysis is needed to explore the relationship between diffusion models and the designed latent Retinex priors. (insights)
2. The mechanisms by which the Retinex priors operate need to be clarified. Ensuring the consistency of the Retinex theory in the latent space is a question worthy of investigation. Further deepening the results-oriented experimental conclusions in the mechanistic level would improve the quality of study.
3. The design of the method results in a significant reliance on the modeling of Retinex theory. However, there seems to be little discussion on the reliability of Retinex decomposition.

Meta-review: (a) Summary:
The paper introduces Reti-Diff, a novel framework leveraging Retinex-based Latent Diffusion Models (RLDM) and Retinex-guided Transformers (RGformer) for illumination degradation image restoration (IDIR). The method estimates compact Retinex priors to enhance degraded images, achieving state-of-the-art results on multiple IDIR tasks and downstream applications like detection and segmentation, while maintaining computational efficiency.

(b) Strengths:
Innovative Approach: Combines Retinex theory, latent diffusion models, and transformers for robust illumination restoration.
Comprehensive Experiments: Validates effectiveness across multiple IDIR tasks and challenging conditions.
Efficiency: Achieves high performance with reduced computational cost.
Clear Presentation: Well-organized, easy to understand, and supported by accessible code.
(c) Weaknesses:
Reliance on Retinex Theory: Heavily depends on Retinex decomposition, with limited discussion on its reliability and robustness.
Gaps in Related Work: Lacks recent citations and explanation for design choices like cross-attention and training strategies.
(d) Recommendation:
The paper offers significant contributions to IDIR tasks, demonstrating innovation and strong empirical results. Despite minor concerns, it provides a solid foundation for future research.

Decision: Accept.

Decision: Accept (Spotlight)