JoReS-Diff: Joint Retinex and Semantic Priors in Diffusion Model for Low-light Image Enhancement

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-20 13:32:10

Abstract: Low-light image enhancement (LLIE) has achieved promising performance by employing conditional diffusion models. Despite the success of some conditional methods, previous methods may neglect the importance of a sufficient formulation of task-specific condition strategy, resulting in suboptimal visual outcomes. In this study, we propose JoReS-Diff, a novel approach that incorporates Retinex- and semantic-based priors as the additional pre-processing condition to regulate the generating capabilities of the diffusion model. We first leverage pre-trained decomposition network to generate the Retinex prior, which is updated with better quality by an adjustment network and integrated into a refinement network to implement Retinex-based conditional generation at both feature- and image-levels. Moreover, the semantic prior is extracted from the input image with an off-the-shelf semantic segmentation model and incorporated through semantic attention layers. By treating Retinex- and semantic-based priors as the condition, JoReS-Diff presents a unique perspective for establishing an diffusion model for LLIE and similar image enhancement tasks. Extensive experiments validate the rationality and superiority of our approach.

Main paper: 
 related work: 
 Low-light Image Enhancement. Numerous works introduce the Retinex theory into deep neural networks [1, 5, 7, 23, 26, 46, 47, 55, 60, 62] . The Retinex-Net [46] is the most inspiring method combining physical model and DNNs. Then, Zhang et al. . proposes KinD [62] and KinD++ [60] to provide more effective solutions. Rather than complex multi-stage training pipeline, Fu et al. . [5] and Cai et al. . [1] explore the possibility of end-to-end frameworks and achieve significant performance improvement. Without Retinex theory, recent works concentrate on directly end-to-end manners [17, 28, 32, 42, 49-51, 58, 64] . LLNet [30] inspires the emergence of end-to-end methods. Primarily, supervised methods show promising capability of enhancement. For reducing color deviation, [22, 48, 52, 63] adopt 3DLUT and histogram to preserve color consistency. In [50, 51] , the SNR-aware prior and the structure-aware features are taken as guidance to produce realistic results. Recently, the Ultra-High-Definition (UHD) images become popular. LLformer [41] and UHDFour [25] are proposed to enhance UHD images and release UHD datasets to promote the following research. Furthermore, unsupervised [7, 16] and zero-shot learning [9, 24] are valuable when training images are limited. Diffusion-Based Image Restoration and Low-Light Image Enhancement. Diffusion models [14, 37] show promising capability in image generation tasks [2, 36] . For solving image processing tasks, diffusion-based methods employ conditional mechanism to incorporate the distorted images as guidance [27] , such as colorization [35] , super-resolution [36] , restoration [4, 20, 31] , and LLIE [15, 40, 43, 55, 56, 65] . To realize efficient diffusion-based LLIE, DiffLL [15] uses wavelet transformation to decrease the input size and a high-frequency restoration module to maintain the details. PyDiff [65] directly down-samples the image in early steps to speed up the sampling process. Considering the characteristics of LLIE task, LLDiffusion [40] and CLE-Diffusion [56] use the color map as extra conditional input to preserve the color information. Furthermore, PyDiff [65] proposes a global corrector to alleviate color degradation. By introducing Retinex theory, Diff-Retinex [55] decomposes the image and utilizes the reflectance and illumination maps as conditional images to form the guidance. Although existing diffusion-based LLIE methods obtain good performance, they are limited by insufficient conditional guidance and produce unsatisfactory color and details. In this paper, we investigate a new condition strategy and propose JoReS-Diff to control the generative diffusion model by forming a Retinex-and semantic-based conditional process, exploring a novel perspective for diffusion-based LLIE and similar image enhancement tasks. 
method: 
 The overview is shown in Fig. 3 . We present the Retinex-and semantic-based condition strategy to explore an effective diffusionbased method for LLIE. We introduce the conditional DDPM in Section 3.1 and present our JoReS-Diff in Section 3.2 and Section 3.3.To deal with image restoration tasks, condition strategies are proposed to develop the conditional DDPM [27] . The conditional DDPM also generates a target image x 0 from a pure noise image x 𝑇 and refines the image through successive iterations. Unlike DDPM, the low-quality images are utilized as conditional inputs in conditional DDPM [36] . Thus, the conditional inference is defined as a reverse Markovian process:EQUATIONwhere 𝛾 𝑡 = 𝑡 𝑖=1 𝛼 𝑖 and 𝛼 1:𝑇 are scale parameters. Due to the absence of high-quality image x 0 in inference, 𝝐 𝜃 estimates noise and approximates x 0 as:EQUATIONTherefore, the mean of 𝑝 𝜃 (x 𝑡 -1 |x 𝑡 , y) can be parameterized by applying the x0 into the posterior distribution [14] :EQUATIONThe training objective is to approximate the precise mean μ𝜃 , which only optimizes the denoising network 𝝐 𝜃 (•). The overall optimization objective can be formulated as:EQUATIONRetinex theory illustrates the assumption that the reflectance and illumination components can describe the original image, and the reflectance map is consistent under various lighting conditions. Low-/normal-light images 𝐼, Î ∈ R 𝑊 ×𝐻 ×3 share the constant R ∈ R 𝑊 ×𝐻 ×3 and consist of diverse 𝐿, L ∈ R 𝑊 ×𝐻 ×1 as:EQUATIONwhere ⊙ denotes the element-wise multiplication. According to multi-scale Retinex [19] , R is the Retinex output by removing the lighting effects in 𝐼 as follows: R = log(𝐼 )log(𝐿) = log(𝐼 )log(G(𝐼 )),EQUATIONwhere G(•) and T (•) denote the convolution with the Gaussian surround function and the linear transformation function. However, although the ideal model maintains color constancy by estimating the reflectance map R, details are easily broken through the removal of illumination. Thus, R is more suitable to guide the color recovery. We introduce the low-light input to preserve the original information and reformulate Eq. ( 6) to model the process as:EQUATIONwhere F (•) denotes the deep network and R acts as an auxiliary guidance. Inspired by the improved Retinex model, we consider the enhancement process from two stages: Retinex-based guidance adjustment and conditional image enhancement. Recalling the conditional DDPM model discussed in Section 3.1, the enhancement task can be performed due to the superiority of the generative model conditioned by low-light images [15, 40] . However, the noise and artifacts in low-light images inevitably mislead the diffusion model and cause detail loss and unsatisfactory color.To mitigate this issue, we utilize the guided enhancement manner in Eq. ( 7) and integrate the Retinex-based priors into the diffusion process. Notably, color constancy can actually be affected by lighting variations and the corrupted 𝑅 may lead to degradation of the enhanced image. Thus, we introduce the adjusted term ĉ = F A (𝑅, 𝐿), including more sufficient Retinex-based guidance and acting as priors in the conditional generation process. Subsequently, we formulate the enhancement model as:EQUATIONwhere 𝝐 𝜃 (•), F R (•), and F A (•) denote UNet, RNet, and ANet in Fig. 3 , respectively. Combining the promising texture generation capability of the diffusion model and the conscious representation with vivid color and detail of Retinex theory, we develop a Retinex-based condition strategy for the conditional DDPM and formulate our JoReS-Diff. Eq. ( 8) represents the denoising and refinement process in one iteration, which can be described as two stages, Retinexbased condition learning and Retinex-conditioned refinement. In the first stage, we obtain the decomposition results from a pretrained DNet and input them into the following ANet to produce Retienx-based conditions, which will be described in Section 3.2.1. Then, the conditions will be introduced into the UNet and RNet to achieve conditional image denoising and refinement as elaborated in Section 3.2.2.Learning. As described in Section 3.2, the basis of conditional image refinement is to learn high-quality conditions. Inspired by Retinex-based methods [46, 62] , we propose the DNet to serve as a pre-processing module and produce initial Retinex-based conditions. Compared to calculating an invariant color map as conditional input [40, 56] , the decomposed maps contain not only color information but also details and illumination, achieving a more sufficient form of conditional inputs. To further exploit the progressively refined images in the diffusion process, DNet inputs x 𝑡 as well and provides competent updated predictions 𝑅 𝑡 and 𝐿 𝑡 for better generation. However, the images in the early steps contain incorrect contents and mislead the decomposition. Therefore, we use ANet to deal with corruption and ameliorate the low-quality conditions in the early steps, achieving approving conditional enhancement during the whole generative process. Thus, we realize the Retinex-based condition learning as:EQUATION) Pre-trained DNet. DNet adopts a lightweight UNet-like network to learn the decomposition mapping. It inputs the images 𝐼 and outputs the reflectance 𝑅 and illumination 𝐿. Inspired by the training strategy in [46] , we utilize the constant reflectance loss, smooth illumination loss, and reconstruction loss to pre-train DNet. More details can be seen in the supplementary materials. Condition Adjustment. Directly using the initial decomposed maps with noise and inaccurate brightness may affect the diffusion process. Thus, we utilize the ANet to adjust the decomposition priors. The ANet employs a similar architecture to the DNet with tiny modifications on input and output layers. It inputs the initial maps c 𝑡 = [𝑅 𝑡 , 𝐿 𝑡 ] and learns to produce the adjusted ĉ𝑡= [𝑅 𝑡 ′ , 𝐿 𝑡 ′ ]from two aspects. We first aim to suppress the noise and calibrate the color in 𝑅 𝑡 by the joint adjustment loss: 10) Then, we propose the joint exposure loss to adjust the suboptimal illumination as follows:L 𝑗𝑎 =∥ 𝑅 𝑡 ′ -R ∥ 1 +𝑆𝑆𝐼𝑀 (𝑅 𝑡 ′ , R). (EQUATIONwhere 𝐻𝑖𝑠𝑡 (•) denotes the L1 loss between the histogram of 𝐿 𝑡 ′ and ground-truth L. Thus, the ANet is trained as: [40, 56, 65] and Retinex decomposition [55] . However, existing condition strategies are insufficient to conduct the favorable iterative enhancement. In our JoReS-Diff, we already learn the Retinex-based conditions as described in Section 3.2.1. Then, following the usage of color map [40, 56] , we first control the generation capability of UNet by leveraging ĉ𝑡 as extra inputs, while the carefully prepared conditions are being underutilized just for denoising. Thus, as illustrated in Eq. ( 8), we apply a post-processing refinement network (RNet) to fully exploit the conditions and guarantee the consistent color and contents of x′ 0 . Different from [55] , RNet adopts a lightweight architecture and implements Retinex-conditioned refinement at both feature-and image-levels by FRCM and IRCM. Overall, the refinement stage can be elaborated as: EQUATIONEQUATIONEQUATION)Conditional Refinement. As shown in Fig. 3 , RNet consists of two Retinex-conditioned modules to realize the color recovery and detail enhancement of the approximated x0 . Specifically, we first implement the Retinex-conditioned refinement in feature space, since 𝐹 𝑡 incorporates rich information of adjustment mapping as illustrated in Section 3.2.1. Thus, as depicted in Fig. 4 , we unify the sizes of multi-scale features in 𝐹 𝑡 and model them as affine transformation parameters 𝐹 𝛾 and 𝐹 𝛽 . Then, the feature fusion is carried out by scaling and shifting operations F 𝑇 (•|•) as: to refine the approximated x0 a produce the final output x′ 0 . where 𝑊 denotes the convolution layer and ⊙ is the dot product. However, although the FRCM and conditional layers in UNet integrate the Retinex-based priors into the diffusion process, applying the calculation of the superior Retinex model can further refine the enhanced result. Therefore, as shown in Fig. 4 , we conduct refinement at image-level by introducing 𝑅 𝑡 ′ and 𝐿 𝑡 ′ into IRCM. To simultaneously alleviate the errors resulting from the ideal model and establish a stable training process, we reformulate Eq. ( 6) into a residual refinement manner as:EQUATIONEQUATIONwhere x′ 0 is the final refined output of RNet. In RNet, FRCM and IRCM fully exploit the Retinex-based conditions and produce results with consistent color and details. Accordingly, in addition to Eq. ( 4), we also utilize the constraint of x ′ 0 to optimize the JoReS-Diff as:EQUATIONApart from the capability of color recovery and detail maintenance brought by the Retinex prior, we introduce the semantic prior to further control the generative diffusion model at semantic-level and alleviate the unnatural textures in the output image. Following the semantic-aware framework in [48] , we utilize the same semantic segmentation model to produce available semantic prior from the input image. The segmentation model inputs the low-light image and outputs the intermediate features and semantic map. However, the low-light image remains the distribution gap comparing to the normal-light image, which may cause the misclassification and thus produce unsatisfactory segmentation results. Directly applying the semantic map will mislead the diffusion model in some cases, resulting in more unnatural textures. Therefore, we only use the intermediate latent features since the misclassification can be smoothed in the high-dimensional feature space. Consequently, we collect the multi-scale features from the segmentation model as the semantic prior,c 𝑠𝑒𝑔 = [𝐹 𝑠 0 , 𝐹 𝑠 1 , 𝐹 𝑠 2 ]. The three semantic features (𝐹 𝑏 𝑠 , 𝑏 = 0, 1, 2) with three spatial resolutions (H/2 4-𝑏 , W/2 4-𝑏 ), where H and W are the height and width of the input image.Then, we propose the semantic attention layers to conduct the introduction of the semantic prior, as shown in Fig. 3 . Similar to the Conditional Denoising in Section 3.2.2, we exploit the semantic information to manipulate the feature in the decoder of the UNet as well. Notably, we design the semantic attention layer based on both the self-attention 𝑆𝐴(•) and cross-attention 𝐶𝐴(•). Primarily, selfattention plays a crucial role in preserving the geometric and shape details and the cross-attention contributes more to generate semantic consistency [29] . And we apply learnable weights to achieve an adaptive addition of the outputs from self-and cross-attention branches. The overall calculation can be described as:EQUATION)where 𝜆 𝑆𝐴 , 𝜆 𝐶𝐴 denote learnable weights, 𝐹 𝑖 , 𝐹 𝑜 denote input and output features. Consequently, Eq. ( 4) can be reformulated as:EQUATION)where 𝜆 𝑠 denote loss weights. 
experiments 4.1 experimental settings: 
 Implementation Details. Our method is trained using an NVIDIA RTX A100 GPU with 600k iterations. We use Adam optimizer with the momentum as (0.9, 0.999). The initial learning rate is set to 1×10 -4 and decays by a factor of 0.5 after every 1×10 5 iterations.The batch size and patch size are set to 8 and 256×256. We use the similar architecture of [36] as our denoising U-Net 𝝐 𝜃 . In the inference stage, we follow the DDIM [37] and output by 8 steps.Datasets. We evaluate our method on several datasets. The LOL dataset [46] includes 485 low/normal-light training pairs and 15 testing pairs. The LOL-v2 dataset [54] has two parts, the real part is an extension of LOL with 689/100 pairs and the synthetic part includes 900/100 pairs for training and testing. The UHD-LL dataset [25] includes 2000 training and 150 testing pairs. The ISTD dataset [39] includes 1330 training and 540 testing triplets. The MIT-Adobe FiveK dataset [38] includes 4500 training and 500 testing pairs. Metrics. We employ peak signal-to-noise ratio (PSNR), structural similarity (SSIM) [44] , perceptual image patch similarity (LPIPS) [59] , and Fréchet Inception Distance (FID) [13] for evaluation. Compared Methods. We compare with a rich collection of stateof-the-art LLIE methods, including LIME [12] , RetinexNet [46] , KinD [62] , DRBN [53] , Zero-DCE [9] , EnlightGAN [16] , MIRNet [58] , SNR [50] , PairLIE [7] , SMG [51] , FourLLIE [25] , Retinexformer [1] , UHDFour [25] and diffusion-based LLIE methods including Diff-Retinex [55] , DiffLL [15] , and PyDiff [65] . For the shadow removal and exposure adjustment tasks, we select DC-ShadowNet [18] , EM-Net [67] , BMNet [66] , ShadowFormer [10] , LFG-Diff [33] , Uformer [45] ,Restormer [57] , LLFormer [41] .Tables 1 and 2 show the comparisons on LOL, LOL-v2, and UHD-LL. It is clear that our JoReS-Diff achieves consistent and significant performance gain over all competing methods. Specifically, our method provides significant improvement of 0.155 dB/0.216 dB on LOL/LOL-v2-real datasets respectively, establishing the new stateof-the-art with PSNR values of 26.491 dB/24.836 dB. Furthermore, our method achieves similar performance in terms of SSIM, yielding the best values of 0.876/0.897. On the LOL-v2-synthetic test set, our JoReS-Diff improves the PSNR by 0.042 than the second-best method. On UHD-LL, we also achieve considerable performanceTable 1 : Quantitative comparison on the LOL [46] and LOL-v2-real [54] datasets. ↑ (↓) denotes that, larger (smaller) values suggest better quality. The best results are highlighted in bold and the second best results are in underline (Special fonts are also used in Table 2 ). Note that the absent results ("-") of Diff-Retinex due to the lack of code. As shown in Tables 3 and 4 , we report the results on ISTD [39] and MIT-Adobe-FiveK [38] to evaluate the capability of other image enhancement tasks, i.e. shadow removal and exposure adjustment. We can observe the remarkable performance of our JoReS-Diff from the comparisons. To be specific, our method improves the PSNR/SSIM by 0.25 dB/0.008 than the latest method LFG-Diff, demonstrating the effectiveness of the joint prior in the shadow removal task. On FiveK [38] , JoReS still outperforms previous strong baselines, such as Uformer and Restormer, and provides competitive performance comparing with LLFormer. The slight drop of PSNR may cause by the gap between the PASCAL-Context and FiveK.Methods LOL LOL-v2-real LOL-v2-synthetic PSNR ↑ SSIM ↑ LPIPS ↓ FID ↓ PSNR ↑ SSIM ↑ LPIPS ↓ FID ↓ PSNR ↑ SSIM ↑ LPIPS ↓ FID ↓ LIME [The qualitative evaluations on LOL and LOL-v2 are shown in Figs. 1, 5 and 6. As indicated by the visual comparisons, our JoReS-Diff shows superior enhancement capability and generates images with more pleasing perceptual quality. Specifically, in the first row of Fig. 5 , previous methods fail to reconstruct the detailed textures of the blanket, while our JoReS-Diff provides rich details and mitigates the color gap. As for images of the middle row, although most methods enhance the white regions well, they produce incorrect color and artifacts in the regions with complex scenes. Notably, our method not only restores the print on the carton but preserves the color consistency. Furthermore, the bottom row exhibits that our JoReS-Diff is capable of recovering the vulnerable content vanishing in the results of other methods, which reasonably indicates the superior capacity of restoring naturalistic details. In Fig. 6 , although PyDiff [65] shows a similar overall look to ours, the JoReS-Diff enhances the green cup and the red text with correct color and better saturation. Hence, our JoReS-Diff yields more visually pleasing results as compared to baselines, supporting our method's excellent performance in quantitative evaluation. More results are provided in supplementary material. Additionally, results in Fig. 7 depict that "w/o FRCM" and "UNet-Cond" lead to the reduction of regions with detailed textures, such as uneven surfaces and closely arranged chairs. Although "w/o IRCM" preserves details better, the lack of direct guidance of Retinex components induces the color shift and unnatural illumination. Fig. 8 shows that ANet reduces the noise (top rows) and enhances the illumination (bottom rows) in early steps, and improves the quality of Retinex-based conditions in the whole process. Furthermore, we can see the effects of SSIM and Hist losses by comparing Retinex output at step 8. The reflectance and illumination maps in the green box contain more artifacts and unsatisfactory brightness distribution. Contribution of semantic prior incorporation. As shown in Table 6, we conduct the ablation studies to investigate the contribution of cross-attention and self-attention in the semantic attention layers. By comparing the performance degradation caused by "w/o CA" and "w/o SA", it is conspicuous that the self-attention mainly Step 8Step 6Step 2 5 .benefits on PSNR and SSIM, while the cross-attention can improve LPIPS more. The results reasonably support the motivation of designing the semantic attention layers in Section 3.3. More ablation studies are provided in supplementary material. 
conclusion: 
 In this paper, we propose a diffusion model with joint Retinex and semnatic priors, JoReS-Diff, for image enhancement tasks. JoReS-Diff combines Retinex prior and diffusion model in two stages: Retinex-based condition learning and conditional refinement. In the learning stage, we utilize DNet to obtain initial decomposed maps and then provide robust Retinex prior by ANet. In the refinement stage, Retinex prior is integrated by Retinex attention layers and RCMs in RNet to control the diffusion process and produce better color and details. The semantic prior is provided by an off-theshelf segmentation model and incorporated by semantic attention layers with both self-and cross-attention, preserving semantic consistency of the output. Extensive experiments demonstrate that our JoReS-Diff outperforms state-of-the-art methods on representative benchmarks. We also hope to encourage more works to develop DMs incorporating intrinsic characteristics of low-level vision.  
semantic prior: 
 UNet, Py-Diff [65] , and our JoReS-Diff on LOL-v2 dataset. Previous diffusion-based methods exhibit detail loss and color distortion. Our method properly maintains color constancy and generates realistic textures thanks to the introduction of the superior Retinex and semantic priors. successful attempts have emerged [15, 43, 55, 56, 65] . It is worth noting that sampling efficiency is a common difficulty for the diffusion model. Therefore, Jiang et al. . [15] adopt 2D discrete wavelet transformations and utilize the low-resolution coefficients as conditions for faster inference speed. However, without the explicit modeling of color information, the results are unpleasing as shown in Fig. 1 (a). To better adapt to LLIE task, the color map [40, 56, 65] and Retinex model [55] are introduced into the diffusion process. [40] and [56] try to maintain color consistency, while they only apply an invariant color map and gain limited ability of enhancement.PyDiff [65] proposes a pyramid resolution setting to perform faster reverse process and adjusts the color through a global corrector, which improves the visual quality to a certain extent as shown in Fig. 1(b ). However, the visual results show limitations in preserving color and details. Thus, Diff-Retinex [55] explores the possibility of establishing the Retinex-based diffusion model and produces better color and content. The denoising networks input reflectance and illumination maps as conditional images, and consistent networks are proposed to preserve the content . However, directly applying the decomposed maps as conditions is expected to be imperfect since Retinex theory is essentially an ideal model and the corruptions in reflectance and illumination need to be considered [1] .Although numerous studies have recognized the significance of Diff-Retinex still relies on the original decomposition and multiplication process uses two diffusion models to process the decomposed maps, which are multiplied as output. Our method is end-to-end and uses both Retinex and semantic priors, which are integrated into one single diffusion model. We also propose RNet to fully exploit the Retinex prior. auxiliary guidance in the diffusion process, they fail to explore a suitable strategy and still produce unfavorable visual results. Thus, the condition strategy has the potential for further improvement in establishing an LLIE-specific diffusion model with significant capability of enhancement.To address the aforementioned issues, we propose a novel Retinexand semantic-conditioned diffusion model, JoReS-Diff, specially designed for the LLIE and similar image enhancement tasks. From a new perspective, we combine physical models and semantic guidance as joint priors for the manipulation of the diffusion process. Retinex prior: JoReS-Diff incorporates the Retinex priors as extra pre-processing condition in the diffusion model, which is different from Diff-Retinex [55] as shown in Fig. 2 . Specifically, JoReS-Diff conducts Retinex-based condition learning and conditional image refinement stages. In the former stage, we first utilize a pre-trained network (DNet) to produce initial decomposed maps. The adjustment network (ANet) suppresses noise in reflectance and adjusts exposure in illumination and outputs more reliable Retinex-based condition for better refinement. Then, JoReS-Diff incorporates the decomposed maps into the denoising U-Net (UNet) through Retinex attention layers for better conditional guidance. Furthermore, unlike [55, 65] directly adding a network after denoising process to improve image quality, we propose a refinement network (RNet) to better preserve color and contents based on the Retinex theory, which separates the illumination while maintaining the color and details in the reflectance. We reformulate the Retinex model into a residual manner to alleviate the corruptions and thus design the feature-and image-level Retinex-conditioned modules (F/IRCM) in the RNet. Semantic prior: Although the Retinex prior provides color and detail refinement, the diffusion model still suffers from unnatural textures [61] without considering semantic guidance. Thus, we incorporate the semantic prior for better controlling the generation ability of the diffusion model. To be specific, we extract the semantic prior from the input with an off-the-shelf semantic segmentation model and incorporate it through semantic attention layers like [48] . We both use self-attention and cross-attention for inherent structure preservation and semantic consistency simultaneously [29] . Finally, as shown in Fig. 1 (c), our JoReS-Diff provides the most pleasing result by using the novel joint condition strategy.The main contributions of our work are as follows:• We propose a novel diffusion-based method for image enhancement with joint Retinex and semantic priors, exploring the role of physical models and semantic guidance in controlling the generation capabilities of diffusion models. • We propose condition learning and conditional refinement stages to integrate Retinex prior and preserve color and content consistency. We also introduce semantic prior by semantic attention layers to control the generation ability of diffusion model and reserve semantic consistency.• Extensive experiments on representative benchmarks demonstrate that the joint Retinex and semantic priors and the well-designed interaction mechanism lead to the superior performance of our JoReS-Diff. 
 
 Appendix:  


Decision: This paper was reviewed by four experts in the field. The paper received mixed reviews BA, WR, BA, WA. The reviewers liked the idea of this paper, its thorough evaluation and clear motivation. The reviewers also raised the following concerns on the application scenarios, e.g., the computing complexity, comparisons with other methods and the module design details. Based on the rebuttal, the AC feels that most concerns have been resolved. Based on the reviews, the AC would like to recommend the acceptance of this paper, and suggest the authors to include the added experiments in the rebuttal to the final version.

Reviews: review: paper_summary: paper_summary: This paper proposes an effective low-light enhancement network based on the diffusion model, named JoReS-Diff. Different from previous studies, the proposed method injects the Retines prior obtained by a pre-trained decomposition network and semantic prior obtained by a pre-trained segmentation network into a diffusion model for producing high-quality images. It evaluates the proposed method on several benchmark datasets and demonstrates that the proposed method can achieve the best performance in terms of distortion and perceptual metrics.
strength_weakness: strength_weakness: 1. The paper clearly points out the limitations of the existing methods and describes the motivation of the proposed method.

2. The structure of the paper is well-organized and easy to follow.

3. The proposed method has been evaluated on several benchmark datasets and achieves the best results, which are convincing.
limitations: limitations: 1. The pipeline illustrated in Fig.3 is not consistent with that in Fig.2. Specifically, the input low-light image is the input of the segmentation network and is concatenated with the noise.

2. What are the motivations for designing such network structures of FRCM and IRCM? These structures seem to be heuristic designs.

3. The proposed method leverages a pre-trained segmentation network and a pre-trained Retinex network. How about the model complexity in terms of model parameters, the number of FLOPs, and the computational resources? Regarding the marginal improvements of this method on several datasets, the illustration of the computational complexity is necessary.
paper_summary: paper_summary: The authors propose a low-light enhancement strategy based on the Diffusion model, which achieves good enhancement performance by fusing the retinex decomposition prior and the semantic prior.
strength_weakness: strength_weakness: 1.The authors focus on two difficult issues in low light enhancement: color and missing details, and designs corresponding strategies to improve them.
2.The authors design comprehensive experiments and validate the effectiveness of the method on multiple datasets.
limitations: limitations: 1.Experimental results. For the experimental results of Table 1 on LOL, some diffusion based methods have achieved better results, such as GlobalDiff (NIPS2023,https://paperswithcode.com/sota/low-light-image-enhancement-on-lol), but the authors did not compare them. For visual results, although the results in Figures 1 and 5 have improved compared to previous methods, there are still deviations in details and colors.
2.The role of DNet and ANet: Is it necessary to divide the prior part of the retinex into these two networks? The author explains that pretrained DNet may initially generate poor results, while randomly initialized ANet may not generate good results before training as well. I hope the author can further explain why use ANet.
3. The author seems to have used the wrong name，are ReCoDiff(in fig.5,6,7) and JoResDiff the same method?.
4.How about the computing complexity? The authors are suggested to give the comparison in terms of memory cost, FLOPS, and inference time.
paper_summary: paper_summary: This paper presents two improvements for diffusion models used in low-light image enhancement. Firstly, to enhance the consistency of color and content in low-light image enhancement, the paper proposes conditional generation and conditional refinement modules based on the Retinex theory. Secondly, to address the issue of unnatural textures in diffusion models, the paper employs a semantic model to constrain the generation capabilities of the diffusion model. In the experimental section, the proposed approach is compared with several state-of-the-art methods on multiple datasets, demonstrating superior performance.
strength_weakness: strength_weakness: 1. The paper introduces a diffusion model for low-light enhancement tasks based on Retinex prior and semantic prior. It investigates the role of physical models and semantic guidance in controlling the generation capabilities of the diffusion model.
2. Quantitative experiments were conducted on multiple datasets, comparing with a wide range of state-of-the-art methods, and the proposed approach achieved the best results in the majority of the experiments.
3. In visual comparisons with multiple state-of-the-art methods, the model demonstrated its capability to preserve details and maintain color consistency.
limitations: limitations: 1. In section 4.4, under the Contribution of semantic prior incorporation subsection, there is a lack of ablation studies that include the removal of both Semantic Attention (SA) and Content Attention (CA), as well as the removal of the Segmentation Network from the experiments. The inclusion of such experiments would provide  evidence to demonstrate the role of semantic information in eliminating unnatural textures.
2. In the first occurrence of a symbol, an explanation should be provided, for instance, the “**y**” in Equation 1.
3. The meaning of $\hat R$ in Equation 5 differs from that in Equation 6, with Equation 6 including an additional logarithmic operation compared to Equation 5.
4. The color distinction between the **Retinex Attention Layer** and the **Semantic Attention Layer** in Figure 3 is not clear.
5. In Figure 3, the arrow above circle $x_{t-1}$ at the bottom has an incorrect source; it should be directed from $\hat{x}^{'}_{0}$.
6. The notation for ${\hat c}_t$ in section 3.2.1 is inconsistent, with the initial representation being ${\hat c}_t=[{R}^{'}_t, {L}^{'}_t]$ and the subsequent representation being ${\hat c}_t=[{R}^{'}_t, {L}^{'}_t, {F}_t]$. It is recommended to either unify the notation or use two distinct symbols to differentiate between the two.
paper_summary: paper_summary: JoReS-Diff is an approach for low-light image enhancement (LLIE) utilizing Retinex- and semantic-based priors to regulate diffusion model generation. By integrating Retinex prior through a decomposition and refinement network and incorporating semantic priors via semantic attention layers, our method achieves superior results. Experimental validation confirms the effectiveness of our approach, highlighting the importance of physical models and semantic guidance in LLIE.
strength_weakness: strength_weakness: The strengths of this article include:
1. The paper effectively integrates Retinex prior through a decomposition and refinement network and incorporation of semantic priors via semantic attention layers.
2. The proposed method generates images with more pleasing perceptual quality, preserving color consistency and restoring naturalistic details.
3. The paper includes ablation studies to analyze the contribution of different components in the proposed method. The results demonstrate the effectiveness of the Retinex attention layers, F/IRCMs, and IRCM in improving the performance.
4. This paper presents a well-organized layout of figures and tables, which enhances readability and comprehension.
limitations: limitations: 1. How to verify the semantic consistency？ The author should conduct further analysis on the impact of the segmentation network, such as visualizing the content of the segmentation output and determining whether the enhancement results are consistent with semantics. Or it would be better to use a segmentation task to evaluate the enhanced output.
2. Computational complexity analysis: This work involves the use of multiple components and network structures, which may lead to increased computational complexity and potentially require more computational resources and time for training and inference. The author should provide the computational complexity analysis.