Title: Global Structure-Aware Diffusion Process for Low-light Image Enhancement
Venue: NeurIPS 2023
Accepted: yes
Keywords: Image enhancement

diffusion models
Abstract: This paper studies a diffusion-based framework to address the low-light image enhancement problem. To harness the capabilities of diffusion models, we delve into this intricate process and advocate for the regularization of its inherent ODE-trajectory. To be specific, inspired by the recent research that low curvature ODE-trajectory results in a stable and effective diffusion process, we formulate a curvature regularization term anchored in the intrinsic non-local structures of image data, i.e., global structure-aware regularization, which gradually facilitates the preservation of complicated details and the augmentation of contrast during the diffusion process. This incorporation mitigates the adverse effects of noise and artifacts resulting from the diffusion process, leading to a more precise and flexible enhancement. To additionally promote learning in challenging regions, we introduce an uncertainty-guided regularization technique, which wisely relaxes constraints on the most extreme regions of the image. Experimental evaluations reveal that the proposed diffusion-based framework, complemented by rank-informed regularization, attains distinguished performance in low-light enhancement. The outcomes indicate substantial advancements in image quality, noise suppression, and contrast amplification in comparison with state-of-the-art methods. We believe this innovative approach will stimulate further exploration and advancement in low-light image processing, with potential implications for other applications of diffusion models. The code is publicly available at https://github.com/jinnh/GSAD.

Global Structure-Aware Diffusion Process for Low-Light Image Enhancement

Abstract

This paper studies a diffusion-based framework to address the low-light image enhancement problem. To harness the capabilities of diffusion models, we delve into this intricate process and advocate for the regularization of its inherent ODEtrajectory. To be specific, inspired by the recent research that low curvature ODE-trajectory results in a stable and effective diffusion process, we formulate a curvature regularization term anchored in the intrinsic non-local structures of image data, i.e., global structure-aware regularization, which gradually facilitates the preservation of complicated details and the augmentation of contrast during the diffusion process. This incorporation mitigates the adverse effects of noise and artifacts resulting from the diffusion process, leading to a more precise and flexible enhancement. To additionally promote learning in challenging regions, we introduce an uncertainty-guided regularization technique, which wisely relaxes constraints on the most extreme regions of the image. Experimental evaluations reveal that the proposed diffusion-based framework, complemented by rank-informed regularization, attains distinguished performance in low-light enhancement. The outcomes indicate substantial advancements in image quality, noise suppression, and contrast amplification in comparison with state-of-the-art methods. We believe this innovative approach will stimulate further exploration and advancement in low-light image processing, with potential implications for other applications of diffusion models. The code is publicly available at https://github.com/jinnh/GSAD .

1 Introduction

Low-light image enhancement (LLIE) aims to improve the visibility and contrast of the image captured in poor lighting while preserving the natural-looking details, which contributes to many downstream applications, such as object detection [15, 58] and semantic segmentation [62, 48].

Traditional works employ some techniques, such as histogram equalization [28], Retinex theory [6, 23], and gamma correction [29], to correct the image illumination. In recent years, owing to the powerful representational ability and large collections of data, a considerable number of deep learning-based methods [53, 54, 63, 5, 21, 64, 43, 51, 40, 65] have been presented to significantly improve the performance of low-light enhancement by learning the mapping between low-light and normal-light images. Generally, most of the existing works tend to adopt pixel-wise objective functions to optimize a deterministic relationship. Consequently, such regularization frequently produces suboptimal reconstructions for indeterminate regions and poor local structures, resulting in visibly lower reconstruction quality. Although the adversarial loss might mitigate this issue, these methods necessitate careful training adjustments, which might lead to overfitting on specific features or data distributions and even creating new content or artifacts. Recently, the popular diffusion

Algorithm 1 Forward process

1: Repeat

2: X 0 ∼ q ( X 0 )

3: t ∼ Uniform( { 1 , 2 , ..., T } )

4: ϵ ∼ N (0 , I )

5: Take gradient descent step on √ √

6: ∇ θ ∥ ϵ - ϵ θ ( α t X 0 + 1 - α t ϵ, t ) ∥ 2

7: Until converged

Algorithm 2 Reverse process

1: X T ∼ N (0 , I )

2: For t = T, ..., 1 do

3: z ∼ N (0 , I ) if t > 1 , else z = 0

4: X t - 1 = 1 √ α t ( X t - 1 - α t √ 1 - ¯ α t ϵ θ ( X t , t ))+ σ t z

5: End for

6: Return X 0

denoising probabilistic model (DDPM) [7] has garnered notable interest in low-level vision domains [9, 32], credited with its outstanding capacity in modeling the distribution of image pixels. However, a straightforward implementation of the diffusion model for low-light image enhancement is insufficient to address this issue.

In this paper, we present a novel diffusion-based method to boost the performance of low-light enhancement from the perspective of regularizing the ODE-trajectory. Drawing inspiration from the evidence that suggests the potential of a low-curvature trajectory in yielding superior reconstruction performance [12], we endeavor to modulate the ODE-trajectory curvature. Specifically, we introduce a global structure-aware regularization scheme into the diffusion-based framework by gradually exploiting the intrinsic structure of image data. This innovative constraint promotes structural consistency and content coherence across similar regions, contributing to a more natural and aesthetically pleasing image enhancement while preserving the image's fine details and textures. In addition, we devise an uncertainty-guided regularization by integrating an uncertainty map into the diffusion process, facilitating adaptive modulation of the regularization strength. Experimental results on a comprehensive set of benchmark datasets consistently demonstrate the superior performance of our proposed method compared to state-of-the-art ones. A thorough ablation study emphasizes the significance of both the global structure-aware regularization and uncertainty-guided constraint components in our approach, revealing their synergistic effects on the overall quality of enhancement.

2 Related work

Low-light image enhancement. The early works tackle low-light image enhancement by applying some traditional techniques like histogram equalization [28, 36], gamma correction [29], and Retinex theory [6, 23]. Some researchers also seek to improve the visibility using additional sensors[67, 66, 42, 41, 37, 17]. In recent years, with the advancement of low-light data collection [11, 6, 45, 14, 54] a significant number of deep learning-based methods [2, 60, 8, 50, 53, 54, 63, 5, 21, 64, 43, 51, 40] have been proposed, which greatly improved the restoration quality of traditional methods. For example, Retinex-based deep learning methods [45, 60, 59, 47] employed deep learning to decompose low-light images into two smaller subspaces, i.e., illumination and reflectance maps. Wang et al. [43] proposed a normalizing flow-based low-light image enhancement approach that models the distributions across normally exposed images. Xu et al. [51] incorporated the Signal-to-Noise-Ratio (SNR) prior to achieve spatial-varying low-light image enhancement. Wang et al. [40] proposed a transformer-based low-light enhancement method. We refer readers to [13] for a comprehensive review of this field.

Diffusion-based image restoration. Recently, diffusion-based generative models [34] have delivered astounding results with the advancements in denoising diffusion probabilistic models (DDPM) [7, 24], making them increasingly influential in low-level vision tasks, such as image super-resolution [9, 32], image inpainting [30, 19], image deraining [26], and image deblurring [46]. Saharia et al. [32] employed a denoising diffusion probabilistic model to image super-resolution by conditioning the low-resolution images in the diffusion process. Lugmayr et al. [19] proposed a mask-agnostic approach for image inpainting by leveraging the pre-trained unconditional DDPM as the generative prior. Ozan et al. [26] developed a patch-based diffusion model for weather removal that utilized a mean estimated noise-guided sampling update strategy across overlapping patches. These models are based on a diffusion process that transforms a clean image into a noisy one during training time, and then reverses that Markov Chain to generate new images in the testing phase. Consider a clean image X 0 ∈ R h × w and a noise variable ϵ ∈ R h × w ∼ N (0 , σ 2 I ) . The training steps of a diffusion process act as Algorithm 1.

Reverse trajectory

---XT

Xt-n

W/oglobalstructureregularization

Ours

---Xo

→Effectofourmethod

GT

Reverse trajectory

W/oglobalstructure-awareregularization

Ours

Hybrid

One sampling

Note √ α t X 0 + √ 1 -α t ϵ is a closed-form noisy sample at timestamp t , and ϵ θ ( · , · ) denotes a noise estimation network, which perceives the closed-form samples with timestamps and predicts the inherent noise in such samples. By applying this process in reverse, shown as Algorithm 2, from X T ∼ N (0 , I ) to X 0 , we obtain a generated image X 0 , where X t ∈ R h × w is the intermediate reconstructed sample at timestamp t ∈ [0 , 1 , ..., T ] , α t denotes a scaling scalar, usually parameterized as a linearly decreasing sequence [7, 32], and ¯ α t = t ∏ i =1 α i .

3 Proposed Method

3.1 Problem Statement and Overview

Low-light image enhancement is a crucial research area in the field of image processing and computer vision, owing to the increasing demand for high-quality images captured in various low-light environments. Mathematically, the degradation of an image captured under low-light conditions can be modeled as

where Y ∈ R H × W × 3 is the observed low-light image of dimensions H × W , X 0 ∈ R H × W × 3 is the latent high-quality image under normal-light conditions, S ∈ R H × W × 3 represents the spatially varying illumination map that accounts for the uneven lighting, and N ∈ R H × W is the noise term introduced due to sensor limitations and other factors. Thus, low-light image enhancement, particularly in reconstructing extreme low-light regions, is of paramount challenge due to the scarcity of available content in such images. The intricacy is further intensified by the content-dependent nature of optimal lighting conditions, necessitating a flexible and adaptive solution. Fortunately, diffusion models, celebrated for their remarkable capacity to synthesize images conditioned on ancillary images or textual cues, emerge as a propitious candidate for tackling the low-light image enhancement conundrum.

In this paper, we advocate capitalizing on the potent generalization capabilities inherent in diffusion models to overcome the obstacles associated with low-light image enhancement. By incorporating low-light images ( Y ) as conditioning inputs into the diffusion models, we can obtain a naïve implementation of low-light enhancement diffusion model ( ϵ θ ( Y , X t , ¯ α t ) ). Nonetheless, the performance achieved by these preliminary models remains unsatisfactory. In pursuit of procuring high-quality diffusion processes, various strategies have been presented, including alterations to the network structure [31], the incorporation of data augmentation techniques [38], and the introduction of innovative loss terms [22]. Among them, trajectory rectification emerges as a promising strategy, facilitating a seamless and stable diffusion process through the regularization of a low curvature reverse trajectory

Adaptiveregularizationwithuncertainty

MLP

Noisy level embedding

IIPtOE-PtOE(Y,Xt,αt)Il1

Constructinglearnable

closed-form samples

Eo(Y,Xt,αt)

Xo

Learningimageintrinsicglobalstructure

Singular_Singular

Values

Values

Clustering

Cluster1

Matrix1

Matrix1

Cluster1

Singular_Singular

Values

Xo

MatrixK

MatrixK

ClusterK

ClusterK

TheSameClusterIndex

[12, 18]. Encouraged by current successes, we endeavor to augment this preliminary model from the following two distinct perspectives.

1) Learning low curvature trajectories with global structure-aware regularization. To learn a low curvature reverse trajectory, we propose a simple yet effective way via directly minimizing the gap between a learnable sample on the reverse trajectory and the ground-truth sample. This process is further elucidated through pertinent experimental results presented in Fig. 1. Moreover, it is crucial to acknowledge that images intrinsically encompass analogous textures or patterns distributed across different locations [1, 27, 3, 4, 49, 16, 56]. As a result, cultivating a diffusion process that adeptly captures such global structures is indispensable to the success of low-light image reconstruction. By integrating the matrix rank-based regularization within the diffusion-based framework, we astutely exploit the intrinsic structure of image data, thereby facilitating the preservation of nuanced details and the enhancement of contrast.

Despite the effectiveness of the trajectory regularization, precipitating it during the beginning steps of diffusion models could unintentionally subvert sample diversity and quality, attributable to the pronounced fluctuations in large components. Consequently, in pursuit of enhancing the effectiveness of our model, it becomes imperative to architect a progressively adaptive regularization schedule.

2) Adaptive regularization with uncertainty. Furthermore, most contemporary loss functions treat all pixels equally in the field of low-light enhancement tasks, regardless of whether they exist in conditions of slightly low-light or extremely low-brightness, or whether they are subject to noise-deterioration or are noise-free. Therefore, drawing inspiration from [25], we introduce an uncertainty-guided regularization into the diffusion process to further improve restoration learning in challenging regions.

Those strategic incorporations ensure the precise and resilient recovery of image details in struggling low-light regions while accommodating scene-specific lighting variations. In the following sections, we give technical details of our method.

3.2 Exploring Global Structures via Matrix Rank Modeling

The denoising diffusion probabilistic models exhibit a unique learning scheme, which involves learning the distribution of latent noise during training. This characteristic makes it challenging to regularize network parameters θ for learning global structure-aware representations. To address this

issue, we first construct a learnable closed-form sample of X t -1 . Subsequently, we implement two further steps to enhance the content-aware regularization in those images.

1) Constructing learnable closed-form samples. The training of diffusion models commences with the procurement of a closed-form X t at an arbitrary timestep t , as delineated in Line 6 from Algorithm 1. In the ensuing step, a learnable function ϵ θ ( Y , X t , ¯ α t ) is employed to estimate the latent noise ϵ . Given that the network can successfully learn this noise, it becomes feasible to progressively eliminate the noise, as demonstrated by:

Consequently, throughout the training phase, the regularization is strategically applied to the learnable sample X t -1 in lieu of the closed-form sample X t . This approach seeks to optimize the performance of the model by focusing on regularization where it can most effectively influence the learning process.

2) Non-local patch-based matrix rank modeling. To find global structures of a clean image X 0 , we first patchify it into a set of non-overlapping blocks { x ( i ) 0 } n i =1 , where x ( i ) 0 ∈ R m denotes the vectorization of a small block from X 0 . To capture the intrinsic structures from the sample X 0 , we

(

i

0

n

i

=1

(

i

0

n

i

j

=1

k

j

=1

It is worth noting that our method benefits from advanced clustering algorithms greatly (See Sec. 4.3 for details). Through stacking each cluster of vectors into a matrix, we could get { M n j 0 } k j =1 , where the rank of matrix M n j 0 ∈ R m × n j is an effective measurement of the global structures. Consequently, to regularize structure consistency in X t -1 and X 0 , we then apply the same aggregation to the learnable sample X t -1 , resulting in { M n j t -1 } k j =1 , and regularize

where L s ( M n j t -1 , M n j 0 ) indicates the regularization function, U j t ∈ R m × m , S j t ∈ R m × n j , and V j t ∈ R n j × n j are the outputs of the singular value decomposition of matrix M n j t , and diag ( · ) returns a vector of the matrix main diagonal.

3) Gradually injecting structure-aware regularization. This strategy delicately calibrates the focus on patches throughout different learning steps, thereby ensuring a nuanced, stepwise infusion of structure-aware regularization. This judicious approach not only preserves the heterogeneity of the reconstructed samples but also optimizes the noise suppression process, culminating in a notable improvement in the overall performance of our model. Specifically, the modified regularization function L t s is written as

where κ t denotes an adaptive factor for regularization term L s ( M n j t -1 , M n j 0 ) . This schedule allows for a more adaptive and flexible approach to regularization, enabling the model to capture better the global structures and details within the low-light images.

Analysis of choosing rank-based global structure-aware regularization . To minimize the curvature of ODE-trajectory, we bridge the deviation between the learnable closed-form trajectory samples X t -1 and the GT X 0 by our rank-based global structure-aware regularization. Moreover, it is essential to acknowledge that while conventional pixel-wise regularization terms, including L 1 , L 2 , and SSIM, do offer some degree of enhancement to the original diffusion model, their impact is constrained. This limitation arises from their inadequate ability to fully encapsulate the non-local structures and intricate patterns within an image. Although regularization within the feature domain potentially aids in modeling a given image's structure, such structure is usually confined within a local region due to the kernel perceptual regions. This type of regularization, moreover, cannot explicitly characterize the properties of the structure and is lacking in theoretical guidance. Another significant concern is that regularization features could result in significant fluctuations in the back-propagated gradients, thereby impeding network training. Lastly, the empirical evidence documented in Table 4 corroborates the superiority of our rank-based global structure-aware regularization.

)

{

)

}

then apply the commonly-used clustering algorithm to get

k

clusters of

{

x

}

as

{

x

}

.

3.3 Integrating Uncertainty into Diffusion Process

To integrate uncertainty into the diffusion process, inspired by [25], we introduce an additional convolutional branch after the last layer of U-Net of the diffusion model for generating the uncertainty map. Specifically, we train the uncertainty model µ θ 1 ( · ) with the same step as the denoising diffusion models, like Algorithm 1, except that we optimize the following objective:

where P t := µ θ 1 ( Y , X t , ¯ α t ) denotes a pixel-wise uncertainty map, i.e., P t ∈ R H × W × 3 , ϵ θ ( · ) represents the corresponding function with weights of a denoising diffusion model. Upon completion of the uncertainty estimation model's pretraining phase, we obtain µ ˆ θ 1 , and ϵ ˆ θ . Subsequently, we fix the parameter of µ ˆ θ 1 to generate the uncertainty map P t . This map serves as a flexible weight, calibrating the importance of each pixel such that pixels with higher uncertainty are assigned greater weight. Further details are provided in Algorithm 3. The integration of uncertainty into the diffusion process in this manner incentivizes the diffusion model to intensify its focus on uncertain or challenging regions. This strategic approach yields visually pleasing results, underscoring the value of incorporating uncertainty-aware regularization in the enhancement of low-light images.

Require: Normal and low-light image pairs ( X 0 , Y ) , and a pre-trained models µ θ 1 and ϵ ˆ θ .

1: Initialize parameters of µ θ 1 ( · ) (resp. ϵ θ ( · ) ) from pretrained ˆ θ 1 (resp. ˆ θ ), and freeze ˆ θ 1 .

2: Repeat

3: Sample an image pair ( X 0 , Y ) , ¯ α t ∼ p (¯ α ) and ϵ ∼ N (0 , I ) √ √

4: X t = ¯ α t X 0 + 1 -¯ α t ϵ

5: P t ← µ θ 1 ( Y , X t , ¯ α t )

6: Construct the learnable closed-form sample X t -1 via Eq. (2)

7: Perform a gradient descent step on

8: ∇ θ { λ ∥ P t ⊙ ϵ -P t ⊙ ϵ θ ( Y , X t , ¯ α t ) ∥ 1 + L t s }, where λ is a balancing factor, and L t s refers to Eq. (4)

9: Until converged

10: Return the resulting diffusion model ϵ ˜ θ

4 Experiments

4.1 Experiment Settings

Datasets. We employed seven commonly-used LLIE benchmark datasets for evaluation, including LOLv1 [45], LOLv2 [54], DICM [11], LIME [6], MEF [14], NPE [39], and VV 2 . Specifically, LOLv1 contains 485 low/normal-light image pairs for training and 15 pairs for testing, captured at various exposure times from the real scene. LOLv2 is split into two subsets: LOLv2-real and LOLv2-synthetic. LOLv2-real comprises 689 pairs of low-/normal-light images for training and 100 pairs for testing, collected by adjusting the exposure time and ISO. LOLv2-synthetic was generated by analyzing the illumination distribution of low-light images, consisting of 900 paired images for training and 100 pairs for testing. In accordance with the settings outlined in the recent works [51, 43, 40], we trained and tested our models on LOLv1 and LOLv2 datasets separately. The DICM, LIME, MEF, NPE, and VV datasets contain several unpaired real low-light images, which are used only for testing.

Evaluation metrics. We adopted both full-reference and non-reference image quality evaluation metrics to evaluate various LLIE approaches. For paired data testing, we utilized peak signal-to-noise ratio (PSNR), structural similarity (SSIM) [44], and learned perceptual image patch similarity (LPIPS) [57]. For datasets such as DICM, LIME, MEF, NPE, and VV, where paired data are unavailable, we adopted only the naturalness image quality evaluator (NIQE) [20].

Methods under comparison. We compared our method with a great variety of state-of-the-art LLIE methods, i.e., LIME [6], RetinexNet [45], KinD [60], Zero-DCE [5], DRBN [52], KinD++ [59],

LOLv1

LoLv2-real

LOLv2-syn

Low

KinD

LLFlow

LLFormer

SNR_Aware

Ours

GroundTruth

EnlightenGAN [8], MIRNet [55], LLFlow [43], DCC-Net [61], SNR-Aware [51], and LLFormer [40].

Implementation details. We trained our models for 2M iteration steps with PyTorch. We employed an Adam optimizer [10] with a fixed learning rate of 1 × 10 -4 without weight decay. We applied an exponential moving average with a weight of 0 . 9999 during parameter updates. During training, we utilized a fine-grained diffusion process with T = 500 steps and a linear noise schedule with the two endpoints of 1 × 10 -4 and 2 × 10 -2 . The patch size and batch size were set to 96 and 8 , respectively. The hyper-parameter λ was empirically set to 10 .

4.2 Comparison with State-of-the-Art Methods

Results on LOLv1 and LOLv2. Table 1 shows the quantitative results of different LLIE methods, where it can be observed that our method consistently achieves the best performance over all the compared methods in terms of PSNR, SSIM, and LPIPS metrics. Particularly, the remarkable improvement on LPIPS provides compelling evidence of the superior perceptual quality of our method, as shown in Fig. 3, where our method effectively suppresses artifacts and reveals image details, leading to visually appealing results that are more faithful to the original scene.

DICM

LIME

MEF

NPE

Low

Zero-DCE

RetinexNet

KinD

KinD++

EnlightenGAN

SNR-Aware

Results on DICM, LIME, MEF, NPE, and

VV. We directly applied the model trained on the LOLv2-synthetic dataset to these unpaired real-world datasets. From Table 2, we can observe that our method yields better NIQE scores than all the other competitors, demonstrating its stronger generalization ability. In addition, a visual comparison of enhancement results from various methods, as showcased in Fig. 4, underscores the superior performance of our approach. Evidently, our model adeptly adjusts the illumination conditions, optimally enhancing the visibility in low-light areas while vigilantly circumventing over-exposure. This

successful balance corroborates the superior efficacy of our proposed method.

4.3 Ablation Study

Clustering algorithms. We investigated how the clustering method used for finding similar patches affects the quantitative performance on LOLv1. As shown in Table 3, our method generally benefits from different clustering algorithms. Especially, when employing advanced hierarchical clustering, the PSNR value experiences an improvement of approximately 0.7 dB, compared to the results achieved with the K-Means clustering. This further underscores the prospective utility of our

Input

1

2)

3)

4)

5)

GroundTruth

regularization term in enhancing diffusion models with advanced clustering algorithms. Consequently, we adopted advanced hierarchical clustering in our approach for all subsequent experiments.

Regularization terms between X t -1 and X 0 . We delved into the impact of utilizing varying regularization terms between X t -1 and X 0 on the enhancement of diffusion models within LOLv1. Table 4 offers insights that although conventional pixel-wise regularization terms such as L 1 , L 2 , and SSIM can moderately elevate the performance of the baseline, their enhancements pale in comparison to our rank-based modeling regularization. The primary reason is the inability of these regularization terms to fully encapsulate the nonlocal structures or patterns inherent within an image. These outcomes distinctly highlight the superiority of our rank-based model in capturing global structures.

Non-local patch-based matrix rank modeling &adaptive regularization schedule. We established a baseline by demounting the non-local matrix rank-based representation learning regularization. By comparing rows 2), 3), 4), and 5) in Table 5, it can be observed that this module equipped with our adaptive regularization schedule significantly boosts the enhancement performance. However, this module without adopting the adaptive regularization schedule only achieves minor improvement. Such observations demonstrate the necessity of our adaptive regularization schedule and the effectiveness of non-local rank-based representation learning.

Furthermore, as depicted in Fig. 6, we elaborate

50

W/oglobalstructure-awareregularization

Ours

40

Ground Truth

Singular yalue

W/og

global

30

20

Ours

10

0

2

4

6

8

GT

Index

50

W/oglobalstructure-awareregularization

Ours

-Ground Truth

Singular yalue

40

W/oglobal

30

P20

Ours

10

0

5

10

15

20

25

GT

Index

on the matrix rank across clusters from several samples, thereby offering a clear visualization of the effectiveness of this regularization. Empirical findings affirm that implementation of our proposed regularization notably narrows the gap of singular values between the reconstruction output and the ground truth, thereby substantiating the indispensability of this regularization approach and validating its strategic integration into the diffusion process.

Uncertainty-guided regularization. The effectiveness of this regularization is substantiated through a comparative analysis of rows 1) and 2) of Table 5. Notably, the implementation of this regularization is evidenced to bolster the performance of image enhancement. Such improvement, however, is confined to pixel-wise metrics, e.g., PSNR and SSIM, and does not extend to global visual metrics

Input

LLFlow

SNRAware

Input

LLFlow

SNR_Aware

Ground Truth

LLFormer

Ours

Ground Truth

LLFormer

Ours

such as LPIPS (visualization as Fig. 5). This observation implies that the incorporation of uncertaintybased regularization in isolation yields incremental advancements but fails to induce substantial modifications to the structure of the reconstructed images. The findings, therefore, underscore the necessity of an integrated approach that combines uncertainty-based regularization with global structure-aware regularization to achieve significant improvements in both local pixel-level and global structural aspects of image enhancement.

4.4 Challenging Cases

While the proposed method has demonstrated notable advancements in performance, as showcased in the previous sections, it is important to acknowledge that certain challenging scenarios persist within the domain of low-light enhancement tasks. Fig. 7 illustrates two challenging examples that highlight the intricacy of the task at hand. In these examples, it is evident that certain regions of the input image are under extreme low-light conditions, to the point where any trace of texture becomes indiscernible. This lack of visual information poses a significant challenge, causing both our proposed method and other state-of-the-art techniques to struggle in generating visually compelling details. To tackle these challenges, we could identify a subset of extremely low-light patches and apply robust regularization to them to improve reconstruction performance.

5 Conclusion and Discussion

We have introduced a diffusion model-based approach for enhancing low-light images effectively. The cornerstone of our approach lies in the innovative training strategy of diffusion models from the perspective of minimizing the ODE-trajectory curvature, which involves the global structureaware rank-based regularization on learnable closed-form samples and uncertainty-based learning of latent noises. Recognizing the potential adverse effects of regularization within the early stage of diffusion models, we have thoughtfully devised a progressively adaptive regularization schedule. This strategy ensures a nuanced, step-wise infusion of structure-aware regularization, preserving the inherent structural integrity in the reconstructed samples. As we continue to refine our approach, we anticipate further improvements and look forward to the potential applications and advancements these developments will facilitate in broader fields.

While our work exhibits admirable performance compared with contemporary methods, we acknowledge the inherent computational burden associated with diffusion-based methods-attributable to their iterative noise removal in a reverse Markov chain. Nonetheless, thanks to the latest strides in the field [33, 35], it is anticipated that the diffusion models may soon be capable of delivering accelerated reconstructions through a minimal number of iterative steps.

References

Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , volume 2, pages 60-65, 2005.

Jianrui Cai, Shuhang Gu, and Lei Zhang. Learning a deep single image contrast enhancer from multiexposure images. IEEE Transactions on Image Processing , 27(4):2049-2062, 2018.

Weisheng Dong, Lei Zhang, Guangming Shi, and Xin Li. Nonlocally centralized sparse representation for image restoration. IEEE Transactions on Image Processing , 22(4):1620-1630, 2012.

Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. Weighted nuclear norm minimization with application to image denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2862-2869, 2014.

Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong. Zero-reference deep curve estimation for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1780-1789, 2020.

Xiaojie Guo, Yu Li, and Haibin Ling. Lime: Low-light image enhancement via illumination map estimation. IEEE Transactions on Image Processing , 26(2):982-993, 2016.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of Advances in Neural Information Processing Systems , volume 33, pages 6840-6851, 2020.

Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang. Enlightengan: Deep light enhancement without paired supervision. IEEE Transactions on Image Processing , 30:2340-2349, 2021.

Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Proceedings of Advances in Neural Information Processing Systems , volume 35, pages 23593-23606, 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Chulwoo Lee, Chul Lee, and Chang-Su Kim. Contrast enhancement based on layered difference representation. In Proceedings of the IEEE/CVF Conference on International Conference on Image Processing , pages 965-968, 2012.

Sangyun Lee, Beomsu Kim, and Jong Chul Ye. Minimizing trajectory curvature of ode-based generative models. In Proceedings of International Conference on Machine Learning , 2023.

Chongyi Li, Chunle Guo, Linghao Han, Jun Jiang, Ming-Ming Cheng, Jinwei Gu, and Chen Change Loy. Low-light image and video enhancement using deep learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(12):9396-9416, 2021.

Mading Li, Jiaying Liu, Wenhan Yang, Xiaoyan Sun, and Zongming Guo. Structure-revealing low-light image enhancement via robust retinex model. IEEE Transactions on Image Processing , 27(6):2828-2841, 2018.

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2117-2125, 2017.

Ding Liu, Bihan Wen, Yuchen Fan, Chen Change Loy, and Thomas S Huang. Non-local recurrent network for image restoration. In Proceedings of Advances in Neural Information Processing Systems , volume 31, 2018.

Lin Liu, Junfeng An, Jianzhuang Liu, Shanxin Yuan, Xiangyu Chen, Wengang Zhou, Houqiang Li, Yan Feng Wang, and Qi Tian. Low-light video enhancement with synthetic event guidance. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 1692-1700, 2023.

Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In Proceedings of International Conference on Learning Representations , 2023.

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11461-11471, 2022.

Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making a 'completely' blind image quality analyzer. IEEE Signal Processing Letters , 20(3):209-212, 2012.

Sean Moran, Pierre Marza, Steven McDonagh, Sarah Parisot, and Gregory Slabaugh. Deeplpf: Deep local parametric filters for image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12826-12835, 2020.

Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner. Diffrf: Rendering-guided 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4328-4338, 2023.

Michael K Ng and Wei Wang. A total variation model for retinex. SIAM Journal on Imaging Sciences , 4(1):345-365, 2011.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of International Conference on Machine Learning , pages 8162-8171, 2021.

Qian Ning, Weisheng Dong, Xin Li, Jinjian Wu, and Guangming Shi. Uncertainty-driven loss for single image super-resolution. In Proceedings of Advances in Neural Information Processing Systems , volume 34, pages 16398-16409, 2021.

Ozan Özdenizci and Robert Legenstein. Restoring vision in adverse weather conditions with patch-based denoising diffusion models. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Gabriel Peyré, Sébastien Bougleux, and Laurent Cohen. Non-local regularization of inverse problems. In Proceedings of European Conference on Computer Vision , pages 57-68. Springer, 2008.

Stephen M Pizer. Contrast-limited adaptive histogram equalization: Speed and effectiveness. In Proceedings of the first Conference on Visualization in Biomedical Computing , volume 337, page 1, 1990.

Shanto Rahman, Md Mostafijur Rahman, Mohammad Abdullah-Al-Wadud, Golam Dastegir Al-Quaderi, and Mohammad Shoyaib. An adaptive gamma correction for image enhancement. EURASIP Journal on Image and Video Processing , 2016(1):1-13, 2016.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Dohoon Ryu and Jong Chul Ye. Pyramidal denoising diffusion probabilistic models. arXiv preprint arXiv:2208.01864 , 2022.

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(4):4713-4726, 2022.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512 , 2022.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of International Conference on Machine Learning , pages 2256-2265, 2015.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In Proceedings of International Conference on Machine Learning , 2023.

J Alex Stark. Adaptive image contrast enhancement using generalizations of histogram equalization. IEEE Transactions on image processing , 9(5):889-896, 2000.

Chuanming Tang, Xiao Wang, Ju Huang, Bo Jiang, Lin Zhu, Jianlin Zhang, Yaowei Wang, and Yonghong Tian. Revisiting color-event based tracking: A unified network, dataset, and metric. arXiv preprint arXiv:2211.11010 , 2022.

Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. arXiv preprint arXiv:2302.07944 , 2023.

Shuhang Wang, Jin Zheng, Hai-Miao Hu, and Bo Li. Naturalness preserved enhancement algorithm for non-uniform illumination images. IEEE Transactions on Image Processing , 22(9):3538-3548, 2013.

Tao Wang, Kaihao Zhang, Tianrun Shen, Wenhan Luo, Bjorn Stenger, and Tong Lu. Ultra-high-definition low-light image enhancement: A benchmark and transformer-based method. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 2654-2662, 2023.

Xiao Wang, Jianing Li, Lin Zhu, Zhipeng Zhang, Zhe Chen, Xin Li, Yaowei Wang, Yonghong Tian, and Feng Wu. Visevent: Reliable object tracking via collaboration of frame and event flows. IEEE Transactions on Cybernetics , 2023.

Xiao Wang, Shiao Wang, Chuanming Tang, Lin Zhu, Bo Jiang, Yonghong Tian, and Jin Tang. Event stream-based visual object tracking: A high-resolution benchmark dataset and a novel baseline. arXiv preprint arXiv:2309.14611 , 2023.

Yufei Wang, Renjie Wan, Wenhan Yang, Haoliang Li, Lap-Pui Chau, and Alex Kot. Low-light image enhancement with normalizing flow. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 2604-2612, 2022.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing , 13(4):600-612, 2004.

Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv:1808.04560 , 2018.

Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16293-16303, 2022.

Wenhui Wu, Jian Weng, Pingping Zhang, Xu Wang, Wenhan Yang, and Jianmin Jiang. Uretinex-net: Retinex-based deep unfolding network for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5901-5910, 2022.

Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In Proceedings of Advances in Neural Information Processing Systems , volume 34, pages 12077-12090, 2021.

Jun Xu, Lei Zhang, Wangmeng Zuo, David Zhang, and Xiangchu Feng. Patch group based nonlocal selfsimilarity prior learning for image denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 244-252, 2015.

Ke Xu, Xin Yang, Baocai Yin, and Rynson WH Lau. Learning to restore low-light images via decomposition-and-enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2281-2290, 2020.

Xiaogang Xu, Ruixing Wang, Chi-Wing Fu, and Jiaya Jia. Snr-aware low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1771417724, 2022.

Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3063-3072, 2020.

Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. Band representation-based semisupervised low-light image enhancement: Bridging the gap between signal fidelity and perceptual quality. IEEE Transactions on Image Processing , 30:3461-3473, 2021.

Wenhan Yang, Wenjing Wang, Haofeng Huang, Shiqi Wang, and Jiaying Liu. Sparse gradient regularized deep retinex network for robust low-light image enhancement. IEEE Transactions on Image Processing , 30:2072-2086, 2021.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Learning enriched features for fast image restoration and enhancement. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(2):1934-1948, 2022.

Zhiyuan Zha, Bihan Wen, Xin Yuan, Jiantao Zhou, and Ce Zhu. Image restoration via reconciliation of group sparsity and low-rank models. IEEE Transactions on Image Processing , 30:5223-5238, 2021.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 586-595, 2018.

Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and Stan Z Li. S3fd: Single shot scale-invariant face detector. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 192-201, 2017.

Yonghua Zhang, Xiaojie Guo, Jiayi Ma, Wei Liu, and Jiawan Zhang. Beyond brightening low-light images. International Journal of Computer Vision , 129:1013-1037, 2021.

Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo. Kindling the darkness: A practical low-light image enhancer. In Proceedings of the 27th ACM International Conference on Multimedia , pages 1632-1640, 2019.

Zhao Zhang, Huan Zheng, Richang Hong, Mingliang Xu, Shuicheng Yan, and Meng Wang. Deep color consistent network for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1899-1908, 2022.

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2881-2890, 2017.

Lin Zhao, Shao-Ping Lu, Tao Chen, Zhenglu Yang, and Ariel Shamir. Deep symmetric network for underexposed image enhancement with recurrent attentional learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12075-12084, 2021.

Chuanjun Zheng, Daming Shi, and Wentian Shi. Adaptive unfolding total variation network for low-light image enhancement. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4439-4448, 2021.

Dewei Zhou, Zongxin Yang, and Yi Yang. Pyramid diffusion models for low-light image enhancement. arXiv preprint arXiv:2305.10028 , 2023.

Zhiyu Zhu, Junhui Hou, and Xianqiang Lyu. Learning graph-embedded key-event back-tracing for object tracking in event clouds. Advances in Neural Information Processing Systems , 35:7462-7476, 2022.

Zhiyu Zhu, Junhui Hou, and Dapeng Oliver Wu. Cross-modal orthogonal high-rank augmentation for rgb-event transformer-trackers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22045-22055, 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a low-light image enhancement based on diffusion model. By incorporating global structure-aware regularization, this work achieves more promising performances than existing methods.\n\nSTRENGTHS:\nBy considering the global structure during the diffusion process, this work achieves better performance.\n\nWEAKNESSES:\nDespite its better performance, I am very sure that this work cannot advance this field. It does not solve a problem in a new scenario and it just make incrimental revision on top of established framework. My concerns for this work can be concluded as follows:\n1. The authors claim that a naive implementation of the diffusion model alone is inadequate for effectively resolving the low-light image enhancement, while it does not prove this claim. All baselines considered in the experiments are non-diffusion based approaches. Thus, it is difficult to say whether a naive implementation is enough for the problem solving. \n2. In line 151, the autors claim that the rank of matrix is an effective measurement of the global structures. This is a very important assumptions in this place. This part needs more experiments and justifactions. Or, the authors should provide very solid references for this claim in this place.\n3. Except the matrix ranking, several approachs have been proposed to better represent the global structures including some deep learning features. The authors should compare with them. \n4. Why is it important to use global structure in this place? If U-Net is a substantial structure of this proposed framework, it is able to capture the global structure during its network propagation. Thus, why do this work design an additional module to capture global structure. \n5. I am not sure about the effectiveness of uncertainty in this place. I cannot find Table 5 mentioned in this manuscript. \n6. This work just puts several different image processing modules together without enough insights. It is easy to come up with such an idea. The non-local solutions have been discussed in traditional solutions. I donot know why they choose to combine it with diffusion model. Besides, weighted loss has been considered for a large number of methods. Using it here cannot be considered as a new contribution.\n\nQUESTIONS:\nSee my question above.\n\nLIMITATIONS:\nSee my questions above.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an innovative and efficacious approach to regularization within the domain of diffusion models, with particular utility for low-light image reconstruction. The method ingeniously takes into account the latent global structure of images. To achieve this, an image is first divided into patches, and clustering algorithms are then employed to group analogous patches, where the matrix rank is further utilized to minimize the discrepancy between the intermediate reconstruction and the final output. Besides, uncertainty is used to boost performance.\n\nSTRENGTHS:\n1) The novel perspective opens up a promising avenue for enhancing the performance of diffusion models, representing an important contribution to the field. Also, it may have great potential for other diffusion-based image/video processing tasks. \n2) The introduction of this form of regularization is a groundbreaking shift for diffusion models, distinguishing this work from existing literature.\n3) The experimental validations lend strong support to the efficacy of the proposed regularization term, and the proposed method achieves the current SOTA.\n4) The paper is well-written, and the supplementary material includes the code.\n\nWEAKNESSES:\nSee the detailed questions.\n\nQUESTIONS:\n1) Are other advanced clustering algorithms more effective in enhancing reconstruction performance? This comparison could provide valuable insights into the optimal algorithm selection for this specific task.\n2) For the ablation studies in Table 5 (Figure 5 should be Table 5), the authors are suggested to add two more settings, i.e., remove  (b) and (c) and keep (a); remove (c) and keep (a) and (b), to directly validate the adaptive rank regularization, which is the main contributions. \n3) In Figure 6, please add the input and GT images. It is better to use different symbols to distinguish the learnable closed-form samples and the input of the forward process.\n4) Can other regularization terms (e.g., the L1 loss between X_0 and X_t) improve the original diffusion model?\n5) There is potential for the proposed reconstruction algorithm to benefit diffusion models in tasks beyond the current scope of the study. It would be beneficial for the authors to explore this further, extending the validations to other potential applications. At least the authors can discuss this issue to improve the paper. \n6) There are some grammar errors, e.g., line 163: a  an\n\nLIMITATIONS:\nMaybe a little additional computation consumption during the training phase.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
Aiming at better low-light image enhancement performance with diffusion models, this paper proposes a global structure-aware regularization utilizing the intrinsic non-local structural constituents of image data. An uncertainty map is incorporated into the diffusion model to ease the strict constraints on indeterminate regions. Experimental results demonstrate the effectiveness of the proposed components.

STRENGTHS:
1.	The paper incorporates global structure regularization via matrix rank modeling into the diffusion models, showing significant performance improvement compared with state-of-the-art methods.
2.	The paper is well-written, and the presentation is pretty good.

WEAKNESSES:
1.	The contribution is not convincing enough. The generality of the proposed global structure-aware regularization based diffusion models is not verified, and the main idea of the adopted uncertainty is not original in this paper.
2.	Additionally, the main concern about this paper is that there are many unclear details. For example, what train data is used in this paper? What is the baseline setting in Table.5? Is the inference time of all the methods also performed on RTX 3080 GPUs?
3.	More perceptual metrics (e.g., LPIPS, MUSIQ) could be reported for better evaluation.

QUESTIONS:
Please see the strengths and weakness section.

LIMITATIONS:
Several challenging cases are provided. The limitations of the proposed approach sholud be discussed in more detail.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a diffusion-based low-light image enhancement framework that exhibits good performance in different benchmarks. A rank-informed regularization term during training and uncertainty-weighted loss is proposed.\n\nSTRENGTHS:\n- This work proposes a diffusion-based low-light image enhancement framework that seems to exhibit good performance in different benchmarks.\n\nWEAKNESSES:\n- The motivation of the \"Non-local patch-based matrix rank modeling\" seems not clear.\n\nAccording to my understanding, the proposed \"Non-local patch-based matrix rank modeling\" is only used during training. How does it make the model have the capacity to be aware of the global structure during inference?\n\nBesides, during the training using paired training data, what's the necessity of using the proposed method to measure the structure consistency between Xt−1 and X0? The difference be directly calculated using metrics like L1, L2, SSIM, or feature-level similarities.\n\n- How does the proposed method increase the training overhead?\n\n- Some formulation is not self-contained. For example, where is the definition of \"\\alpha_1^2\" in Eq. (4)?  How to set the value of k_t?\n\n- It's better to illustrate which dataset/subset of the dataset is used to train the proposed method and other competitors. Since it greatly affects the performance of the model. And including a comparison of inference cost is better.\n\nQUESTIONS:\n- Please refer to the weakness section.\n\n- While the weighting strategy of the training loss is reasonable for idea cases. But for real-captured high-resolution images where small misalignments may exist, will it lead to some side effects?\n- Could you explain why the constraint need only be added to the singular value, while no need for the singular vector?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper presents a diffusion-based framework to enhance low-light images. They propose a global structure regularization, which leverages the intrinsic non-local structural constituents of image data, besides, they introduce an uncertainty-guided regularization technique, which relaxes constraints on the most extreme portions of the image. The result outperforms some SOTA methods in some low-light image enhancement datasets.

STRENGTHS:
The paper proposes a global structure-aware regularization scheme, which capitalizes on image intrinsic non-local structures and gradually adjusts the regularization strength according to the sampling steps.

The method achieves good results in the LOL dataset.

WEAKNESSES:
My main concern is about the novelty of the method, the uncertainty-guided regularization seems from[4]

 Although the author has done comparison experiments in some datasets. I think the number of these images is quite limited, I hope the paper can provide a comparison experiment in the Adobe-FiveK dataset[1], following the setting of[2]

The method needs to resample 500 steps. it will cost a long time to infer an image.

The paper should cite and compare with[3].


[1] Learning Photographic Global Tonal Adjustment with a Database of Input / Output Image Pairs
[2] Underexposed Photo Enhancement Using Deep Illumination Estimation
[3]Pyramid Diffusion Models For Low-light Image Enhancement
[4]. Uncertainty-driven loss for single image super-resolution.

QUESTIONS:
Why does the method choose DDPM? How about DDIM?

LIMITATIONS:
See in weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11