Global Structure-Aware Diffusion Process for Low-light Image Enhancement
Authors: Jinhui HOU
Zhiyu Zhu
Junhui Hou
Hui LIU
Huanqiang Zeng
Hui Yuan
Venue: NeurIPS 2023 poster (2023)
Keywords: Image enhancement
diffusion models
TL;DR: global structure-aware diffusion process for low-light image enhancement, achieving state-fo-the-art performance.

Abstract: This paper studies a diffusion-based framework to address the low-light image enhancement problem. To harness the capabilities of diffusion models, we delve into this intricate process and advocate for the regularization of its inherent ODE-trajectory. To be specific, inspired by the recent research that low curvature ODE-trajectory results in a stable and effective diffusion process, we formulate a curvature regularization term anchored in the intrinsic non-local structures of image data, i.e., global structure-aware regularization, which gradually facilitates the preservation of complicated details and the augmentation of contrast during the diffusion process. This incorporation mitigates the adverse effects of noise and artifacts resulting from the diffusion process, leading to a more precise and flexible enhancement. To additionally promote learning in challenging regions, we introduce an uncertainty-guided regularization technique, which wisely relaxes constraints on the most extreme regions of the image. Experimental evaluations reveal that the proposed diffusion-based framework, complemented by rank-informed regularization, attains distinguished performance in low-light enhancement. The outcomes indicate substantial advancements in image quality, noise suppression, and contrast amplification in comparison with state-of-the-art methods. We believe this innovative approach will stimulate further exploration and advancement in low-light image processing, with potential implications for other applications of diffusion models. The code is publicly available at https://github.com/jinnh/GSAD.

Global Structure-Aware Diffusion Process for Low-Light Image Enhancement
Jinhui Hou', Zhiyu Zhu', Junhui Hou Liu', Huanqiang Zeng", and Hui Yuan"
*City University of Hong Kong, "Caritas Institute of Higher Education 3 Huaqiao University, *Shandong University
Abstract
This paper studies a diffusion-basee framework address the low-light image enhancement problem. To harness the capabilities of diffusion models, we delve into this intricate process and advocate for the regularization of its inherent ODE- trajectory. To be specific, inspired by the recent research that low curvature ODE-trajectory results in a stable and effective diffusion process, we formulate curvature regularization term anchored in the intrinsic non-local structures of image data, i.c., global structure-awar regularization, which gradually facilitates the preservation of complicated details and the augmentation of contrast during the diffusion process. This incorporation mitigates the adverse effects of noise and arti- facts resulting from the diffusion process, leading more precise and flexible hancement. To additionally promote learning in challenging regions, we introduce uncertainty -uiided regularization technique, which wisely relaxes constraints on the most extreme regions of the image. Experimental evaluations reveal that the proposed iiffusion-based framework, complemented by rank-informed regulariza- tion, attains distinguished performance in low-light enhancement. The outcomes indicate substantial advancements in image quality, noise suppression, and contrast amplification in comparison with state-of-the-ard methods. We believe this inno- vative approach will stimulate further exploration and advancement in low-light image processing, with potential implications for other applications of diffusion models. The code is publicly available https /// //tgt/tttubb//// com/ com/jinntl jinnh/ GSAD.
Introduction
Low-light image enhancement (LLIE) aims to improve the visibility and contrast of the image captured in poor lighting while preserving the natural-looking details, which contributes to many downstream applications, such as object detection [15, 58) and semantic segmentation [62, 48).
Traditional works employ some techniques, such as histogram equalization [28], Retinex theory [6, 23], and gamma correction [29], to correct the image illumination. In recent years, owing to the powerful representational ability and large collections data, a considerable number of deep bearning-based methods [53, 54, 63, 5, 21, 64, 43, 51, 40, 65] have been presented to significantly improve the performance low-light enhancement by learning the mapping between low-light and normal-ligtt images. Generally, most of the existing works tend to adopt pixel-wise objective functions to optimize a deterministic relationship. Consequently, such regularization frequently produces suboptimal reconstructions for indeterminate regions and poor local structures, resulting in visibly lower reconstruction quality. Although the adversarial loss might mitigate this issue, these methods necessitate careful training adjustments, which might lead to overfitting specific features or data distributions and even creating new content or artifacts. Recently, the popular diffusion
*Corresponding author (Email: jh ht..houuity... edu.hk). Jinhui Hou and Zhiyu Zhu contributed this work equally.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Algorithm Forward process
Algorithm Reverse process
1: Repeat
XT I(O,1)
2: Xo ~ ((Xo)
2: For L = T do
3: L ~ U
Z ~ ~~( ift else z =0
4: E e (0.II)
4:
X,-1 var (X+ vi-a: co (X,, to,z
5: Take gradient descent step
6: Vall cø (v + /I aee, t)1?
5: End for
7: Until converged
6: Return X.
denoising probabilistic model (DDPM) [7] has garnered notable interest in low-level vision domains [9. 32], credited with its outstanding capacity in modeling the distribution of image pixels. However, straightforward implementation of the diffusion model for low-light image enhancement is insufficient address this issue.
In this paper, we present novel diffusion-basee method to boost the performance of low-light enhancement from the perspective of regularizing the ODE-trajectory. Drawing inspiration from the evidence that suggests the potential of a low-curvature trajectory in yielding superior recon- struction performance [12], endeavor modulate the ODE-trajectory curvature. Specifically, we introduce a global structure-aware regularization scheme into the diffusion based framework by gradually exploiting the intrinsic structure of image data. This innovative constraint promotes structural consistency and content coherence across similar regions, contributing to a more natural and aesthetically pleasing image enhancement while preserving the image's fine details and textures. In addition, we devise an uncertainty -uudded regularization by integrating an uncertainty map into the diffusion process, facilitating adaptive modulation of the regularization strength. Experimental results on a comprehensive set of benchmark datasets consistently demonstrate the superior performance of our proposed method compared to state-of-the th--hhe-tt ones. A thorough ablation study emphasizes the significance of both the global structure-awar regularization and uncertainty-guiie constraint components in our approach, revealing their synergistic effects on the overall quality of enhancement.
2 Related work
Low-light image enhancement. The early works tackle low-light image enhancement by applying some traditional techniques like histogram equalization [28, 36]. gamma correction [29], and Retinex theory [6, 23]. Some researchers also seek to improve the visibility using additional sensors[67, 66, 42, 41, 37, 17]. In recent years, with the advancement of low-light data collection [11, 45, 14, 54] significant number of deep barning-based methods [2, 60, 8, 50, 53, 54, 63, 21, 64, 43, 51, 40] have been proposed, which greatly improved the restoration quality of traditional methods. For example, Retinex- based deep learning methods [45, 60, 59, 47] employed deep learning to decompose low-light images into two smaller subspaces. i.c., illumination and reflectance maps. Wang et al. [43] proposed a normalizing flow-based low-light image enhancement approach that models the distributions across normally exposed images. Xu et al. [51] incorporated the Signal-to-Note Noisoo-- Ratio (SNR) prior to achieve spatial-vary varying low-light image enhancement. Wang er al. [40] proposed a transformer- low-light enhancement method. We refer readers [13] for a comprehensive review of this field.
crffusion-based image restoration. Recently, diffusion- -based generative models [34] have delivered astounding results with the advancements in denoising diffusion probabilistic models (DDPM) [7, 24), making them increasingly influential in low-level vision tasks, such as image super-resolution [9, 32). image inpainting [30, 19]. image deraining [26]. and image deblurring [46]. Saharia et al. [32] employed a denoising diffusion probabilistic model to image super-resolution by conditioning the low-resolution images in the diffusion process. Lugmayr al. [19] proposed mask-agnostic approach for image inpainting by leveraging the pre- traited unconditional DDPM as the generative prior. Ozan et al. [26] developed patch-based diffusion model for weather removal that utilized mean estimated noise- noise-guided sampling update strategy across overlapping patches. These models are based on a diffusion process that transforms a clean image into a noisy one during training time, and then reverses that Markov Chain to generate new images in the testing phase. Consider clean image Xn b and noise variable E R NO0 a² T). The training steps of a diffusion process act as Algorithm 1.

---

Reverse trajectors
X--x
Elbct nethed
Reverse trajectury
global structure-aware regularization
sampling
Figure 1: Reverse trajectories of diffusion model with and without our global structure- regular- ization on the testing set of LOLvI dataset. Our method effectively compactifies the distribution of reverse trajectories of multiple samplings, producing low-curvature reverse trajectories, which makes them stably approach the Ground Truth (GT) values.
Note Xo aye is a closed-form form noisy sample at timestamp I, and c0(5s) denotes a noise estimation network, which perceives the closed-form samples with timestamps and predicts the inherent noise in such samples. By applying this process in reverse, shown as Algorithm 2, from Xr (O I) to Xo. obtain generated image Xo. where X, gxx is the intermediate reconstructed sample at timestamp E [0, 1, ..., Tj, at denotes a scaling scalar, usually parameterized
as linearly decreasing sequence [7, 32], and di = II 0g.-
1=1
Proposed Method
3.1 Problem Statement and Overview
Low-light image enhancement is a crucial research area in the field of image processing and com- puter vision, owing the increasing demand for high-quality images captured in various low-light environments. Mathematically, the degradation of image captured under low-light conditions can be modeled as
X O S
where Y e #xxx3 is the observed low-light image of dimensions H x W, X, E Bfxx33 is the latent high-quality image under normal-light conditions. S #x x3 represents the spatially varying illumination map that accounts for the uneven lighting, and N E #R is the noise term introduced due to sensor limitations and other factors. Thus, low-light image enhancement, particularly in reconstructing extreme low-light regions, is of paramount challenge due to the scarcity of available content such images. The intricacy further intensified by the content-dependes nature of optimal lighting conditions, necessitating flexible and adaptive solution. Fortunately, diffusion models, celebrated for their remarkable capacity to synthesize images conditioned on ancillary images or textual cues, emerge as a propitious candidate for tackling the low-light image enhancement conundrum.
In this paper, we advocate capitalizing on the potent generalization capabilities inherent in diffusion models to overcome the obstacles associated with low-light image enhancement. By incorporating low-light images (Y) as conditioning inputs into the diffusion models, we can obtain naïve imple- mentation of low-light enhancement diffusion model (co(Y Y, X,, at)). Nonetheless, the performance achieved by these preliminary models remains unsatisfactory. In pursuit of procuring high-qualin diffusion processes, various strategies have been presented, including alterations to the network struc- ture [31], the incorporation of data augmentation techniques [38], and the introduction of innovative loss terms [22]. Among them, trajectory rectification emerges as a promising strategy, facilitating seamless and stable diffusion process through the regularization of low curvature reverse trajectory

---

Adaptive regularization with uncertainty
MLP
Notey level embedddig
||P,OE &,
Constructing learnable
dlsee-form samplex
X
X
Learning image global
Clustering
Cluster
MOD
NoAuee
Volue
Glosee
Cluster
The Same Charter Prdea
Figure 2: Illustration of the training workflow of the proposed method. We begin by obtaining closed-form sample X, from Xo and the noise level og, which is subsequently fed into U-Net c(-) conditioned with the corresponding low-light image Y and ūg estimate the employed noise c. To comprehensively regularize reconstructed images, we proceed to construct learnable closed-form sample X,-1 using the estimated noise eø( (Y,,, X,, āt) and optimize the model using uncertainty- guided noise estimation and intrinsic image global structure. Meanwhile, we use a pre-trained uncertainty estimation model μig obtain the uncertainty map P.-
[12, 18]. Encouraged by current successes, we endeavor to augment this preliminary model from the following two distinct perspectives.
1) Learning low curvature trajectories with global structure-aware regularization. To learn a low curvature reverse trajectory, propose simple yet effective way via directly minimizing the gap between a learnable sample on the reverse trajectory and the ground-truth sample. This process further elucidated through pertinent experimental results presented in Fig. Moreover, is crucial to acknowledge that images intrinsically encompass analogous textures or patterns distributed across different locations [1, 27, 3, 4, 49, 16, 56). As result, cultivating diffusion process that adeptly captures such global structures indispensable to the success low-light image reconstruction. By integrating the matrix rank-based regularization within the diffusion-basee framework, we astutely exploit the intrinsic structure of image data, thereby facilitating the preservation of nuanced details and the enhancement of contrast.
Despite the effectiveness of the trajectory regularization, precipitating during the beginning steps of diffusion models could unintentionally subvert sample diversity and quality, attributable to the pronounced fluctuations in large components. Consequently, in pursuit of enhancing the effectiveness of our model, it becomes imperative to architect progressively adaptive regularization schedule.
2) Adaptive regularization with uncertainty. Furthermore, most contemporary loss functions treat all pixels equally in the field of low-light enhancement tasks, regardless of whether they exist in conditions of slightly low-light or extremely now-brightness, or whether they are subject to noise-deterioration are noise-free. Therefore, drawing inspiration from [25], we introduce an umcertainty-guiiae guided regularization into the diffusion process to further improve restoration learning in challenging regions.
Those strategic incorporations ensure the precise and resilient recovery of image details in struggling low-light regions while accommodating scene- specifcc lighting variations. In the following sections, we give technical details of our method.
3.2 Exploring Global Structures via Matrix Rank Modeling
The denoising diffusion probabilistic models exhibit unique learning scheme, which involves learning the distribution of latent noise during training. This characteristic makes it challenging to regularize network parameters 0 for learning global structure-aware representations. To address this

---

j=1 issue, we first construct learnable closed-fone sample of X Subsequently, we implement two further steps enhance the content-awar regularization those images.
1) Constructing learnable closed-form samples. The training of diffusion models commences with the procurement of closed-form X; at arbitrary timestep i, delineated in Line 6 from Algorithm 1. In the ensuing step, a learnable function ea (Y, Xx, att) is employed to estimate the latent noise e, Given that the network can successfully learn this noise, it becomes feasible progressively eliminate the noise, as demonstrated by:
1 1 ar (Y,,,, + o.Z. v1-a
X (X,
04
Consequently, throughout the training phase, the regularization strategically applied to the learnable sample X in licu of the closed-form sample X,. This approach seeks to optimize the performance of the model by focusing on regularization where it can most effectively influence the learning process.
2) Non-local patch- based matrix rank modeling. To find global structures of clean image X (+) we first patchify it into set of non-overlapping blocks {xg Ji where X E R denotes the vectorization of small block from Xo. To capture the intrinsic structures from the sample Xo. we
then apply the commonly-use ussed clustering algorithm to get & clusters of (x6 (9) yizı as { [xx (i)
):=1
It is worth noting that our method benefits from advanced clustering algorithms greatly (See Sec. 4.3 for details). Through stacking each cluster of vectors into a matrix, we could get (M, %
k Jー=1 where
the rank of matrix M E _m*x, is an effective measurement of the global structures. Consequently, regularize structure consistency in X,-1 and X.. we then apply the same aggregation to the } learnable sample X resulting in (M 15-1+ and regularize
C. (M,,,, ) = =||dia((S)_ diag(S)) M = V/S/VE
(3)
where L. ,,, My) indicates the regularization function, U E RRmmm, S E Rmxx, and V Xny are the outputs of the singular value decomposition of matrix M and diag(.) returns a vector of the matrix main diagonal.
3) Gradually injecting structure-aware regularization. This strategy delicately calibrates the focus on patches throughout different learning steps, thereby ensuring a nuanced, stepwise infusion of structure-aware regularization. This judicious approach not only preserves the heterogeneity of the reconstructed samples but also optimizes the noise suppression process, culminating in a notable improvement in the overall performance of our model. Specifically, the modified regularization function C is written as
C M. K az ..... "...,&e?) a62), t ~ (1,2, T},
(4)
where K2 denotes an adaptive factor for regularization term C. (M,", M ). This schedule allows for more adaptive and flexible approach to regularization, enabling the model to capture better the global structures and details within the low-light images.
Analysis of choosing rank-based global structure-aware regularization. To minimize the curvature of ODE-trajectory we bridge the deviation between the learnable closed-form trajectory samples X,-1 and the GT X, by our rank-based global structure-aware regularization. Moreover, it is essential acknowledge that while conventional pixel-wise regularization terms, including L1. Lg- and SSIM, do offer some degree of enhancement to the original diffusion model, their impact is constrained. This limitation arises from their inadequate ability fully encapsulate the non-local structures and intricate patterns within an image. Although regularization within the feature domain potentially aids in modeling a given image's 's structure, such structure is usually confined within a local region due to the kernel perceptual regions. This type of regularization, moreover, cannot explicitly characterize the properties of the structure and is lacking theoretical guidance. Another significant concern is that regularization features could result in significant fluctuations in the back-propagated gradients, thereby impeding network training. Lastly, the empirical evidence documented in Table corroborates the superiority of rank-based global structure-an aware regularization.
5

---

3.3 Integrating Uncertainty into Diffusion Process
To integrate uncertainty into the diffusion process, inspired by [25], we introduce an additional convolutional branch after the last layer of U-Net of the diffusion model for generating the uncertainty map. Specifically, we train the uncertainty model μe, (-) with the same step as the denoising diffusion models, like Algorithm I, except that we optimize the following objective:
(5) where P. == 10, X, denotes pixel-wise uncertainty map, i.e., P, E RHWWx3 co(.) represents the corresponding function with weights of a denoising diffusion model. Upon completion of the uncertainty estimation model's pretraining phase, we obtain Mø.- and cg- Subsequently, we fix the parameter of Mii to generate the uncertainty map Pt. This map serves as a flexible weight, calibrating the importance of each pixel such that pixels with higher uncertainty are assigned greater weight. Further details are provided in Algorithm 3. The integration of uncertainty into the diffusion process in this manner incentivizes the diffusion model to intensify its focus on uncertain or challenging regions. This strategic approach yields visually pleasing results, underscoring the value of incorporating uncertainty-awaat regularization in the enhancement of low-light images.
Ln -le --=lleP -P- c (E cY X,, āt) Mll +
Algorithm 3 Training diffusion- based low-light enhancement model
Require: Normal and low-light image pairs (Xo, Y), and a pre-trained models μe, and cØ-
1: Initialize parameters of μed (-) (resp. ea (-)) from pretrained 0, (resp. 0), and freeze 01.
2: Repeat
3: Sample an image pair (Xo. Y). āg ~ p(ā) and ~ (0,1))
4: X, Vāz Xo + VI- ā
5: P.+ No, (Y,X, (x)aa,)) a.)
6:
Construct the learnable closed-form sample X,-1 via Eq. (2)
7:
Perform gradient descent step on
8:
V IP, O P, о co (Y, X az) + Cg ), where À is balancing factor, and C refers
to Eq.
9: Until converged
10: Return the resulting diffusion model
Experiments
4.1 Experiment Settings
Datasets. We employed seven commonly-used LLIE benchmark datasets for evaluation, including LOLvI [45], LOLv2 [54], DICM [11], LIME [6], MEF [14], NPE [39], and VV2. Specifically, LOLVI contains 485 low/nornal-light image pairs for training and 15 pairs for testing, captured various exposure times from the real scene. LOLv2 is split into two subsets: LOLv2-real and LOLv2-synthetic. LOLv2-real comprises 689 pairs of low-/normal--in pairs for testing, collected by adjusting the exposure time and ISO. LOLv2-synthetic was generated by analyzing the illumination distribution of low-light images, consisting of 900 paired images for training and 100 pairs for testing. accordance with the settings outlined in the recent works [51, 43, 40], we trained and tested our models on LOLvI and LOLv2 datasets separately. The DICM, LIME, MEF, NPE, and VV datasets contain several unpaired real low-light images, which are used only for testing.
images for training and 100
Evaluation metrics. We adopted both full-reference and non-reference image quality evaluation metrics to evaluate various LLIE approaches. For paired data testing, we utilized peak signal-to-noise ratio (PSNR), structural similarity (SSIM) [44], and learned perceptual image patch similarity (LPIPS) [57]. For datasets such as DICM, LIME, MEF, NPE, and VV, where paired data are unavailable, we adopted only the naturalness image quality evaluator (NIQE) [20].
Methods under comparison. We compared our method with a great variety of ofate-of-the-art LLIE methods, i..., LIME [6], RetinexNet [45], KinD [60], Zero-DCE [5], DRBN [52], KinD++ [59].

---

Table 1: Quantitative comparisons of different methods on LOLvI and LOLv2. The best and second- best results are highlighted in bold and underlined, respectively. "f" (resp. "4") means the larger (resp. smaller), the better. Note that we obtained these results either from the corresponding papers, or by running the pre-trained models released by the authors, and some of them lack relevant results on the LOLv2-
LOLv2-synthetic synthetic dataset.
LOLv1
LOLv2-rcal
LOLv2-syntheti
Methods
Param (M)
PSNRT
SSIM
LPIPS)
PSNRT
SSIM
LPIPS
PSNRT
SSIM
LPIPS]
LIME 16P
16,760
0.560
0.350
15.240
0.470
0.415
16.880
0.776
0.675
[5]
0.562
EnlightenGANN [8] TIP
17.483
0.652
0.322
18.640
0.677
0.309
16.570
0.734
8.64
RetinexNet [45] BMVC'18 18
16.770
0.462
0.474
18.371
0.723
0.365
17.130
0.798
0.754
0.62
DRBN [52] CVPR 20
19,860
0.834
0.155
20.130
0.830
0,147
23 22.200
0.927
2.21
KinD [60]
0.799
0.669
0.591
KinD++ [59] UCV
21.300
0.823
19.087
0.817
MTRNet [55] TRAMU22
24.140
0.842
0.131
20.357
0.782
0.317
21.940
0.846
5.90
LLFlow [43] AAAU22
25.132
0.872
0.117
26.200
0.888
0.137
24.807
0,9193
0.067
37.68
LLFormer AAAU2S
25.758
0.823
0.167
26.197
0.819
0,209
26,006
0.927
0.061
24.55
[51]
Ours
27.839
0.877
0.091
28.818
0.895
0.095
28.670
0.944
0.047
17.36
LOLv2-raal
LOLv2-syn
Low
KinD
LLFlow
LLFormer
SNR_Aware
Ours
Ground
Figure Visual comparisons of the enhanced results by different methods on LOLvv and L.Lv2.
EnlightenGAN [8], MIRNet [55], LLFlow [43], DCC-Net [61]. SNR-Aware [51], and L.Former [40].
Implementation details. We trained our models for 2M iteration steps with PyTorch. We employed
an Adam optimizer [10] with fixed learning rate of 1 10-4 without weight decay. We applied an exponential moving average with weight of 0.9999 during parameter updates. During training, we utilized fine-grained diffusion process with T = 500 steps and linear noise schedule with the two endpoints of x 10 and 2 10 10-2. The patch size and batch size were set to 96 and 8, respectively. The hyper- parameter À empirically set to 10.
4.2
Comparison with State-of-the-Art Methods
Results on LOLv1 and LOLv2. Table shows the quantitative results of different LLIE methods, where it can be observed that our method consistently achieves the best performance over all the compared methods in terms PSNR, SSIM, and L.PIS metrics. Particularly, quality of our improvement on LPIPS provides compelling evidence of the superior perceptual method, as shown Fig. 3, where our method effectively suppresses artifacts and reveals image details, leading to visually appealing results that are more faithful to the original scene.
remarkable
7

---

DICM
MEF
VV.
Low
Zero-DCE
RetinexNet
KinD
KinDee
EnlightenGAN
SNR-Aver Aware
Ours
Figure 4: Visual comparisons of different methods DICM, LIME, MEF, NPE, and VV datasets.
Results on DICM, LIME, MEF, NPE, and VV. We directly applied the model trained on the LOLv2-synthetic dataset to these un- paired real-world datasets. From Table we can observe that our method yields bet- ter NIQE scores than the other competi- tors, demonstrating its stronger generaliza- tion ability. In addition, a visual comparison of enhancement results from various meth- ods, as showcased Fig. 4, underscores the superior performance of our approach. Evidently, our model adeptly adjusts the lumination conditions, optimally enhancing the visibility in low-light areas while vigi- lantly circumventing over-exposure. This successful balance corroborates the superior efficacy of our proposed method.
Table 2: Quantitative comparisons of different meth- ods on DICM, LIME, MEF, NPE, and VV datasets in terms NIQE, where smaller values lead to better performance.
DICM
Pero-DCEPp [5]
4.58
4.93
4.53
4.81
EnlighteeGAN [8]
4.06
4.59
4.70
3.99
4,04
RetinexNet [45]
4.33
5.75
4.93
4.95
4.32
KinD [60]
3,95
3.72
[59]
SNR-Aware [51]
6.12
5.93
6.44
6.44
11.50
DCC-Ner [61]
3,70
4.42
4.59
3.70
3.28
Ours
3.28
4.32
3.55
2.60
Table 3: Ablation studies of effectiveness of various clustering algorithms used in our method.
Clustering Algorithms Baseline
K. Means
Spectral clustering
Gaussian Mixture Model
Hierarchical clustering
PSNR 26.02 SSIM LPIPS]
27.02 0.872
27.41
27.55
27.70
0.876
0.877
0.880
0,123
0.097
0.095
0,095
0.092
Table 4: Ablation studies of the effectiveness of choosing our rank-based global structure-aware regularization.
Regularization
Baseline
Lr loss
Lg loss
SSIM
Perceptual loss
Our rank-based modeling
PSNRT
26.02 0.859
26.45
26.53
26.23
26.42
27.70
SSIM"
0.868
0.869
0.870
0.863
0.850
0.101
0.099
0.092
4.3 Ablation Study
Clustering algorithms. We investigated how the clustering method used for finding similar patches affects the quantitative performance on LOLvI. As shown in Table 3, our method generally benefits from different clustering algorithms. Especially, when employing advanced hierarchical clustering, the PSNR value experiences an improvement of approximately 0.7 dB, compared to the results achieved with the K-Means clustering. This further underscores the prospective utility of our

---

Table 5: Quantitative results of the ablation stud- Figure 5: Visual results of the four settings of
ies 1.Lx1. "X"" (resp. "v") means the cor- the ablation studies in Table 5. responding module unused (resp. used). (a)
Non-local patch-based matrix rank modeling, (b)
Gradually injecting structure-aware regularization, and (c) Uncertainty-guided regularization. Note
Input
that (b) is applicable only if (a) is utilized.
(a)
(b)
(c)
PSNR1
SSIM
LPIPS
1)
X
X
x
26.02
0.8593
0.1226
2)
X
X
26.79
0,8708
0.1049
1)
2)
3)
X
x
27.02
0.8753
0.0980
4)
V
x
27.70
0.8795
0.0923
5)
V
27.84
0.8774
0.0905
Ground Truth
regularization term in enhancing diffusion models with advanced clustering algorithms. Consequently, we adopted advanced hierarchical clustering our approach for subsequent experiments.
Regularization terms between X,-1 and Xo- We delved into the impact of utilizing varying reg- ularization terms between X₁-1 and Xo on the enhancement of diffusion models within LOLv1. Table 4 offers insights that although conventional pixel-wise regularization terms such L1, L2, and SSIM can moderately elevate the performance of the baseline, their enhancements pale in com- parison to our rank-based modeling regularization. The primary reason is the inability of these regu- larization terms to fully encapsulate the nonlocal structures or patterns inherent within an image. These outcomes distinctly highlight the superior- ity of our rank- based model in capturing global structures.
50
regularization
40
Truth
W/o global
Singular
30
20
Ours
10
GT
Index
50
structure regularization
Ground Truth
Non-local patch-based matrix rank modeling & adaptive regularization schedule. We estab- lished baseline by demounting the non-local ma- trix rank-based representation learning regulariza- tion. By comparing rows 2), 3), 4), and 5) in Table 5, it can be observed that this module equipped with our adaptive regularization schedule signifi- cantly boosts the enhancement performance. How- ever, this module without adopting the adaptive regularization schedule only achieves minor im- provement. Such observations demonstrate the necessity of our adaptive regularization schedule and the effectiveness of non-local rank-based rep- resentation learning.
Singular value
W/o global
30
20
Ours
10
10
15
20
Index
Figure 6: Visual comparisons of the distribu- tion of the singular values of cluster from en- hanced images by our method with and with- out global structure-aware regularization. Note that the patches contained in the cluster are high- lighted with a white box.
Furthermore, as depicted in Fig. 6, we claborate on the matrix rank across clusters from several samples, thereby offering clear visualization of the effectiveness this regularization. Empirical findings affirm that implementation of our proposed regularization notably narrows the gap of singular values between the reconstruction output and the ground truth, thereby substantiating the indispensability of this regularization approach and validating its strategic integration into the diffusion process.
intertainty-guideee regularization. The effectiveness this regularization substantiated through a comparative analysis of rows 1) and 2) of Table 5. Notably, the implementation of this regularization evidenced to bolster the performance of image enhancement. Such improvement, however, confined to pixel-wise metrics, e.g., PSNR and SSIM, and does not extend to global visual metrics

---

Inpul
L.P
SNR Aware
Empat
L.FFlew
SVR Aware
Ground Truch
L.Former
Onrs
Ground Truth
L.F. Fermer
Ours
Figure 7:
Illustration of challenging scenarios, where both the proposed method and other existing methods exhibit limitations achieving visually pleasing reconstructions.
such as LPIPS (visualization as Fig. 5). This observation implies that the incorporation of uncertainty- based regularization in isolation yields incremental advancements but fails to induce substantial modifications to the structure the reconstructed images. The findings, therefore, underscore the necessity of an integrated approach that combines uncertainty- bbased regularization with global structure-aware regularization to achieve significant improvements in both local pixel-level and global structural aspects of image enhancement.
4.4 Challenging Cases
While the proposed method has demonstrated notable advancements in performance, as showcased in the previous sections, it is important acknowledge that certain challenging scenarios persist within the domain of low-light enhancement tasks. Fig. 7 illustrates two challenging examples that highlight the intricacy of the task at hand. In these examples, it is evident that certain regions of the input image are under extreme low-light conditions, to the point where any trace of texture becomes indiscernible. This lack of visual information poses significant challenge, causing both our proposed method and other state-of-the-ar techniques to struggle in generating visually compelling details. To tackle these challenges, could identify subset of extremely low-light patches and apply robust regularization to them to improve reconstruction performance.
Conclusion and Discussion
We have introduced a diffusion model-based approach for enhancing low-light images effectively. The cornerstone of our approach lies in the innovative training strategy of diffusion models from the perspective of minimizing the ODE-trajectory curvature, which involves the global structure- aware rank- based regularization learnable closed-fone samples and uncertainty -basd learning of latent noises. Recognizing the potential adverse effects of regularization within the early stage of diffusion models, we have thoughtfully devised progressively adaptive regularization schedule. This strategy ensures a nuanced, step-wise infusion of structure-awan regularization, preserving the inherent structural integrity in the reconstructed samples. As continue refine approach, anticipate further improvements and look forward to the potential applications and advancements these developments will facilitate in broader fields.
While our work exhibits admirable performance compared with contemporary methods, we acknowl- edge the inherent computational burden associated with diffusion-buud based their iterative noise removal reverse Markov chain. Nonetheless, thanks to the latest strides the field [33, 35], it is anticipated that the diffusion models may be capable of delivering accelerated reconstructions through minimal number iterative steps.
to
Acknowledgements
This work was supported in part by the Hong Kong Research Grants Council under Grant CityU 11 11218121, 1, in part by the Hong Kong Innovation and Technology Fund under Grant MHP/1 17/21, and in part by Hong Kong University Grants Committee under Grant UGC/FDS and part by the Natural Science Foundation of Shandong Province under Grant ZR2022ZD38.
10

---

References
[1] Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, volume 2, pages 60-65, 2005.
[2] Jianrui Cai, Shuhang Gu, and Lei Zhang. Learning deep single image contrast enhancer from multi- exposure images. IEEE Transactions on Image Processing. 27(4):2049 2062, 2018.
[3] Weisheng Dong, Lei Zhang, Guangming Shi, and Xin Li. Nonlocally centralized sparse representation for image restoration. IEEE Transactions on Image Processing, 22(4): 1630, 2012.
[4] Shuhang Gu, Lei Zhang. Wangmeng Zuo, and Xiangchu Feng. Weighted nuclear minimization with application to image denoising. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2862-2869, 2014.
[5] Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong. Zero-reterent reference deep curve estimation for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1780- 1789, 2020.
[6] Xiaojie Guo, Yu Li, and Haibin Ling. Lime: Low-light image enhancement via illumination map estimation. JEEE Transactions on Image Processing, 26(2):982-99) 2016.
[7] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of Advances in Neural Information Processing Systems, volume 33, pages 0840-6851, 2020.
[8] Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang. Pan Zhou, and Zhangyang Wang. Enlightengan: Deep light enhancement without paired supervision. IEEE Transactions on Image Processing, 30:2340 2349, 2021.
[9] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Proceedings Advances in Neural Information Processing Systems, volume 35, pages 23593-23606. 2022.
[10] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.66 6980, 2014.
[11] Chulwao Lee, Chul Lee, and Chang-Su Kim. Contrast enhancement based layered difference represen- tation. Proceedings of the IEEE/CVF Conference on International Conference on Image Processing, pages 965 968, 2012.
[12] Sangyun Lee, Beomsu Kim, Jong Chul Ye. Minimizing trajectory curvature of ode-based generative models. In Proceedings of International Conference on Machine Learning, 2023.
[13] Chongyi Li, Chunle Guo, Linghao Han, Jun Jiang. Ming-Ming Cheng, Jinwei Gu, Chen Change Loy. Low-light image and video enhancement using deep learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):939-995 941666996222299 2021.
[14] Mading Li, Jiaying Liu, Wenhan Yang, Xiaoyan Sun, and Zongming Guo. Structure-revaaling low-light image enhancement via robust retinex model. IEEE Transactions on Image Processing, 27(6):2828 2841, 2018.
[15] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He. Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21 2112-2, 2017.
[16] Ding Liu, Bihan Wen, Yuchen Fan, Chen Change Loy, and Thomas S Huang. Non-local recurrent network for image restoration. In Proceedings of Advances in Neural Information Processing Systems, volume 31, 2018.
[17] Lin Liu, Junfeng An, Jianzhuang Liu, Shanxin Yuan, Xiangyu Chen, Wengang Zhou, Houqiang Li, Yan Feng Wang, and Qi Tian. Light video enhancement with synthetic event guidance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages (692-1706 2023.
[18] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In Proceedings of International Conference on Learning Representations, 2023.
[19] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timotte, and Lue Van Gool. Repaint: Inpainting using denoising difffusion probabilistic models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11461-11477 2022.
11

---

[20] Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making a "completely" blind image quality analyzer. IEEE Signal Processing Letters, 2063):20--21,, 2012.
[21] Sean Moran, Pierre Marza, Steven McDonagh, Sarah Parisot, and Gregory Slabaugh. Deepipf: Deep local parametric filters for image enhancement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12826-12835, 2020.
[22] Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner. Diffrf: Rendering- srrddriin-guidedd guided 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4328-4338, 2023.
[23] Michael Ng and Wei Wang. A total variation model for retinex. SIAM Journal on Imaging Sciences, 4(1):345- 365, 2011.
[24] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of International Conference on Machine Learning, pages ,, 2021.
[25] Qian Ning, Weisheng Dong, Xin Li, Jinjian Wu, and Guangming Shi. Uncertainty loss for single image super-resolution. In Proceedings of Advances in Neural Information Processing Systems, volume 34, pages 16398-16609, 2021.
[26] Ozan Özdenizei and Robert Legenstein. Restoring vision in adverse weather conditions with patch-based denoising diffusion models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[27] Gabriel Peyré, Sébastien Bougleux, and Laurent Cohen, Non-local regularization of inverse problems. In Proceedings of European Conference Computer Vision, pages 57 77-8. Springer, 2008
[28] Stephen M Pizer. Contrast-limited adaptive histogram equalization: Speed and effectiveness. Proceed- ings of the first Conference on Visualization Biomedical Computing, volume 337, page 1, 1990.
[29] Shanto Rahman, Md Mostafijur Rahman, Mohammad Abdullah-Al Saaullah-Al-Waduu, Golam Dastegir Al-Quaderi, and Mohammad Shoyaib. An adaptive gamma correction for image enhancement. EURASIP Journal on Image and Video Processing, 2016(1):1-B) 2016.
[30] Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[31] Dohoon Ryu and Jong Chul Ye. Pyramidal denoising diffusion probabilistic models. arXiv preprint arXiv:2208.0186/ 2022.
[32] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pottern Analysis and Machine Intelligence, 45(4):4713 4726. 2022.
[33] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling diffusion models. arXiv preprint w-Xiv:220.000512 2022.
(34] Jascha Sohl- bocl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thernodynamics. In Proceedings of International Conference on Machine Learning. pages 2256-2265, 2015.
[35] Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. In Proceedings of International Conference on Machine Learning, 2023.
[36] Alex Stark. Adaptive image contrast enhancement using generalizations of histogram equalization. IEEE Transactions on image processing, 9(5)8889 896, 2000.
[37] Chuamming Tang, Xiao Wang, Ju Huang, Bo Jiang, Lin Zhu, Jianlin Zhang, Yaowei Wang, and Yonghong Tian. Revisiting color-event based tracking: A unified network, dataset, and metric. arXiv preprint arXiv:2111.10110. 2022.
[38] Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. arXiv preprint arXiv:2302.079.. 079444 2023.
[39] Shuhang Wang. Jin Zheng, Hai-Miao Hu, and Bo Li. Naturalness preserved enhancement algorithm for non-uniform illumination images. IEEE Transactions on Image Processing, 22(9)3353-33444 2013.
12

---

[40] Tao Wang, Kaihao Zhang, Tianrun Shen, Wenhan Luo, Bjorn Stenger, and Tong Lu. Ultrahiig--defiitiio low-light image enhancement: A benchmark and transformer-based method. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2654-2662 2023.
[41] Xiao Wang, Jianing Li, Lin Zhu, Zhipeng Zhang, Zhe Chen, Xin Li, Yaowei Wang, Yonghong Tian, and Feng Wu. Visevent: Reliable object tracking via collahoration of frame and event flows. IEFE Transactions on Cybernetics, 2023.
[42] Xiao Wang, Shiao Wang, Chuanming Tang, Lin Zhu, Bo Jiang, Yonghong Tian, and Jin Tang. Event stream steeam-based visual object tracking: A high-resolution benchmark dataset and novel bascline, arXiv preprint arXi: 1461 2023.
[43] Yufei Wang, Renjie Wan, Wenhan Yang, Haoliang Li, Lap-Pui Chau, and Alex Kot. Low-light image enhancement with normalizing flow. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 2604-2612, 2022.
[44] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing. 13(4):000 612. 2004.
[45] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. Deep retinex decomposition for low-light enhancement. arXiv preprint /808 04560, 2018.
[46] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16293-16303, 2022.
[47] Wenhui Wu, Jian Weng. Pingping Zhang, Xu Wang, Wenhan Yang, and Jianmin Jiang, Uretinex-net: Retinex-based deep unfolding network for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5901-5910, 2022.
[48] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo., Segformer: Simple and efficient design for semantic segmentation with transformers. In Proceedings of Advances Neural Information Processing Systems, volume 34, pages 12077-12090, 2021.
[49] Jun Xu, Lei Zhang, Wangmeng Zuo, David Zhang, and Xiangchu Feng. Patch group based nonlocal self- similarity prior learning for image denoising. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 244-252, 2015.
[50] Ke Xu, Xin Yang. Baocai Yin, and Rynson WH Lau. Learning to restore low-light images via decomposition and-aandddnn eehhnccmeenn. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2281-2296 2020.
[51] Xiaogang Xu, Ruixing Wang, Chi-Wing Fu, and Jiaya Jia. Sur-aware low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17714- 17724, 2022.
[52] Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 3063-3072, 2020.
[53] Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu, Band representation based semi- supervised lightight image enhancement: Bridging the gap between signal fidelity and perceptual quality. IEEE Transactions on Image Processing, 30:3461-341 3473, 2021.
[54] Wenhan Yang, Wenjing Wang, Haofeng Huang. Shiqi Wang, and Jiaying Liu. Sparse gradient regularized deep retinex network for robust low-light image enhancement. IEEE Transactions on Image Processing, 30:2072- 2086, 2021.
[55] Syed Wagas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Learning enriched features for fast image restoration and enhancement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1934 1948, 2022.
[56] Zhiyuan Zha, Bihan Wen, Xin Yuan, Jiantao Zhou, and Ce Zhu. Image restoration via reconciliation of group sparsity and low-rank models. IEEE Transactions on Image Processing, 0552223-538, 2021.
[57] Richard Zhang, Phillip Isola, Alexii A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 586-59 566-595,45 2018.
13

---

[58] Shifeng Zhang. Xiangyu Zhu. Zhen Lei, Hailin Shi. Xiaobo Wang, and Stan Z Li. S3id: Single shot scale-invariant face detector. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 192-201, 2017.
[59] Yonghua Zhang, Xiaojie Guo, Jiayi Ma, Wei Liu, and Jiawan Zhang. Beyond brightening low-light images. International Journal of Computer Vision, 129:1013-1033 2021.
[60] Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo. Kindling the darkness: practical low-light image enhancer. Proceedings of the 27th ACM International Conference on Multimedia, pages 1632 1640, 2019.
[61] Zhao Zhang, Huan Zheng, Richang Hong, Mingliang Xu, Shuicheng Yan, and Meng Wang, Deep color consistent network for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1899-1908, 2022.
[62] Hengshuang Zhao, Jianping Shi. Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2881 2881-2890, 2017.
[63] Lin Zhao, Shao-Ping Lu, Tao Chen, Zhenglu Yang. and Ariel Shamir. Deep symmetric network for underexposed image enhancement with recurrent attentional learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12075 12084, 2021.
[64] Chuanjun Zheng, Daming Shi, and Wentian Shi. Adaptive unfolding total variation network for low-light image enhancement. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4439 4448, 2021.
[65] Dewei Zhou, Zongxin Yang, and Yang. Pyramid diffusion models for low-light image enhancement. arXiv preprint arXiv:2.305. 10028 2023.
[66] Zhiyu Zhu, Junhui Hou, and Xianqiang Lyu. Learning graph-embedded key-event back-tracing for object tracking in event clouds. Advances in Neural Information Processing Systems, 35:7462 7476, 2022.
[67] Zhiyu Zhu, Junhui Hou, Dapeng Oliver Wu. Cross-modal orthogonal high-rank augmentation for rgb-event transeor In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22045-22055 2023.
14


Reviews:
[{"review_id": "kVsaJT4maa", "reviewer": "Reviewer_ts6H", "summary": "This paper presents a diffusion-based framework to enhance low-light images. They propose a global structure regularization, which leverages the intrinsic non-local structural constituents of image data, besides, they introduce an uncertainty-guided regularization technique, which relaxes constraints on the most extreme portions of the image. The result outperforms some SOTA methods in some low-light image enhancement datasets.", "questions": "Why does the method choose DDPM? How about DDIM?  ", "limitations": "See in weakness.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper proposes a global structure-aware regularization scheme, which capitalizes on image intrinsic non-local structures and gradually adjusts the regularization strength according to the sampling steps.\n\nThe method achieves good results in the LOL dataset.", "weaknesses": "My main concern is about the novelty of the method, the uncertainty-guided regularization seems from[4]\n\n Although the author has done comparison experiments in some datasets. I think the number of these images is quite limited, I hope the paper can provide a comparison experiment in the Adobe-FiveK dataset[1], following the setting of[2]\n\nThe method needs to resample 500 steps. it will cost a long time to infer an image.\n\nThe paper should cite and compare with[3].\n\n\n[1] Learning Photographic Global Tonal Adjustment with a Database of Input / Output Image Pairs\n[2] Underexposed Photo Enhancement Using Deep Illumination Estimation\n[3]Pyramid Diffusion Models For Low-light Image Enhancement\n[4]. Uncertainty-driven loss for single image super-resolution. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **[Responses to Reviewer ts6H]**\n\n### ***1** Response to Weakness 1 (W1): Novelty of the Method*\n\nWe would like to underscore that the **MAJOR** innovation presented in our paper is the integration of global structure regularization into diffusion models.  We also refer the reviewer to the **second response to Reviewer gyFW** for detailed analysis. Furthermore, by leveraging sophisticated clustering algorithms for grouping non-local patches, the overall PSNR of our method is further improved to 0.7 dB, convincingly attesting to the importance and efficacy of regularizing global structures (see **the first response to Reviewer U5ii**.  We posit that our contributions will pave the way for fresh perspectives on diffusion models for low-level image processing tasks, as discussed by **Reviewer U5ii**.\n\nIn the manuscript, we have indeed acknowledged that our work is inspired by [4]. As the first time, we demonstrate that uncertainty-aware regularization is a simple yet effective way to boost the performance of the diffusion process under low-light image enhancement. But we also agree that this contribution is somewhat minor. We believe the major contribution meets the NeurIPS's standard based on its novelty and impressive performance.\nIn the final version, we will re-summarize the contribution part to emphasize the key contribution and list the uncertainty as a minor contribution.\n\n### ***2** Response to W2: Evaluation Dataset*\n\nFollowing your valuable suggestion, we conducted the experiments on the Adobe-FiveK dataset in the same setting as [2]. The quantitative results shown in the following table still verify the advantages of our method. We will add further discussion and review of [1,2] in the final version.\n\n| Methods      | PSNR  | SSIM  |\n|--------------|-------|-------|\n| DeepUPE [2]  | 23.04 | 0.893 |\n| Ours         | 23.77 | 0.912 |\n\n### ***3** Response to W3: The Number of Inference Steps*\n\nOur diffusion model takes only **10 steps for inference**, as described in supplementary material. 500 steps are only utilized for training. It is essential to recognize that the use of multiple sampling steps is not an attribute exclusive to our method, but rather a common issue across all diffusion methods. Moreover, some methodologies have been developed with the aim of reducing the steps required within the diffusion model. In addition, it is pivotal to highlight that the inference method employed in our methods operates in alignment with a standard diffusion model. Consequently, this allows for the integration and utilization of advanced fast-sampling techniques, as detailed in references [C1].\n\n[C1] C. Lu, et al., Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps, NeurIPS'22\n\n### ***4** Response to W4: Related Work*\n\nIt is worth noting that the mentioned work [3] was accepted in IJCAI'23 and first available in arXiv on 2023 May 17, which definitely coincided with the paper submission deadline for NeurIPS'23. Thus, it should be considered as a concurrent work with ours. But we made comparisons with it as its test code is available. Under the same test settings as ours, we obtained the results on LOLv1 by running the publicly released pre-trained model that was trained with the same training dataset as ours. As shown in the following table, the advantage of our method is still verified.\n\n| Methods    | Architecture    | Loss term                                                 | PSNR   | SSIM  | LPIPS |\n|------------|---------------|------------------------------------------|------|-----|----|\n| PyDiff [3] | Pyramid         | Vanilla with Multi-scale L1                               | 27.090 | 0.879 | 0.100 |\n| Ours       | Vanilla (U-Net) | Rank-based modeling with basic KMeans clustering          | 27.336 | 0.874 | 0.097 |\n| Ours++     | Vanilla (U-Net) | Rank-based modeling with advanced Hierarchical clustering | **27.697** | **0.880** | **0.092** |\n\nAdditionally, we also want to note that our approach enhances the diffusion process through a plug-and-play regularization term rather than the modification of the network structure adopted in [3]. As a result, there exists potential for integrating our regularization term into [3] to achieve superior reconstruction. We anticipate exploring this integration once the training code from [3] is released.\n\n### ***5** Response to Question 1 (Q1): DDPM or DDIM*\n\nDDPM has shown its great capability in producing high-quality images, to explore the effectiveness of our global structure-aware regularization within diffusion models,  we thus integrate our regularization terms into this foundational diffusion model,  exhibiting superior performance relative to state-of-the-art (SOTA) methods. This clearly highlights the potential of our approach. With regard to DDIM, it aims to accelerate the sampling process by generalizing DDPM using non-Markovian diffusion processes. In essence, DDIM can be considered as a generalized version of DDPM, and they have the same training procedure. Consequently, we are confident that our regularization terms could be seamlessly integrated into DDIM, offering significant potential for achieving exceptional performance."}, {"review_id": "WS8aPOwfJE", "reviewer": "Reviewer_gyFW", "summary": "This work proposes a diffusion-based low-light image enhancement framework that exhibits good performance in different benchmarks. A rank-informed regularization term during training and uncertainty-weighted loss is proposed.", "questions": "- Please refer to the weakness section.\n\n- While the weighting strategy of the training loss is reasonable for idea cases. But for real-captured high-resolution images where small misalignments may exist, will it lead to some side effects?\n- Could you explain why the constraint need only be added to the singular value, while no need for the singular vector?", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This work proposes a diffusion-based low-light image enhancement framework that seems to exhibit good performance in different benchmarks.\n\n\n", "weaknesses": "- The motivation of the \"Non-local patch-based matrix rank modeling\" seems not clear.\n\nAccording to my understanding, the proposed \"Non-local patch-based matrix rank modeling\" is only used during training. How does it make the model have the capacity to be aware of the global structure during inference?\n\nBesides, during the training using paired training data, what's the necessity of using the proposed method to measure the structure consistency between Xt−1 and X0? The difference be directly calculated using metrics like L1, L2, SSIM, or feature-level similarities.\n\n- How does the proposed method increase the training overhead?\n\n- Some formulation is not self-contained. For example, where is the definition of \"\\alpha_1^2\" in Eq. (4)?  How to set the value of k_t?\n\n- It's better to illustrate which dataset/subset of the dataset is used to train the proposed method and other competitors. Since it greatly affects the performance of the model. And including a comparison of inference cost is better.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **[Responses to Reviewer gyFW]**\n\n### ***1**  Response to Weakness 1 (W1): Relation between Global Structure Modeling and Non-local Patch Rank Modeling*\n\nIt is imperative to highlight that by \"global structure-aware,\" we are referring to the network's ability to account for various patterns, such as non-local patches, rather than solely focusing on individual pixels. By regularizing the matrix rank formed from a cluster of these non-local patches, our model can concurrently consider structures from various positions, resulting in a strong capability of global structure modeling.\n\nSecond, the characteristics and properties of a neural network are primarily acquired during the training phase. In our approach, we incorporate specific regularization to encourage the network weights to reconstruct the global structure. During inference, the network can still effectively exhibit the learned function, by the established weight pattern in the training phase.\n\n### ***2**   Response to W2: Theoretical Soundness*\n\nWe answer this question from two levels: why apply regularization between $X_{t-1}$ and $X_0$, and then why choose the rank-based global structure-aware regularization.\n\n**1) Analysis of applying regularization between $X_{t-1}$ and $X_0$, from two distinct perspectives.**\na) **Trajectory Curvature**. Recent works [C1, C2] show that an efficient and effective ODE-based generative model, e.g., diffusion models, should have a **lower curvature (straight) trajectory**, which could help them stably converge to high-quality solutions. Our regularization term could indeed minimize the trajectory curvature. We further illustrate such properties by corresponding experimental results of the reverse trajectory curve in the additional pdf file.\nb) **Accumulated Error**. During inference, each step is established on the previous prediction. Thus, accumulated errors inevitably exist. Through regularizing $X_{t-1}$ towards the ground-truth, we expect that each reconstruction step would consistently move towards GT images, which may also potentially minimize the influence of such accumulated error.\n\n**2) Analysis of choosing rank-based global-structure aware regularization**\nWhile the current approaches often involve pixel-wise treatments, inherent global structures can be overlooked to some extent. Modeling such global structures potentially augments the performance [C3]. Additionally, pixel-wise regularization terms, such as L1, L2, and SSIM, do not adequately encapsulate nonlocal structures. However, regularization within the feature domain is usually confined within a local region due to the kernel perceptual regions.  Moreover, regularizing features can result in fluctuations in the back-propagated gradients, thereby impeding network training.\n\nFinally, the following table also verifies the above analyses, i.e., the necessity of introducing regularization between $X_{t-1}$ and $X_0$, and the advantage of our rank-based model.\n\n|Method| PSNR| SSIM| LPIPS|\n|--|--|--|--|\n| Baseline| 26.02| 0.859| 0.123 |\n| L1 Reg. | 26.45| 0.868| 0.101 |\n| L2 Reg. | 26.53| 0.869| 0.102 |\n| SSIM Reg.| 26.23| 0.870| 0.101|\n| Perceptual Reg.| 26.42| 0.863| 0.099|\n| Rank Reg. (Ours) | **27.02** | **0.872** | **0.097** |\n\n[C1] S. Lee, et al., Minimizing trajectory curvature of ode-based generative models, ICML'23.\n\n[C2] X. Liu, et al., Flow straight and fast: Learning to generate and transfer data with rectified flow,  ICLR'23.\n\n[C3] S. Gu, et al., Weighted nuclear norm minimization with application to image denoising, CVPR'14.\n\n### ***3**  Response to W3: Training Overhead* \n\nOur method significantly improves the quality of enhanced images by exploring global structures. Still, it simultaneously introduces little additional computational consumption, only about 0.03 extra seconds (from 0.105 to 0.135 S) to train for one iteration on the LOLv1 dataset. Besides, no additional computational cost is introduced during inference.\n\n\n### ***4**   Response to W4: Notions*\n\nSorry for the confusion caused. $\\alpha_t$ in Eq. (4) is the same as the $\\alpha_t$ in Eq. (2) and **Algorithm 1**, which is defined in Line 88. Moreover, $k_t$ means $\\alpha_t^2$. We will clarify them in the final version.\n\n### ***5**   Response to W5: Training Dataset*\n\nIn alignment with existing methods under comparison, we conducted the network training utilizing the respective training datasets. For instance, during the LOLv1 testing experiment, we base the network's training on the LOLv1 training set. Additionally, for a comprehensive understanding of the inference cost, **we kindly refer the reviewer to the first section of the supplementary.**\n\n### ***6**   Response to Question 1 (Q1): Treating Misalignments*\n\nWith misalignments, a decline in reconstruction performance across all methods might be observed. However, generative models, such as diffusion models, just rely on conditioning from input images rather than directly translating inputs to the reconstruction, as seen in regression-based methods. Therefore, their requirements for alignment might be less stringent. Furthermore, the uncertainty module in our method may potentially address misalignments by moderating the regularization on misaligned pixels.\n\n### ***7**   Response to Q2: Why not Regularize Singular Vectors*\n\nWe regularize the global structure consistency between two images. With such a regularization term, it is expected that the similarity of different non-local patches in $X_{t-1}$ and $X_{0}$ should be identical, or a same component proportions of different images. Thus, by approaching singular values, we could minimize the gaps between $X_{t-1}$ and $X_{0}$. Note that simultaneously regularizing the singular values and vectors to be the same is equal to regularizing all patches to be pixel-wise similar. Moreover, as listed in **the second response to Reviewer gyFW**, the L1 regularization produces inferior performance. Thus, we do not further regularize singular vectors.\n"}, {"review_id": "K8vCpFyr5Z", "reviewer": "Reviewer_9ew6", "summary": "Aiming at better low-light image enhancement performance with diffusion models, this paper proposes a global structure-aware regularization utilizing the intrinsic non-local structural constituents of image data. An uncertainty map is incorporated into the diffusion model to ease the strict constraints on indeterminate regions. Experimental results demonstrate the effectiveness of the proposed components.", "questions": "Please see the strengths and weakness section.", "limitations": "Several challenging cases are provided. The limitations of the proposed approach sholud be discussed in more detail.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper incorporates global structure regularization via matrix rank modeling into the diffusion models, showing significant performance improvement compared with state-of-the-art methods.\n2.\tThe paper is well-written, and the presentation is pretty good.\n", "weaknesses": "1.\tThe contribution is not convincing enough. The generality of the proposed global structure-aware regularization based diffusion models is not verified, and the main idea of the adopted uncertainty is not original in this paper.\n2.\tAdditionally, the main concern about this paper is that there are many unclear details. For example, what train data is used in this paper? What is the baseline setting in Table.5? Is the inference time of all the methods also performed on RTX 3080 GPUs?\n3.\tMore perceptual metrics (e.g., LPIPS, MUSIQ) could be reported for better evaluation.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **[Responses to  Reviewer 9ew6]**\n\n### ***1** Response to W1: Paper Contributions*\n\nThanks for the comments. We conducted additional experiments in terms of image super-resolution ($16\\times$ SR on CelebA-HQ dataset) to validate the generality of our method on different tasks.\nThe quantitative results in the following table show that our global structure-aware regularization can improve the reconstruction performance, demonstrating its generality to some extent. The results of the SOTA baseline [C1] are also provided for comparison.\n\n| Methods                                        | PSNR  | SSIM  |\n|------------------------------------------------|-------|-------|\n| IDM [C1]                                       | 24.01 | 0.710 |\n| Ours w/o global structure-aware regularization | 23.94 | 0.713 |\n| Ours                                           | 24.52 | 0.728 |\n\nBesides, it is also worth mentioning that as discussed in the **first response to Reviewer U5ii**, the performance of our method is further improved significantly by applying advanced clustering methods for grouping non-local patches, i.e., the overall PSNR value increases about 0.7 dB, compared to the other clustering algorithms. We also refer the reviewer to the **second response to Reviewer gyFW** for the detailed analysis of our method and additional experimental results of various regularization terms.\n\nIn all, we posit that our contributions will pave the way for fresh perspectives on diffusion models for low-level image processing tasks. And our method improves the SOTA performance of low-light image enhancement to a new higher level, which will contribute to the community.\n\n[C1] S. Gao, et al., Implicit diffusion models for continuous super-resolution, CVPR'23.\n\n\n### ***2** Response to W2: Setting Details*\n\nSorry for the confusion caused. We will complement the details.\n\n1) We kindly remind the reviewer to refer to Section 4.1 of the manuscript for the details of the used datasets. For example, we utilized 485 pairs of low/normal-light images for training and 15 pairs for testing on the LOLv1 dataset.\n2) The baseline was constructed by removing the non-local patch-based matrix rank modeling and uncertainty-guided regularization. The baseline corresponds to the basic diffusion model.\n3) The inference time per image of all the recent SOTA methods in Table S1 of Supplementary Material was conducted using an RTX3080 GPU on the same server.\n\n### ***3** Response to W3: Comparison of More Perceptual Metrics*\n\nIn the manuscript, we have indeed provided the LPIPS comparisons in Table 1. For better evaluation, we further adopted MUSIQ (trained on KonIQ-10k) for assessing the enhancement quality. The following table shows quantitative comparisons of the recent SOTA methods in terms of MUSIQ, which consistently demonstrate the superiority of our method.\n\n| Datasets        | SNR-Aware         | LLFormer | Ours      |\n|--------------|-----------------|---------|--------|\n| LOLv1           | 61.00             | 58.26    | **71.74** |\n| LOLv2-real      | 57.76             | 51.13    | **69.34** |\n| LOLv2-synthetic | 63.18             | 62.84    | **64.51** |\n"}, {"review_id": "Ho7CsGKcMy", "reviewer": "Reviewer_U5ii", "summary": "This paper presents an innovative and efficacious approach to regularization within the domain of diffusion models, with particular utility for low-light image reconstruction. The method ingeniously takes into account the latent global structure of images. To achieve this, an image is first divided into patches, and clustering algorithms are then employed to group analogous patches, where the matrix rank is further utilized to minimize the discrepancy between the intermediate reconstruction and the final output. Besides, uncertainty is used to boost performance.", "questions": "1) Are other advanced clustering algorithms more effective in enhancing reconstruction performance? This comparison could provide valuable insights into the optimal algorithm selection for this specific task.\n2) For the ablation studies in Table 5 (Figure 5 should be Table 5), the authors are suggested to add two more settings, i.e., remove  (b) and (c) and keep (a); remove (c) and keep (a) and (b), to directly validate the adaptive rank regularization, which is the main contributions. \n3) In Figure 6, please add the input and GT images. It is better to use different symbols to distinguish the learnable closed-form samples and the input of the forward process.\n4) Can other regularization terms (e.g., the L1 loss between X_0 and X_t) improve the original diffusion model?\n5) There is potential for the proposed reconstruction algorithm to benefit diffusion models in tasks beyond the current scope of the study. It would be beneficial for the authors to explore this further, extending the validations to other potential applications. At least the authors can discuss this issue to improve the paper. \n6) There are some grammar errors, e.g., line 163: a  an", "limitations": "Maybe a little additional computation consumption during the training phase.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1) The novel perspective opens up a promising avenue for enhancing the performance of diffusion models, representing an important contribution to the field. Also, it may have great potential for other diffusion-based image/video processing tasks. \n2) The introduction of this form of regularization is a groundbreaking shift for diffusion models, distinguishing this work from existing literature.\n3) The experimental validations lend strong support to the efficacy of the proposed regularization term, and the proposed method achieves the current SOTA.\n4) The paper is well-written, and the supplementary material includes the code.", "weaknesses": "See the detailed questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **[Responses to Reviewer U5ii]**\n\n### ***1** Response to Weakness 1 (W1): Clustering Algorithm*\n\nThanks for your valuable comment. As shown in the following table, our method benefits from advanced clustering algorithms greatly, and the PSNR value is further improved by 0.7 dB, compared with K-means adopted in our manuscript. These advancements further underscore the prospective utility of our regularization terms in enhancing diffusion models.\n\n| Clustering algorithms   | PSNR  | SSIM  | LPIPS |\n|:-------------------------:|:------|-------|-------|\n| K-Means                 | 27.02 | 0.872 | 0.097 |\n| Spectral clustering     | 27.41 | 0.876 | 0.095 |\n| Gaussian Mixture Model  | 27.55 | 0.877 | 0.095 |\n| Hierarchical clustering | 27.70 | 0.880 | 0.092 |\n\n### ***2** Response to W2: Ablation Settings*\n\nThanks for the comments. We experimentally verified the importance of our adaptive rank regularization according to the suggested settings. As shown in the following table, our approach demonstrates a significant boost in the baseline, elevating it from 26.02 dB to 27.02 dB, when eliminating (c) uncertainty-guided regularization. This outcome strongly suggests that our major contribution lies in the introduction of adaptive rank regularization.\n\n| (a)      | (b)      | (c)      | PSNR  | SSIM   | LPIPS  |\n|----------|----------|----------|:------|:-------|:-------:|\n| ✕ | ✕ | ✕ | 26.02 | 0.8593 | 0.1226 |\n| ✓ | ✕ | ✕ | 26.63 | 0.8701 | 0.1016 |\n| ✓ | ✓ | ✕ | 27.02 | 0.8722 | 0.0973 |\n| ✓ | ✓ | ✓ | 27.34 | 0.8739 | 0.0969 |\n\n### ***3** Response to W3: Symbol Issue*\n\nThanks for the comments. we will certainly add corresponding images to have a better comparison. For symbolic representations, we intend to modify the learnable closed-form sample to enhance its difference from the input.\n\n### ***4** Response to W4: Other Regularization Terms*\n\nAs experimentally verified in the table of the **second response to Reviewer gyFW**, other regularization terms, such as L1 and L2, can also enhance the baseline, e.g.,  the L1 regularization could improve the baseline diffusion model from 26.02 dB to 26.45 dB, and the L2 regularization from 26.02 dB  to 26.53 dB. However, their effectiveness is still much lower than our rank-based regularization improving the baseline diffusion model from 26.02 dB to 27.02 dB, because both L1 and L2 regularization manners fail to fully capture nonlocal structures or patterns within an image and cannot explicitly characterize the properties of the structure. We also refer the reviewer to the response for more detailed analysis.\n\n### ***5** Response to W5: General Validation*\n\nAs shown in the **first response to Reviewer 9ew6**, we also validated the effectiveness of the proposed regularization terms on the image super-resolution task. Upon experiments, it is evident that, by incorporating the proposed regularization term, our approach significantly surpasses the state-of-the-art (SOTA) diffusion SR method by a margin of 0.51 dB. It is widely recognized that SR represents a cornerstone in the domain of image reconstruction. Consequently, this enhancement underscores the profound effectiveness and potential of our methodologies. Besides, in the conclusion section of the final version, we will discuss its potential in other low-level image processing tasks as future work.\n\n### ***6** Response to W6: Typos*\n\nThanks for the comments. We will correct those typos in the final version.\n"}, {"review_id": "YSGLI1R6yj", "reviewer": "Reviewer_2x92", "summary": "This work introduces a low-light image enhancement based on diffusion model. By incorporating global structure-aware regularization, this work achieves more promising performances than existing methods. ", "questions": "See my question above.", "limitations": "See my questions above. ", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "By considering the global structure during the diffusion process, this work achieves better performance. ", "weaknesses": "Despite its better performance, I am very sure that this work cannot advance this field. It does not solve a problem in a new scenario and it just make incrimental revision on top of established framework. My concerns for this work can be concluded as follows:\n1. The authors claim that a naive implementation of the diffusion model alone is inadequate for effectively resolving the low-light image enhancement, while it does not prove this claim. All baselines considered in the experiments are non-diffusion based approaches. Thus, it is difficult to say whether a naive implementation is enough for the problem solving. \n2. In line 151, the autors claim that the rank of matrix is an effective measurement of the global structures. This is a very important assumptions in this place. This part needs more experiments and justifactions. Or, the authors should provide very solid references for this claim in this place.\n3. Except the matrix ranking, several approachs have been proposed to better represent the global structures including some deep learning features. The authors should compare with them. \n4. Why is it important to use global structure in this place? If U-Net is a substantial structure of this proposed framework, it is able to capture the global structure during its network propagation. Thus, why do this work design an additional module to capture global structure. \n5. I am not sure about the effectiveness of uncertainty in this place. I cannot find Table 5 mentioned in this manuscript. \n6. This work just puts several different image processing modules together without enough insights. It is easy to come up with such an idea. The non-local solutions have been discussed in traditional solutions. I donot know why they choose to combine it with diffusion model. Besides, weighted loss has been considered for a large number of methods. Using it here cannot be considered as a new contribution. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **[Responses to Reviewer 2x92]**\n\n### ***1** Response to Weakness 1 (W1): Baseline Model*\n\nWe refer the reviewer to the results of ablation studies depicted in Figure 5 on page 9, where the first case corresponds to the baseline diffusion model. It only achieves\n26.02 dB, which is noticeably lower than the SOTA method SNR-Aware's achievement of 26.70 dB.\n\n### ***2** Response to W2: Theoretical Soundness of Rank-based Regularization*\n\nFirst, we clarify that as stated in Lines 112 - 114, the ``global structure\" means that the non-local similar patterns.  We have indeed cited some related works [5, 48] in Sec 3.1.1  of the manuscript. Moreover, [41 , 16] also acknowledge the effectiveness of a matrix rank to model the image global structure. Besides, it is somewhat common sense that the rank of a 2D matrix measures the correlation between its elements in a global manner rather than an element-wise manner.\n\n### ***3** Response to W3: Different loss Terms*\n\nWe made comparisons with the perceptual loss, i.e., we replaced our rank-based regularization with the perceptual loss. The quantitative results listed in the following table show ours outperforms the perceptual loss to a large extent. We also refer the reviewer to **the second response to Reviewer gyFW** for the results of more loss terms.\n\n| Methods                  | PSNR      | SSIM      | LPIPS     |\n|:--------------------------:|-----------|-----------|:----------|\n| Baseline                 | 26.02     | 0.859     | 0.123     |\n| Perceptual loss          | 26.42     | 0.863     | 0.099     |\n| Our matrix rank modeling | **27.02** | **0.872** | **0.097** |\n\n### ***4** Response to W4: Architecture or Loss Terms for Structure-aware Diffusion Process*\n\nWe note numerous factors influence network performance, where the network structure represents merely one of these variables. A network may not reach its full potential without the appropriate configuration of these diverse training settings. While the U-Net architecture is adept at integrating feature maps across multiple resolutions, the vanilla loss doesn't thoroughly harness these properties. We also refer the reviewer to the **second response to Reviewer gyFW** for the detailed quantitative and theoretical analyses of introducing adaptive rank-based regularization into the diffusion model.\n\nFor your convenience, we have pasted some of the content.\n\n\" .....\nWhile the current approaches often involve pixel-wise treatments, inherent global structures can be overlooked to some extent. Modeling such global structures potentially augments the performance [C3]. Additionally, conventional pixel-wise regularization terms, such as L1, L2, and SSIM, do not adequately encapsulate nonlocal structures. However, regularization within the feature domain is usually confined within a local region due to the kernel perceptual regions.  Moreover, regularizing features can result in fluctuations in the back-propagated gradients, thereby impeding network training.\n\nFinally, the results in the following table also verify the above analyses, i.e., the necessity of introducing regularization between $X_{t-1}$ and $X_0$, and the advantage of our rank-based model.\n\n|Method| PSNR| SSIM| LPIPS|\n|--|--|--|--|\n| Baseline| 26.02| 0.859| 0.123 |\n| L1 Reg. | 26.45| 0.868| 0.101 |\n| L2 Reg. | 26.53| 0.869| 0.102 |\n| SSIM Reg.| 26.23| 0.870| 0.101|\n| Perceptual Reg.| 26.42| 0.863| 0.099|\n| Rank Reg. (Ours) | **27.02** | **0.872** | **0.097** |\n\n[C3] S. Gu, et al., Weighted nuclear norm minimization with application to image denoising, CVPR'14.\n\"\n\n\n### ***5** Response to W5: Missing of Table 5*\n\nSorry for our mistake.  Current Figure 5 indicates the ablation studies of different loss terms. We will correct the label in the final version.\n\n### ***6** Response to W6: Contributions of the Method*\n\n**We beg to differ in your evaluation of our method. Our method is by no means an assembly of prior art.** Our major contribution lies in introducing global structure-aware regularization into the learning process of the diffusion model, which **Reviewer U5ii** acknowledges. Current diffusion models, with their naive loss terms, cannot fully capture the global properties. Thus, we introduce non-local patch-based matrix rank modeling to address such an issue. Moreover, our regularization method tends to minimize the trajectory curvature, which potentially helps image reconstruction [C1,C2] and leads to outstanding performance compared to SOTA methods. We refer the reviewer to the **second response to Reviewer gyFW** for the detailed analyses of the necessity and advancement of introducing rank-based regularization into the diffusion process for capturing the global structure and the theoretical analysis in the attached pdf file.\n\nBesides, it is also worth mentioning that, as discussed in the **first response to Reviewer U5ii** , the performance of our method is further improved significantly by applying advanced clustering methods for grouping non-local patches, i.e., the overall PSNR value increases about 0.7 dB, compared to those reported in our manuscript.\n\nWe posit that our contributions will pave the way for fresh perspectives on diffusion models for low-level image processing tasks. And our method improves the SOTA performance of low-light image enhancement to a higher level, which will contribute to the community.\n\n[C1] S. Lee, et al., Minimizing trajectory curvature of ode-based generative models, ICML'23.\n\n[C2] X. Liu, et al., Flow straight and fast: Learning to generate and transfer data with rectified flow,  ICLR'23."}]

Author rebuttal:
### General Purpose

We thank all reviewers for your time, constructive comments, and recognition of our work. We believe all concerns have been clearly and directly addressed. Here we also want to summarize a few key clarifications concerning the contributions of our work.

Our **MAJOR** contribution lies in introducing an adaptive rank-based regularization term to promote the baseline diffusion model to capture the global structure. Specifically, we aim to minimize the discrepancy between $X_{t-1}$ and $X_{0}$ to establish the reverse trajectory with reduced curvature, which can potentially benefit high-quality samples generation [C1, C2]. Moreover, during this process, we incorporate rank-based modeling focused on non-local similar patches, thereby exploiting the global structure modeling capabilities in diffusion models. We have experimentally and precisely validated the necessity of regularization between $X_{t-1}$ and $X_{0}$, as well as the effectiveness of rank-based modeling. See the figures in the submitted pdf file and the second response to **Reviewer gyFW** for the detailed analyses. Besides, as experimentally verified in the **first response to Reviewer U5ii**, by simply leveraging advanced clustering algorithms for grouping image patches, the PSNR value of our approach further increases about 0.7 dB, emphasizing the potential utility of our regularization terms in enhancing diffusion models.

We posit that our contributions will pave the way for fresh perspectives on diffusion models for low-level image processing tasks. And our method improves the SOTA performance of low-light image enhancement to a new higher level, providing a promising benchmark to this community.

Last but not least, **we will make the reviews and author discussion public regardless of the final decision**.  Besides, we will include the newly added experiments and analysis in the final manuscript/supplementary material.


Meta-review: This paper presents a novel regularization approach within the domain of diffusion models for low-light image enhancement. The method utilizes the rank of clustered non-local image patches as a measure of global structure. Experimental results show good performance. The paper received five reviews. Most of the reviews are positive, except that one reviewer has concerns on its applications to other low-level image enhancement tasks (other than low-light images) and the issues with the paper writing/citation. 

The author did a thorough rebuttal and carefully addressed most of the reviewers' question. The main contribution, i.e., introducing the rank-based constraint to enforce global structure for diffusion-based image restoration seems novel and effective. In the rebuttal, the authors also tried advanced clustering algorithms as well as other image restoration tasks such as super-resolution, and the results are good. While the AC agreed this rank-based regularization can be used for general image restoration tasks, the paper itself is self contained as a complete work, considering the novelty in the idea, the SOTA experimental results in low-light image enhancement, and the added results and analysis in the rebuttal.

Taking into account all the reviews, the rebuttal, and the discussions, the AC decided to accept the paper.
Decision: Accept (poster)