SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets. We employed seven commonly-used LLIE benchmark datasets for evaluation, including LOLv1 [45], LOLv2 [54], DICM [11], LIME [6], MEF [14], NPE [39], and  $\mathrm{VV}^2$ . Specifically, LOLv1 contains 485 low/normal-light image pairs for training and 15 pairs for testing, captured at various exposure times from the real scene. LOLv2 is split into two subsets: LOLv2-real and LOLv2-synthetic. LOLv2-real comprises 689 pairs of low-/normal-light images for training and 100 pairs for testing, collected by adjusting the exposure time and ISO. LOLv2-synthetic was generated by analyzing the illumination distribution of low-light images, consisting of 900 paired images for training and 100 pairs for testing. In accordance with the settings outlined in the recent works [51, 43, 40], we trained and tested our models on LOLv1 and LOLv2 datasets separately. The DICM, LIME, MEF, NPE, and VV datasets contain several unpaired real low-light images, which are used only for testing.
Evaluation metrics. We adopted both full-reference and non-reference image quality evaluation metrics to evaluate various LLIE approaches. For paired data testing, we utilized peak signal-to-noise ratio (PSNR), structural similarity (SSIM) [44], and learned perceptual image patch similarity (LPIPS) [57]. For datasets such as DICM, LIME, MEF, NPE, and VV, where paired data are unavailable, we adopted only the naturalness image quality evaluator (NIQE) [20].
Methods under comparison. We compared our method with a great variety of state-of-the-art LLIE methods, i.e., LIME [6], RetinexNet [45], KinD [60], Zero-DCE [5], DRBN [52], KinD++ [59],
$^{2}$ https://sites.google.com/site/vonikakis/datasets
Table 1: Quantitative comparisons of different methods on LOLv1 and LOLv2. The best and second-best results are highlighted in **bold** and **underlined**, respectively. "↑" (resp. "↓") means the larger (resp. smaller), the better. Note that we obtained these results either from the corresponding papers, or by running the pre-trained models released by the authors, and some of them lack relevant results on the LOLv2-synthetic dataset.
Methods	LOLv1	LOLv2-real	LOLv2-synthetic	Param (M)
PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓
LIME [6] TIP'16	16.760	0.560	0.350	15.240	0.470	0.415	16.880	0.776	0.675	-
Zero-DCE [5] CVPR'20	14.861	0.562	0.335	18.059	0.580	0.313	-	-	-	0.33
EnlightenGAN [8] TIP'21	17.483	0.652	0.322	18.640	0.677	0.309	16.570	0.734	-	8.64
RetinexNet [45] BMVC'18	16.770	0.462	0.474	18.371	0.723	0.365	17.130	0.798	0.754	0.62
DRBN [52] CVPR'20	19.860	0.834	0.155	20.130	0.830	0.147	23.220	0.927	-	2.21
KinD [60] MM'19	20.870	0.799	0.207	17.544	0.669	0.375	16.259	0.591	0.435	8.03
KinD++ [59] IJCV'20	21.300	0.823	0.175	19.087	0.817	0.180	-	-	-	9.63
MIRNet [55] TPAMI'22	24.140	0.842	0.131	20.357	0.782	0.317	21.940	0.846	-	5.90
LLFlow [43] AAAI'22	25.132	0.872	0.117	26.200	0.888	0.137	24.807	0.9193	0.067	37.68
LLFormer [40] AAAI'23	25.758	0.823	0.167	26.197	0.819	0.209	28.006	0.927	0.061	24.55
SNR-Aware [51] CVPR'22	26.716	0.851	0.152	27.209	0.871	0.157	27.787	0.941	0.054	39.13
Ours	27.839	0.877	0.091	28.818	0.895	0.095	28.670	0.944	0.047	17.36
Figure 3: Visual comparisons of the enhanced results by different methods on LOLv1 and LOLv2.
EnlightenGAN [8], MIRNet [55], LLFlow [43], DCC-Net [61], SNR-Aware [51], and LLFormer [40].
Implementation details. We trained our models for 2M iteration steps with PyTorch. We employed an Adam optimizer [10] with a fixed learning rate of  $1 \times 10^{-4}$  without weight decay. We applied an exponential moving average with a weight of 0.9999 during parameter updates. During training, we utilized a fine-grained diffusion process with  $T = 500$  steps and a linear noise schedule with the two endpoints of  $1 \times 10^{-4}$  and  $2 \times 10^{-2}$ . The patch size and batch size were set to 96 and 8, respectively. The hyper-parameter  $\lambda$  was empirically set to 10.
Results on LOLv1 and LOLv2. Table 1 shows the quantitative results of different LLIE methods, where it can be observed that our method consistently achieves the best performance over all the compared methods in terms of PSNR, SSIM, and LPIPS metrics. Particularly, the remarkable improvement on LPIPS provides compelling evidence of the superior perceptual quality of our method, as shown in Fig. 3, where our method effectively suppresses artifacts and reveals image details, leading to visually appealing results that are more faithful to the original scene.
Figure 4: Visual comparisons of different methods on DICM, LIME, MEF, NPE, and VV datasets.
Results on DICM, LIME, MEF, NPE, and VV. We directly applied the model trained on the LOLv2-synthetic dataset to these unpaired real-world datasets. From Table 2, we can observe that our method yields better NIQE scores than all the other competitors, demonstrating its stronger generalization ability. In addition, a visual comparison of enhancement results from various methods, as showcased in Fig. 4, underscores the superior performance of our approach. Evidently, our model adeptly adjusts the illumination conditions, optimally enhancing the visibility in low-light areas while vigilantly circumventing over-exposure. This
Table 2: Quantitative comparisons of different methods on DICM, LIME, MEF, NPE, and VV datasets in terms of NIQE, where smaller values lead to better performance.
Methods	DICM	LIME	MEF	NPE	VV
Zero-DCE [5]	4.58	5.82	4.93	4.53	4.81
EnlightenGAN [8]	4.06	4.59	4.70	3.99	4.04
RetinexNet [45]	4.33	5.75	4.93	4.95	4.32
KinD [60]	3.95	4.42	4.45	3.92	3.72
KinD++ [59]	3.89	4.90	4.55	3.91	3.82
SNR-Aware [51]	6.12	5.93	6.44	6.44	11.50
DCC-Net [61]	3.70	4.42	4.59	3.70	3.28
Ours	3.28	4.32	3.40	3.55	2.60
successful balance corroborates the superior efficacy of our proposed method.
Table 3: Ablation studies of effectiveness of various clustering algorithms used in our method.
Clustering Algorithms	Baseline	K-Means	Spectral clustering	Gaussian Mixture Model	Hierarchical clustering
PSNR↑	26.02	27.02	27.41	27.55	27.70
SSIM↑	0.859	0.872	0.876	0.877	0.880
LPIPS↓	0.123	0.097	0.095	0.095	0.092
Table 4: Ablation studies of the effectiveness of choosing our rank-based global structure-aware regularization.
Regularization	Baseline	L1 loss	L2 loss	SSIM	Perceptual loss	Our rank-based modeling
PSNR↑	26.02	26.45	26.53	26.23	26.42	27.70
SSIM↑	0.859	0.868	0.869	0.870	0.863	0.880
LPIPS↓	0.123	0.101	0.102	0.101	0.099	0.092
Clustering algorithms. We investigated how the clustering method used for finding similar patches affects the quantitative performance on LOLv1. As shown in Table 3, our method generally benefits from different clustering algorithms. Especially, when employing advanced hierarchical clustering, the PSNR value experiences an improvement of approximately  $0.7\mathrm{dB}$ , compared to the results achieved with the K-Means clustering. This further underscores the prospective utility of our
responding module is unused (resp. used). (a) Non-local patch-based matrix rank modeling, (b) Gradually injecting structure-aware regularization, and (c) Uncertainty-guided regularization. Note that (b) is applicable only if (a) is utilized.
Table 5: Quantitative results of the ablation stud- Figure 5: Visual results of the four settings of ties on LOLv1. "X" (resp. "√") means the cor- the ablation studies in Table 5.
(a)	(b)	(c)	PSNR↑	SSIM↑	LPIPS↓
1)	X	X	X	26.02	0.8593	0.1226
2)	X	X	✓	26.79	0.8708	0.1049
3)	✓	X	X	27.02	0.8753	0.0980
4)	✓	✓	X	27.70	0.8795	0.0923
5)	✓	✓	✓	27.84	0.8774	0.0905
regularization term in enhancing diffusion models with advanced clustering algorithms. Consequently, we adopted advanced hierarchical clustering in our approach for all subsequent experiments.
Regularization terms between  $X_{t-1}$  and  $X_0$ . We delved into the impact of utilizing varying regularization terms between  $X_{t-1}$  and  $X_0$  on the enhancement of diffusion models within LOLv1. Table 4 offers insights that although conventional pixel-wise regularization terms such as  $L_1$ ,  $L_2$ , and SSIM can moderately elevate the performance of the baseline, their enhancements pale in comparison to our rank-based modeling regularization. The primary reason is the inability of these regularization terms to fully encapsulate the nonlocal structures or patterns inherent within an image. These outcomes distinctly highlight the superiority of our rank-based model in capturing global structures.
Non-local patch-based matrix rank modeling & adaptive regularization schedule. We established a baseline by demounting the non-local matrix rank-based representation learning regularization. By comparing rows 2),3),4),and 5) in Table 5, it can be observed that this module equipped with our adaptive regularization schedule significantly boosts the enhancement performance. However, this module without adopting the adaptive regularization schedule only achieves minor improvement. Such observations demonstrate the necessity of our adaptive regularization schedule and the effectiveness of non-local rank-based representation learning.
Furthermore, as depicted in Fig. 6, we elaborate
Figure 6: Visual comparisons of the distribution of the singular values of a cluster from enhanced images by our method with and without global structure-aware regularization. Note that the patches contained in the cluster are highlighted with a white box.
on the matrix rank across clusters from several samples, thereby offering a clear visualization of the effectiveness of this regularization. Empirical findings affirm that implementation of our proposed regularization notably narrows the gap of singular values between the reconstruction output and the ground truth, thereby substantiating the indispensability of this regularization approach and validating its strategic integration into the diffusion process.
Uncertainty-guided regularization. The effectiveness of this regularization is substantiated through a comparative analysis of rows 1) and 2) of Table 5. Notably, the implementation of this regularization is evidenced to bolster the performance of image enhancement. Such improvement, however, is confined to pixel-wise metrics, e.g., PSNR and SSIM, and does not extend to global visual metrics
LLFormer
Ours
Figure 7: Illustration of challenging scenarios, where both the proposed method and other existing methods exhibit limitations in achieving visually pleasing reconstructions.
such as LPIPS (visualization as Fig. 5). This observation implies that the incorporation of uncertainty-based regularization in isolation yields incremental advancements but fails to induce substantial modifications to the structure of the reconstructed images. The findings, therefore, underscore the necessity of an integrated approach that combines uncertainty-based regularization with global structure-aware regularization to achieve significant improvements in both local pixel-level and global structural aspects of image enhancement.
4.4 Challenging Cases
While the proposed method has demonstrated notable advancements in performance, as showcased in the previous sections, it is important to acknowledge that certain challenging scenarios persist within the domain of low-light enhancement tasks. Fig. 7 illustrates two challenging examples that highlight the intricacy of the task at hand. In these examples, it is evident that certain regions of the input image are under extreme low-light conditions, to the point where any trace of texture becomes indiscernible. This lack of visual information poses a significant challenge, causing both our proposed method and other state-of-the-art techniques to struggle in generating visually compelling details. To tackle these challenges, we could identify a subset of extremely low-light patches and apply robust regularization to them to improve reconstruction performance.
We have introduced a diffusion model-based approach for enhancing low-light images effectively. The cornerstone of our approach lies in the innovative training strategy of diffusion models from the perspective of minimizing the ODE-trajectory curvature, which involves the global structure-aware rank-based regularization on learnable closed-form samples and uncertainty-based learning of latent noises. Recognizing the potential adverse effects of regularization within the early stage of diffusion models, we have thoughtfully devised a progressively adaptive regularization schedule. This strategy ensures a nuanced, step-wise infusion of structure-aware regularization, preserving the inherent structural integrity in the reconstructed samples. As we continue to refine our approach, we anticipate further improvements and look forward to the potential applications and advancements these developments will facilitate in broader fields.
While our work exhibits admirable performance compared with contemporary methods, we acknowledge the inherent computational burden associated with diffusion-based methods—attributable to their iterative noise removal in a reverse Markov chain. Nonetheless, thanks to the latest strides in the field [33, 35], it is anticipated that the diffusion models may soon be capable of delivering accelerated reconstructions through a minimal number of iterative steps.
This work was supported in part by the Hong Kong Research Grants Council under Grant CityU 11218121, in part by the Hong Kong Innovation and Technology Fund under Grant MHP/117/21, and in part by Hong Kong University Grants Committee under Grant UGC/FDS11/E02/22, and in part by the Natural Science Foundation of Shandong Province under Grant ZR2022ZD38.
[58] Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and Stan Z Li. S3fd: Single shot scale-invariant face detector. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 192-201, 2017.
[59] Yonghua Zhang, Xiaojie Guo, Jiayi Ma, Wei Liu, and Jiawan Zhang. Beyond brightening low-light images. International Journal of Computer Vision, 129:1013-1037, 2021.
[60] Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo. Kindling the darkness: A practical low-light image enhancer. In Proceedings of the 27th ACM International Conference on Multimedia, pages 1632-1640, 2019.
[61] Zhao Zhang, Huan Zheng, Richang Hong, Mingliang Xu, Shuicheng Yan, and Meng Wang. Deep color consistent network for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1899-1908, 2022.
[62] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2881-2890, 2017.
[63] Lin Zhao, Shao-Ping Lu, Tao Chen, Zhenglu Yang, and Ariel Shamir. Deep symmetric network for underexposed image enhancement with recurrent attentional learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12075-12084, 2021.
[64] Chuanjun Zheng, Daming Shi, and Wentian Shi. Adaptive unfolding total variation network for low-light image enhancement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4439-4448, 2021.
[65] Dewei Zhou, Zongxin Yang, and Yi Yang. Pyramid diffusion models for low-light image enhancement. arXiv preprint arXiv:2305.10028, 2023.
[66] Zhiyu Zhu, Junhui Hou, and Xianqiang Lyu. Learning graph-embedded key-event back-tracing for object tracking in event clouds. Advances in Neural Information Processing Systems, 35:7462-7476, 2022.
[67] Zhiyu Zhu, Junhui Hou, and Dapeng Oliver Wu. Cross-modal orthogonal high-rank augmentation for rgb-event transformer-trackers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22045-22055, 2023.