Embedding Fourier for Ultra-High-Definition Low-Light Image Enhancement
Authors: Chongyi Li
Chun-Le Guo
man zhou
Zhexin Liang
Shangchen Zhou
Ruicheng Feng
Chen Change Loy
Venue: ICLR 2023 notable top 5% (2023)
Keywords: low-light image enhancement
high-resolution image processing
Fourier transform
benchmark

Abstract: Ultra-High-Definition (UHD) photo has gradually become the standard configuration in advanced imaging devices. The new standard unveils many issues in existing approaches for low-light image enhancement (LLIE), especially in dealing with the intricate issue of joint luminance enhancement and noise removal while remaining efficient. Unlike existing methods that address the problem in the spatial domain, we propose a new solution, UHDFour, that embeds Fourier transform into a cascaded network. Our approach is motivated by a few unique characteristics in the Fourier domain:  1) most luminance information concentrates on amplitudes while noise is closely related to phases, and 2) a high-resolution image and its low-resolution version share similar amplitude patterns. Through embedding Fourier into our network, the amplitude and phase of a low-light image are separately processed to avoid amplifying noise when enhancing luminance. Besides, UHDFour is scalable to UHD images by implementing amplitude and phase enhancement under the low-resolution regime and then adjusting the high-resolution scale with few computations. We also contribute the first real UHD LLIE dataset, UHD-LL, that contains 2,150 low-noise/normal-clear 4K image pairs with diverse darkness and noise levels captured in different scenarios. With this dataset, we systematically analyze the performance of existing LLIE methods for processing UHD images and demonstrate the advantage of our solution. We believe our new framework, coupled with the dataset, would push the frontier of LLIE towards UHD. The code and dataset are available at https://li-chongyi.github.io/UHDFour/.

Published as a conference paper at ICLR 2023
EMBEDDING FOURIER FOR ULTRA-HIGH-DEFNNITION LOW-LIGHT IMAGE ENHANCEMENT
Chongyi Li', Chun-Le Guoi, Man Zhou', Zhexin Liang', Shangchen Zhou', Ruicheng Feng', and Chen Change Loy' LS-Lab, Nanyang Technological University Nankai University
ABSTRACT
Sltra-High-Definttii (UHD) photo has gradually become the standard configu- ration in advanced imaging devices. The new standard unveils many issues in existing approaches for low-light image enhancement (LLIE), especially in deal- ing with the intricate issue of joint luminance enhancement and noise removal while remaining efficient. Unlike existing methods that address the problem in the spatial domain, we propose new solution, UHDFour, that embeds Fourier transform into cascaded network. Our approach is motivated by few unique characteristics in the Fourier domain: 1) most luminance information concentrates amplitudes while noise is closely related to phases, and 2) high-resolution image and its low-resolution version share similar amplitude patterns. Through embedding Fourier into our network, the amplitude and phase of a low-light im- age are separately processed to avoid amplifying noise when enhancing lumi- nance. Besides, UHDFour is scalable tude and phase enhancement under the fow-resolution regime and then adjusting the high-resolution scale with few computations. We also contribute the first real UHD LLIE dataset, UHD-LL, that contains 2,150 now-noise/normal-ccees 4K im- age pairs with diverse darkness and noise levels captured different scenarios. With this dataset, we systematically analyze the performance of existing LLIE methods for processing UHD images and demonstrate the advantage of our solu- tion. We believe our new framework, coupled with the dataset, would push the frontier of LLIE towards UHD. The code and dataset are available at https://li- chongyi.ginn wwtpii.tttoooooooo /HHurr/ur//
UHD images by implementing ampli-
INTRODUCTION
With the advent advanced imaging sensors and displays, Satra-High-Defnniii (UHD) imaging has witnessed rapid development in recent years. While UHD imaging offers broad applications and makes significant difference in picture quality, the extra pixels also challenge the efficiency of existing image processing algorithms.
In this study, we focus on one the most challenging tasks in image restoration, namely low-light image enhancement (LLIE), where one needs to jointly enhance the luminance and remove inherent noises caused by sensors and dim environments. Further to these challenges, we lift the difficulty by demanding efficient processing in the UHD regime.
Despite the remarkable progress in low-light image enhancement (LLIE) (Li et al., 202 la), existing methods (Zhao al. 2021] Wu al., 2022; Xu al., (022). as shown in Figure I show appar- ent drawbacks when they are used to process world UHD low-light images. This is because (1) most methods (Guo al. 2020 Liu et al. 202Tb Ma et al. 2022) only focus on luminance enhancement and fail in removing noise; (2) some approaches (Wu et al., 2022 Xu et al.) 2022)
simultaneously enhance luminance and remove noise the spatial domain. resulting in the subop-
Wu et al. 2022 Xu et al. 2022) are mainly trained on low-resolution (LR) data, leading to the incompatibility with high-resolution (HR) inputs; and (4) some studies (Xu et al. 2022 Zamir al. 2022) adopt heavy structures, thus being inefficient for processing UHD images. More discussion on related work is provided in the Appendix.
timal enhancement; and (3) existing methods (Wei et al. 2018 Zhao et al. 202 1

---

Published as a conference paper at ICLR 2023
UHD low-light image
Zhao et (ICCV" URetinex-Net (CVPR 22) SNR-Aware (CVPR 22)
Figure I: Visual results of state of the arts (Zhao et al. (Zhao et al. 2021 URetinex-Net (Wu et al. 2022 and SNR-Aware (Xu et al. (2022) pre-trained
existing low-lige light image dataset for processing the real- world UHD low-light images. We amplify the brightness of the input UHD low-light images 10 times (top right comnrr of the first column) to show details and noise. These officially released models were trained using existing paired LR images with mild noise (i.e., the LOL dataset (Wei et al. 2018)). Existing models cannot cope with challenging UHD low-light images well.
To overcome the challenges aforementioned, we present a new idea for performing LLIE in the Fourier Domain. Our approach differs significantly from existing solutions that process images in the spatial domain. In particular, our method, named as UHDFour, is motivated by our observation of two interesting phenomena in the Fourier domain of low-light noisy images: i) luminance and noise can be decomposed to a certain extent in the Fourier domain. Specifically, luminance would manifest amplitude while noise is closely related to phase, and îi) the amplitude patterns of images of different resolutions are similar. These observations inspire the design of our network, which handles luminance and noise separately in the Fourier domain. This design is advantageous as it avoids amplifying noise when enhancing luminance, a common issue encountered in existing spatial domain-based methods. In addition, the fact that amplitude patterns of images of different resolutions are similar motivates us to save computation by first processing in the low-resolution regime and performing essential adjustments only in the high-resolution scale.
We also contribute the first benchmark for UHD LLIE. The dataset, named UHD-LL, contains 2,150 low-noisennormallceea 4K UHD image pairs with diverse darkness and noise levels captured in different scenarios. Unlike existing datasets (Wei et al., 2018 Lv et al. 2022 Bychkovsky et al.) 201T) that either synthesize or retouch low-light images to obtain the paired input and target sets, we capture real image pairs. During data acquisition, special care is implemented to minimize geometric and photometric misalignment due to camera shake and dynamic environment. With the new UHD-LL dataset, we design series of quantitative and quantitative benchmarks to analyze the performance of existing LLIE methods and demonstrate the effectiveness of our method.
Our contributions are summarized as follows: (1) We propose new solution for UHD LLIE that is inspired by unique characteristics observed in the Fourier domain. In comparison to existing LLIE methods, the proposed framework shows exceptional effectiveness and efficiency in addressing the joint task of luminance enhancement and noise removal in the UHD regime.
(2) We contribute
the first UHD LLIE dataset, which contains 2,150 pairs of 4K UHD now-noisenoormal-cee data, covering diverse noise and darkness levels and scenes. (3) We conduct a systematical analysis of existing LLIE methods on UHD data.
OUR APPROACH
In this section, we first discuss our observations in analyzing low-light images in the Fourier domain, and then present the proposed solution.
2.1 OBSERVATIONS IN FOURIER DOMAIN
Here we provide more details to supplement the observations we highlighted in Sec. 1 We analyze real UHD low-light images in the Fourier domain and provide a concise illustration in Figure 2 Specifically, (a) Swapping the amplitude of low-light and noisy (low-noise) image with that of its corresponding normal-light and clear (normal-clear) image produces a normal-light and noisy moormal-noise image and low-light and clear (low-clear) image. We show more examples in the Appendix. The result suggests that the luminance and noise can be decomposed to a certain extent in the Fourier domain. In particular, most luminance information is expressed amplitudes, and
2

---

Published as a conference paper at ICLR 2023
Amplitack
Plase
composttonna
4 cestessmmpll
normal-clcar
low-clear
bormal-cker
normal-clear
normal-dlear
low-noise
real
Aspliindd
Pluse
Amplinace
Amplinade
Amplitude
Amplinude
(a)
(b)
Figure 2: Motivations, We observed that (a) Juminance and noise can be "decomposed" to a certain extent in the Fourier domain and (b) HR image and its LR versions share similar amplitude patterns. The amplitude and phase are produced by Fast Fourier Transform (FFT) the compositional images are obtained by Inverse FFT (IFFT). For visualization, we show amplitude and phase in imagery format with common transformations. Lines of the same color indicate a set of FFT/IFFT transforms. The red triangles mark the similar pattern (obviously different from the gray one). Zoom in for the details and noise. We show more examples and analysis in the Appendix.
noises are revealed in phases. This inspires us to process luminance and noise separately in the Fourier domain. (b) The amplitude patterns of an HR normal-clear image and its LR versions similar and are different from the corresponding HR low-noise counterpart. Such characteristic offers the possibility to first enhance the amplitude of LR scale with more computations and then only make minor adjustments in the HR scale. this way, most computations can be conducted in the LR space, reducing the computational complexity.
2.2 THE UHDFOUR NETWORK
PixciInnhuulle (8x1)
A,
Plesishuflee (Pa2)
861
Tushyp
Fodlyp
P,
y
LRNet
HRNet
Figure 3: Overview of UHDFour. Our approach consists of an LRNet and an HRDNet. LRNet an encoder-decoder network that produces 8x downsampled result iis and the refined amplitude A, and phase P, features. We omit the skip connections for brevity. The HRNet contains Adjustment Block and the upsampling operation, producing the final result ÿ. Most computation is conducted in the LRNet.
Overview. UHDFour aims to map an UHD low-noise input image x E #x its correspond- ing normal-cleat version IR #xxXXC where H, W, and C' represent height, width, and channel, respectively. Figure shows the overview of UHDFour. consists an LRNet and an HRNet.
Motivated by the observation in Sec. 2.1 LRNet takes the most computation of the whole network. Its input is first embedded into the feature domain by a Conv layer. To reduce computational com- plexity, downsample the features to 1/8 of the original resolution by bilinear interpolation. Then, the LR features go through an encoder-decoder network, which contains four FouSpa Blocks with two 2 downsample and two upsample operations, obtaining outputs features. The outputs fea- tures are respectively fed to FFT to obtain the refined amplitude A, and phase P, features and a Conv layer estimate the L.R normal-clear image vss E H//SX//8XCC
The outputs of LRNet coupled with the input are fed to the HRNet. Specifically, the input x is first reshaped to pm HW xCx64 via PixelUnshuffle (8x L) to preserve original information. and then fed to an Adjustment Block. With the refined amplitude A, and phase P, features, the Adjustment Block produces adjusted features that are reshaped to the original height and width of input x via Pixelshuffle (8x 1). Finally, we resize the estimated LR normal-cled image ys to the original size of input x via bilinear interpolation and combine with the upsampled features to estimate the final HR normal-clear image y. We detail the key components follows.
3

---

Published as a conference paper at ICLR 2023
FonSpa Block
Fourler branch
Adjustment Block
(b)
X
Spatial
Spadial branch
Figure Structures of the FouSpa Block (a) and Adjustment Block (b).
FouSpa Block. In Sec. 2.1 we observe that luminance and noise can be decomposed in the Fourier domain. Hence, we design the FouSpa Block to parallelly implement amplitude and phase enhance- ment in the Fourier domain and feature enhancement in the spatial domain. As shown in Figure [a), the input features are forked into the Fourier and Spatial branches. In the Fourier branch, FFT first used to obtain the amplitude component (A) and phase component (P). The two components are separately fed to two Cony layers with 1 xx 1 kernel. Note that when processing amplitude and phase, we only use 1x1 kernel avoid damaging the structure information. Then, we transform them back to the spatial domain via IFFT and concatenate them with the spatial features enhanced by Half Instance Normalization (HIN) unit (Chen et al.. 2021ap. We adopt the HIN unit based on its efficiency. The concatenated features are further fed to a Conv layer and then combined with the input features in a residual manner. Although our main motivations are the Fourier domain, the use of the spatial branch is necessary. This is because the spatial branch and Fourier branch are complementary. The spatial branch adopts convolution operations that can model the structure dependency well in spatial domain. The Fourier branch can attend global information and benefit the disentanglement energy and degradation.
Adjustment Block. The Adjustment Block is the main structure of the HRNet, and it is lightweight. As shown in Figure 4b), the Adjustment Block shares a similar structure with the FouSpa Block. Differently, in the Fourier branch, with the refined amplitude A, features obtained from the LRNet, we use Spatial Feature Transform (SFT) (Wang et al., 2018) to modulate the amplitude features of the input Tpu via simple affine transformation. Such transformation adjustment is possible be- cause the luminance, as global information, manifests as amplitude components, and the amplitude patterns of an HR scale and its LR scales are similar (as discussed in Sec. (.I).. Note that we cannot modulate the phase because of its periodicity. Besides, we do not find an explicit relationship be- tween the HR scale's 's phase and its LR scales. However, we empirically find that concatenating the refined phase P. features achieved from the LRNet with the phase features of the input Tpu improves the final performance. We thus apply such concatenation our solution.
Losses. We use 1 to supervise is and ÿ. We also add perceptual loss to supervise yi while the use of perceptual loss on ý is impracticable because its high resolution. Instead, we add SSIM loss Lesim on The final loss L is the combination of these losses:
L | vl1 + 0.0004 Lesim (Ú. y) + 0.1 x Igs y/s |1 + 0.0002
(1)
where y is the ground truth, Y/s is the 8x downsampled version of y, VGG is the pre-trained VGG19 network, in which we use four scales to supervise training Zhou et al. (2022).
3 UHD-LL DATASET
We collect a real low-noise/normal-ceen paired image dataset that contains 2,150 pairs of 4K UHD data saved in 8bit sRGB format. Several samples are shown in Figure 5
Images are collected from a camera mounted on a tripod to ensure stability. Two cameras, i.e., a Sony a7 III camera and a Sony Alpha a6300 camera, used to offer diversity. The ground truth (or normal-cles -lear)) image is captured with a small ISO [100, 800] in a bright scene (indoor or outdoor). The corresponding low-noise image is acquired by increasing the ISO [1000, 20000] and reducing the exposure time. Due to the constraints of exposure gears in the cameras, shooting in the large ISO range may produce bright images, which opposes the purpose of capturing low-light and noisy
4

---

Published as a conference paper at ICLR 2023
Table 1: Comparison between classic LLIE datasets and our UHD-LLI dataset. 'Numher': the number of paired images. 'Resolution': the average resolution of the dataset. 'Noise': low-light images contain noise. 'Real': both low-light images and GT acquired in real scenes.
Dataset
Number
Resolution
Noise
Real
4240x2832
SID (RAW)
5,094
6000x4000
MIT-Adobe FiveK
5,000
4000x2500
Exposure-Errow Errors
24,000
1000x900
LOL
500/789
600x400
Figure 5: Samples from the proposed UHD-LL datasct.
UHD-LL (Ours)
2,150
3840x2166
images. Thus, some cases, we put neutral-density (ND) filter with different ratios on the camera lens to capture low-noise images. In this way, we can increase the ISO to generate heavier noises and simultaneously obtain extremely dark images, enriching the diversity of darkness and noise levels.
The main challenge of collecting paired data is to reduce misalignment caused by camera shakes and dynamic objects. We take several measures ameliorate the issue. Apart from using tripod, we also use remote control software (Imaging Edge) to adjust the exposure time and ISO value to avoid any physical contact with the camera. To further reduce subtle misalignments, we adopt an image alignment algorithm (Evangelidis & Psarakis 2008) to estimate the affine matrix and align the low-light image and its ground truth. We improve the alignment method by applying AdaIN (Huang) & Belonggee 2017 before the affine matrix estimation to reduce the intensity gap. Finally, hire annotators to check all paired images carefully and discard those that still exhibit misalignments.
We split the UHD-LL dataset into two parts: 2,000 pairs for training and 115 pairs for testing. The training and test partitions are exclusive in their scene and data. We also ensure consistency in pixel intensity distribution between the training and test splits. More analysis of this data, e.g., the pixel intensity and Signal-to-Noise Ratio (SNR) distributions, can be found in the Appendix.
A comparison between our UHD-LL dataset and existing paired low-light image datasets is pre- sented in Table | The LOL dataset (two versions: LOL-v1: 500 images; LOL-v2: 789 images) is most related to our UHD-LL dataset as both focus on real low-light images with noise. The LOL-v2 contains all images of the LOL-v1. In contrast to the LOL dataset, our dataset features a more ex- tensive collection, where diverse darkness and noise levels from rich types of scenes are considered. Moreover, the images of our dataset have higher resolutions than those from the LOL dataset. As shown in Figure [ the models pre-trained on the LOL dataset cannot handle the cases in our UHD- LL dataset due to its insufficient training data, which are low-resolution and contains mostly mild noises. Diffcrent from SID dataset that focuses on RAW data, our dataset only studies the data with RGB format. The images in the SID dataset captured in extremely dark scenes. Its diversity of darkness levels and scenes is limited. When these RAW data with extremely low intensity are trans- formed into sRGB images, some information would be truncated due to the bit depth constraints of 8bit sRGB image. In this case, it is challenging to train network for effectively mapping noise and low-light images to clear and normal-light images using these sRGB images training data.
EXPERIMENTS
Implementation. We implement our method with PyTorch and train it on six NVIDIA Tesla V100 GPUs. We use an ADAM optimizer for network optimization. The learning rate is to 0.0001, A batch size of 6 is applied. We fix the channels of each Conv layer to 16, except for the Conv layers associated with outputs. We use the Conv layer with stride 2 and 4 4 kernels implement the 2x downsample operation in the encoder and interpolation to implement the 2x upsample operation in the decoder in the LRNNet. Unless otherwise stated, the Conv layer uses stride = and 3x3 kernels. We use the training data in the UHD-LL dataset to train model. Images are randomly cropped into patches of size 512 x 512 for training.
Compared Methods. We include 14 state-of-the-art methods (21 models in total) for our bench- marking study and performance comparison. These methods includes 12 light enhancement meth- ods: NPE (TIP' 13) (Wang al., 2013) SRIE (CVPR' 16) (Fu et al.] 2016). DRBN (CVPR 20) (Yang et al. 2020a). Zero- DCE (0) (Guo et al. 2020), Zero-De DCE+ (TPAMI21) (Li et al.
202 RUAS (CVPR'21) (Liu et al. 2021b), Zhao el al. 21) (Zhao al. 2021 Enlighten-
5

---

Published as a conference paper at ICLR 2023
Table 2: Benchmarking study on the testing set of our UHD-LLL. All models are released from the original
papers and trained on the corresponding datasets. The best and second results are in red and blue, respectively.
Methods
PSNRI
SSIM
LPIPS.
MUSIOT
NTQU,
NIMA
Training
Ingur
9.926
0.482
0.551
26.779
5,379
2.269
NPE Wang at
18.293
0.587
0.547
35,200
4.916
2.368
SRTE (Fu nl 2010
16.316
0.652
0.503
31,345
4.927
2110
DRBN Yang al 2020n
13,455
0.689
0.450
34.925
4.408
2.154
LOL-V2
Zero- IXC (Guo ni 2020
17.081
0.064
0.509
35.488
5,006
2.1.9
SICE
Zero DCE++ Li
17.648
0.672
0.506
32,520
4.887
2.213
SICE
RUASSTOL. ( mB ot 202Tb
11,761
0.701
0.514
28. 196
5.907
2.565
LVL-v2
RUAS-MITSK
212T1
14,250
0.586
5.407
2.270
AIT-Adobe FiveK
RUAS-DarkFace
(Threr al: 202th
11.325
0.583
0.596
256
6.100
2:561
DarkFace
Zhao al-MITSK
(Zho
2021
15.177
0.547
0.530
32.127
4,495
2.208
MIl-Adobe FiveK
7han al-1.01 (2in of or 2014
18:004
0. 0.694
0.479
32.392
4,248
2.183
1.01.vl
EnlighienGAN Jann
17,637
0.767
0.459
27:441
5,497
1.977
Assembled
Ahñi et al. (Anh et dll 2021
18.212
0.610
0.479
33.970
4,793
2.217
Exposare- Frross
SCI-casy Ma d.
15,536
0.610
0.501
31,848
4.897
2.166
MIT--dobe FiveK
SCI-mediunT Midee
15.481
0.622
0.528
31.474
4.941
2:211
SCI-difficuit Mi
17.872
0.578
0.544
36.219
5.218
2.106
DarkFace
SNR- LOrz Lraatr
Xu al
2022
15.737
0.802
0:448
20.385
9.591
2.275
LOL.vl
SNR-Aware-LOvv
20.4
15,536
0.695
0.468
33.098
3.961
2.387
LOL-v1
SNR-Aware-L.OO 200
Air
20722
15,954
0.742
0.471
23,494
9.257
2.001
LOLxV2
L.R-Awaane-OOOxx2rruul 1OI siich
Xil
212-
14.616
0.634
0.488
33.447
4.143
2.577
LOLLx2
SNR-Aware 20
Xu
16.031
0.748
20,065
9.963
2.448
LOL-syn
SNR-Aware- Ol L0 xxyymllimmmmmee....o
Xu 21 2022
15,887
0.675
0.497
31,473
4,460
2.484
LOI-syn
URetinex-Net Wa
20.689
0.706
0.457
35,434
4.974
2.181
LOL-V
GAN (TIP'21) Jiang al. 2021 Afifi al. (CVPR'21) (Afiii al.
2021), SCI (CVPR'2) 22) (Ma
ct al. 2022), ANR-Aware (CVPR 22) (Xu et al. 2022). URetinex-Net (CVP 22) (Wu et al. 2022)
and 2 Transformers: Uformer (CVPR"2) (Wang al. 2022) and Restormer (CVPR 22) Zamir
jet al. 2022). We use their released models and also retrain them using the same training data as our method. Note that some methods provide different models trained using different datasets. Due to the heavy models used in Restormer (Zamir et al. 2022) and SNR-Aware (Xu et al.) 2022). we cannot infer the full-resolution results of both methods on UHD images, despite using a GPU with 48G memory. Following previous UHD study (Zheng et al. (021I, we resort to two strategies for this situation: (1) We downsample the input to the largest size that the model can handle and then resize the result to the original resolution, denoted by the subscript 'resize'. (2) We split the input into four patches without overlapping and then stitch the result, denoted by the subscript 'stitch'.
Evaluation Metrics. We employ full-reference image quality assessment metrics PSNR, SSIM (Wang et al. 2004), and LPIPS (Alex version) (Zhang et al., 2018) to quantify the perfor- mance of different methods. We also adopt the non-reference image quality evaluator (NIQE) (Mit- tal et al., 2013) and the multi-scale image quality Transformer (MUSIQ) (trained on KonIQ-10k dataset) (ee al. (021) for assessing the restoration quality. We notice that the quantitative sults reported by different papers diverge. For a fair comparison, we adopt the commonly-used IQA PyTorch Toolbox to compute the quantitative results of all compared methods. We also test the trainable parameters and running time for processing UHD 4K data.
4.1 BENCHMARKING EXISTING MODELS
To validate the performance of existing LLIE methods that were trained using their original training data, we directly use the released models for evaluation on the UHD low-light images. These origi- nal training datasets include LOL (Wei et al.) 2018). MIT-Adobe-Fived (Bychkovsky et al. 201 Exposure-Errors (Afif et al.) 2021). SICE (Cai et al. 2018). LSRW (Hai et al. 2024). and Dark- Face (Yang et al. 2020b). EnlightenGAN uses the assemble training data from existing datasets (Wei et al. 2018 8 Dang-Nguyen et al. 2015) Kalantari & Ramamoorthi 2017b; Cai et al.) 2018). In addition, the LOL-vi and 1.OL-22 contain real low-light images while LOI--yn is a synthetic dataset. Due to the limited space, we only show relatively good results. As shown in Figure 6 all methods can improve the luminance of the input image. However, they fail to produce visually pleasing results. DRBN and EnlightenGAN introduce artifacts. RUAS-LOL and RUAS-DarkFace yield over-exposed results. Color deviation is observed in the results of EnlightenGAN and Afifi et al. All methods cannot handle the noise well and even amplify noise.
We also summarize the quantitative performance of different methods and verify the effectiveness of commonly used non non-reference metries for UHD low-light images in Table URetinex-Ner achieves the highest PSNR score while SNR-AWarr-LLOLvl is the best performer in terms of SSIM and

---

Published as a conference paper at ICLR 2023
Corr
Corn
Corr
Corn
input
DRBN
Zero-DCE
RUAS-LOL
Corn
Corn
Corr
Corn
RUAS-DarkFace
Zhao et al,-LOL
EnlightenGAN
Affii al.
Corn
Corn
Corn
Corn
URetinex-Net
GT
Figure 6: Visual comparison between state of the arts for restoring a UHD low-light image. The compared methods include DRBN (Yang et al. 2020aa.. Z.ro-DCE (Guo et al. (020), RUAS-LOL (Liu et al. 2021b
(Liu et al. (2021b), Zhao et al.-LOL (Zhao al. (021). EnlightenGAN (Jiang al. 2021
Afifi al. (Afifi al. 202 SCI-diffcult(Ma
al. 2022), ANR-Aware-LOL LOLV (Xu al. 2022). and
URetinex-Net(Wa al. 2022). use the released model directly in evaluation. released models
cannot handle the OHD low-light image well. More results can be found in the Appendix.
LPIPS. For non-reference metrics, SCI-difficult, Zhao et al.-LOL, and RUAS-LOL are the win- ners under MUSIQ, NIQE, and NIMA, respectively. From Figure 6 and Table 2 found the non-reference metrics designed for generic image quality assessment cannot accurately assess the subjective quality of the enhanced UHD low-light images. For example, RUAS-LOL suffers from obvious over-exposure in the result while is the best performer under the NIMA metric.
In summary, the performance of existing released models is unsatisfactory when they used to enhance the UHD low-light images. The darkness, noise, and artifacts still exist in the results. Compared with luminance enhancement, noise is the more significant challenge for these methods. No method can handle the noise issue well. The joint task of luminance enhancement and noise removal raises new challenge for LLIE, especially under limited computational resources. We also observe a gap between visual results and the scores of non non-reference metrics for UHD LLIE. The gap calls for more specialized non-reference metries for UHD LLIE.
4.2 COMPARING RETRAINED MODELS
Besides the released models, we also retrain existing methods on our UHD-LL training data and compare their performance with our method. Due to the limited space, we only compare our method with several good performers. More results can be found in the Appendix. As shown in Figure our UHDFour produces clear and normal-light result close to the ground truth. In comparison, Zero- DCE++, RUAS, Afifi al., SCI, and Restormer experience color deviations. Zero-DCE, Zero- DCE++, RUAS, Zhao et al., Afifi et al., and SCI cannot remove the noise due to the limitations of their network designs. These methods mainly focus luminance enhancement. SNR-Aware, Uformer, and Restormer have strong modeling capability because of the use of Transformer struc- tures. However, the three methods still leave noise the results and introduce artifacts.
The quantitative comparison is presented in Table 3 Our UHDFour achieves state-of-the-ar perfor- mance in terms of PSNR, SSIM, and IPIPS scores and outperforms the compared methods with

---

Published as a conference paper at ICLR 2023
input
ZereDCE
Zero-DCE++
Zhao et al.
Afffi
SCI
SRN-Aware
Uformer
Restorner
UHDFour (Ours)
GT
Figure 7: Visual comparison between the retrained state of the arts the UHD-LL dataset. The compared al. (021b), RUAS (Liu et al. 2021b), Zhao al. (Zhao et al., (02T), Affi et al. (Atifi et al. 2021). SCT (Ma et al. (2022), SNR-Aware (Xu al.) 2022), Uformer (Wang et al. 2022), and Restormer (Zamir et al. 2022). All compared models leave noise, artifacts, color deviations in the results. Our method achieves visually pleasing result.
methods include Zero-DCE (Guo et al. 2020), Zero Zero-DCE++ (Li DCE++
Quantitative comparison the retrained state of arts on the UHD-LL dataset. The result in red whereas the second one is in blue. RT: Running time. The training code of URetinex-Ne -Ne is not released.
Table 3:
Methods
PSNR1
SSIMT
LPIPS)
Parameter.]
RTL
DCE
2020
17.075
0.663
0.513
79.416K
0.353s
Zero-DCE++ Li APT
16.410
0.630
0.530
10.561
0.327s
RUAS (Liu e al 20210
13.562
0.749
0.460
3,438K
0.379s
Zhao et
2021
21.964
0.870
0.324
11.560M
6,900s
Affii et al. et 2020
20.805
0.740
0.440
70.154M
1631s
Ma
0.625
0.533
0.308s
SoR-Awarryosiie XB
2022
22.717
0.877
0.304
40.084M
0.026s
SNR SNR-Aware, Aek-Aware,aaca cutol.
2
20773
22.177
0.866
0.307
40.084M
0.035s
Uformer Wang
2022
19.283
0.849
0.356
20.628M
0.235x
Restormer Zamir et Il 2022
22.597
0.878
0.280
26. 112M
0.368s
Restormer aiitok
Zainir
26.112M
0.368s
UHDFour (Oirs)
26. 26.226
0.900
0.233
17.537M
0.024s
large margin. The Transformer-bass based SNR-Aware and Restormer rank the second best. Our method has the fastest processing speed for UHD images as most computation is conducted in the LR space.
To further verify the effectiveness of network, we compare ap- proach with several methods, includ- ing Retinex-Ner /Wei al. (2018). Zero-DCE (Guo et al.222000). AGLL- Net (Lv et al. 2021). Zhao et
Table 4:
Quantitative comparison on the LOL-v1 and LOL-v2 datasets. The best result is in red whereas the second one is in blue. indicates the pre-trained model is not available.
LOL-vi
LOL-V2
Methods
PSNK
SSIMF
PSNKT
SSIMI
input
7.77
0.19
9.72
0.21
al. (Zhao et al. 2021), RUAS (Liu jet al.) (0221)) SCI (Ma et al. 2022). and URetinex-Net (Wu et al. 2022
Retinex Net
(Wei al.
2018
16.77
15.43
0.64
Zero-DCE Guo
16.79
0.67
12.84
0.54
AGLLNet LV ét al
2021
17.52
0.77
20.69
0.78
Zhuo et al. Zhao et al
2021
21.67
0.87
18.84
0.84
that were pre-trained or fine-tuned on the LOL-v1 and LOL-v2 datasets (Wei et al. 20188. Due to the mild noise and low-resolution images in
RUAS (Liu et 202lb
16.44
15.48
SCI MT
14.78
0.62
16.74
0.62
URetinex-Net Wu 2022
19.84
0.87
UHDFour (Ours)
23.09
0.87
21.78
0.87
8

---

Published as a conference paper at ICLR 2023
the LOL-v1 and LOL-v2 datasets, we change the 8x downsample and upsample operations to 2 and retrain our network. And such characteristics of LOL-v1 and LOL-v2 datasets prohibit us from showing the full potential of our method in removing noise and processing high-resolution images. Even though our goal is not to pursue state-of-the-art performance on the LOL-v1 and LOL--v datasets, method achieves satisfactory performance presented in Table 4 The visual results are provided in the Appendix.
4.3 ABLATION STUDY
We present ablation studies to demonstrate the effectiveness of the main components our design. For the FouSpa Block, we remove the Fourier branch (FB) (#1), remove the Spatial branch (SB) (#2), and replace the FouSpa Block (ie.., without FB and SB) with the Residual Block of comparable parameters (#3). We also replace the FB with the SB (i.e.,, us-
Table Quantitative comparison of ablated models. FB: Fourier Branch; SB: Spatial Branch; AM: Amplitude Modulation; PG: Phase Guidance; and Concat: Concatenation. x: multiple repeated modules.
FouSpa Block
Adjustment Block
Output Concal
Performance
FB
SB
AM
PG
SB
PSNR/SSIM
24. 8777
24.722/0.88
24.005/0.853
24.310/0.876
28.52970.88
ing two SB) (#4). For the Adjustment Block, we remove the Amplitude Modulation (AM) (#5), remove the Phase Guidance (PG) (#6), remove the SB (#7), and remove both AM and PG (#8). We also replace the AM and PG with two SB (#9), and replace the Adjustment Block with the Residual Block of comparable parameters (#10). For the final output, remove the concatenation of the LR normal-clear result (1is), indicated as #11. We also replace all FB with SB, indicated as #12. Unless otherwise stated, all training settings remain unchanged the implementation of full model, denoted as #13.
824/0.874
14.513/0.872
22.421/0.85
3x
34.36600.66
106/0.60..6
5
4
25.616018887
12
2x
3x
23. 85
26.226/0.00
The quantitative comparison of the ablated models on the UHD-LL testing set is presented in Table 5 We also show the visual comparison of some ablated models in the Appendix. As shown, all the key designs contribute to the best performance of the full model. Without the Fourier branch (#1), the quantitative scores significantly drop. The result suggests that processing amplitude and phase separately improves the performance of luminance enhancement and noise removal. From the results of #2, the Spatial branch also boosts the performance. However, replacing the FouSpa Block with the Residual Block (#3) cannot achieve comparable performance with the full model (#13), indicating the effectiveness of the FouSpa Block. For the Adjustment Block, the Amplitude Modulation (#5), Phase Guidance (#6), and Spatial branch (#7) jointly verify its effectiveness. Such a block cannot be replaced by a Residual Block (#10). From the results of #11, we can see that is necessary to estimate the LR result. In addition, replacing the Fourier branch with spatial branch (#4,#9.Ht ## 2)) cannot achieve comparable performance with the full model (# 13), showing the efficacy of Fourier branch.
CONCLUSION
The success of our method is inspired by the characteristics of real low-light and noisy images in the Fourier domain. Thanks to the unique design of our network that handles luminance and noises in the Fourier domain, it outperforms state-of-the-as methods in UHD LLIE with appealing efficiency. With the contribution of the first real UHD LLIE dataset, it becomes possible to compare existing methods with real UHD low-light images. Our experiments are limited to image enhancement; we have not provided data and benchmarks in the video domain. Our exploration has not considered adversarial losses due to memory constraints. Moreover, as our data is saved in sRGB format, the models trained on our data may fail in processing the extreme cases, in which the information is lost due to the limited bit depth. HDR data may be suitable for these cases. Nevertheless, we believe our method and the dataset can bring new opportunities and challenges to the community. The usefulness of Fourier operations may go beyond our work and see potential in areas like image decomposition and disentanglement. With improved efficiency, it may be adopted for applications that demand real-time response, e.g.., enhancing the perception of autonomous vehicles in the dark.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
This study focuses on low-light image enhancement and does not involve any ethics issues. The dataset proposed in this paper also does not involve privacy issues.
ACKNOWLEDGEMENT
This study is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). It is also partially supported by the NTU NAP grant. Chun-Le Guo is also supported by MindSpore, CANN, and Ascend AI Processor.
REFERENCES
Mahmoud Afifi, Konstantinos G. Derpanis, Bjorn Ommer, and Michael S. Brown. Learning multi- scale photo exposure correction. In CVPR, 2021.
Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Fredo Durand. Learning phtographic global tonal adjustment with a dataset of input/output image pairs. CVPR, 2011.
Jianrui Cai, Shuhang Gu, and Lei Zhang. Learning a deep single image contrast enhancer from multi-exposure image. IEEE Transactions Image Processing, (144))2049-0033 2018.
Chen Chen, Qifeng Chen, Jia Xu, and Koltun Vladlen. Learning to see in the dark. In CVPR, 2018.
Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. Hinet: Half instance normal- ization network for image restoration. In CVPR, 2021a.
Xiangyu Chen, Yihao Liu, Zhengwen Zhang, Yu Qiao, and Chao Dong. HDRUNet: Single image HDR reconstruction with denoising and dequantization. In CVPRW, 2021b.
Duc-Tien Dang-Nguyen, Cecilia Pasquini, Valentina Conotter, and Giulia Boato. Raise: A raw images dataset for digital image forensics. ACMMSC, 2015.
Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafal K. Mantiuk, and Jonas Unger. HDR image reconstruction from single exposure using deep cnns. ACM Transactions on Graphics, 2017.
Yuki Endo, Yoshihiro Kanamori, and Jun Mitani. Deep reverse tone mapping. ACM Transactions Graphics, 2017.
Georgios Evangelidis and Emmanouil Psarakis. Parametric image alignment using enhanced corre- lation coefficient maximization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(10))1858- 1865, 2008
Xueyang Fu, Delu Zeng, Yue Huang, Xiaoping Zhang, and Xinghao Ding. A weighted variational model for simultaneous reflectance and illumination estimation. In CVPR, 2016.
Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong. Zero-reference deep curve estimation for low-light image enhancement. In CVPR, 2020.
Xin Guo, Xueyang Fu, Man Zhou, Zhen Huang, Jialun Peng, and Zhengjun Zha. Exploring fourier prior for single image rain removal. In ISCAI, 2022.
Jiang Hai, Zhu Xuan, Ren Yang, Yutong Hao, Fengzhu Zou, Fang Lin, and Songchen Han. R2met: Low-light image enhancement via real-low to real-normal network. arXiv preprint arXiv:2106. 14501, 2024.
Litao Hu, Huaijin Chen, and Jan P. Allebach. Joint multi-scale tone mapping and denoising for HDR image enhancement. In WACVW, 2022a.
Xiangyu Hu, Liquan Shen, Mingxing Jiang, Ran Ma, and Ping An. LA-HDR: Light adaptive HDR reconstruction framework for single LDR image considering varied light conditions. IEEE Trans- actions Multimedia, 2022b.
10

---

Published as a conference paper at ICLR 2023
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, 2017.
Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang. EnlightenGAN: Deep light enhancement without paired supervision. IEEE Transactions on Image Processing, 30:2340-222 2349, 2021.
ANima Khademi Kalantari and Ravi Ramamoorthi. Deep high dynamic range imaging of dynamic scenes. ACM Transactions on Graphics, 2017a.
Nima Khademi Kalantari and Ravi Ramamoorthi. Deep high dynamic range imaging of dynamic scenes. ACM Transactions on Graph, 36:144, 2017b.
Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. MUSIQ: Multi-scale image quality transformer. In ICCV, 2021.
Hieu Le and Dimitris Samaras. Shadow removal via shadow image decomposition. In ICCV, 2019.
Chongyi Li, Chunle Guo, Linhao Han, Jun Jiang, Ming-Ming Cheng, Jinwei Gu, and Chen Change Loy. Low-light image and video enhancement using deep learning: A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2021a. a.
Chongyi Li, Chunle Guo, and Chen Change Loy. Learning to enhance low-light image via zero- reference deep curve estimation. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2021b.
Jiaying Liu, Dejia Xu, Wenhan Yang, Minhao Fan, and Haofeng Haung. Benchmarking low-light image enhancement and beyond. International Journal of Computer Vision, 2021a.
Risheng Liu, Long Ma, Jiaao Zhang, Xin Fan, and Zhongxuan Luo. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement. In CVPR, 2021b.
Yu-Laun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsual Yang, Yung- Yu Chuang, and fia-Bin Huang. Single image HDR reconstruction by learning reverse the camera pipeline. In CVPR, 2020.
Feifan Lv, Yu Li, and Feng Lu. Attention guided low-light image enhancement with a large scale low-light simulation dataset. International Journal of Computer Vision, 2021.
Long Ma, Dian Jin, Risheng Liu, Xin Fan, and Zhongxuan Luo. Joint over and under exposures correction by aggregated retinex propagation for image enhancement. IEEE Signal Processing Letters, 17:1210-1214, 2020.
Long Ma. Tengyu Ma, Risheng Liu, Xin Fan, and Zhongxuan Luo. Toward fast, flexible, and robust low-light image enhancement. In CVPR, 2022.
Xintian Mao, Yiming Liu, Wei Shen, Qingli Li, and Yan Wang. Deep residual fourier transformation for single image deblurring, arXiv preprint arXiv:2111 11.i::2111.1774:: 2021.
Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters, 20(3):209-212, 2013.
Minh Tuan Pham, Van Quan Nguyen, Cao Duy Hoang, Hoang Long Vo, Dinh Khoi Phan, and An Hung Nguyen. Efficient complex valued neural network with fourier transform on image denoising. In ICFNDS, 2021.
Yasir Salih, Wazirah bt. Md-Esa, Aamir S. Malik, and Naufal Saad. Tone mapping of hdr images: A review. In ICIAS, 2012.
Abhilash Srikantha and Desire Sidibe. Ghost detection and removal for high dynamic range images: Recent advances. Signal Processing: Image Communication, 2012.
Lin Wang and Kuk-Jin Yoon. Deep learning for hdr imaging: State-of-the and future trends. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.

---

Published as a conference paper at ICLR 2023
Ruixing Wang. Qing Zhang, Chi- Wing Fu, Xiaoyong Shen, Wei-Shi Zheng, and Jiaya Jia. Under- exposed photo enhancement using deep illumination estimation. In CVPR, 2019.
Shuhang Wang, Jin Zheng, Hai-Miao Hu, and Bo Li. Naturalness preserved enhancement algorithm for non-uniform illumination images. IEEE Transactions on Image Processing, 22(9):3538-3488 2013.
Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. In CVPR, 2018.
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In CVPR, 2022.
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4): 600-612, 2004.
Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. Deep retinex decomposition for low-light enhancement. In BMVC, 2018.
Shangzhe Wu, Jiarui Xu, Yu-Wing Tai, and Chi-Keung Tang. Deep high dynamic range imaging with large foreground motions. In ECCV, 2018.
Wenhui Wu, Jian Weng, Pingping Zhang, Xu Wang, Wenhan Yang, and Jianmin jiang. Uretinex-net: Retinex-based deep unfolding network for low-light image enhancement. In CVPR, 2022.
Ke Xu, Xin Yang, Baocai Yin, and Rynson H. Lau. Learning to restore low-light images via In CVPR, 2020.
Xiaogang Xu, Ruixing Wang, Chi-Wing Fu, and jiaya Jia. SNR-aware low-light image enhance- ment. In CVPR, 2022.
Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement. In CVPR, 2020a.
Wenhan Yang, Ye Yuan, Wenqi Ren, and al. Advancing image understanding in poor visibility environments: A collective benchmark study. IEEE Transactions on Image Processing, 29:5737- 5752, 2020b.
Xin Yang, Ke Xu, Yibing Song, Qiang Zhang, Xiaopeng Wei, and Lau Rynson. Image correction via deep reciprocating HDR transformation. In CVPR, 2018.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2022.
Jinsong Zhang and lean-Francois Lalonde. Learning high dynamic range from outdoor panoramas. In ICCV, 2017
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In CVPR, 2018
Lin Zhao, Shaoping Lu, Tao Chen, Zhenglu Yang, and Ariel Shamir. Deep symmetric network for underexposed image enhancement with recurrent attentional learning. In ICCV, 2021.
Zhuoran Zheng, Wenqi Ren, Xiaochun Cao, Tao Wang, and Xiuyi Jia. Ulraa-high-definttoon image hdr reconstruction via collaborative bilateral learning. In ICCV, 2021.
Shangchen Zhou, Chognyi Li, and Chen Change Loy. Lednet: Joint low-light enhancement and deblurring in the dark. In arXiv preprint 22x1v:2202.03373, 2022.
12

---

Published as a conference paper at ICLR 2023
Appendix
In this Appendix, we present the related work and provide additional results and analysis.
A RELATED WORK
Low-Light Image Enhancement Methods. The focus of our work is on deep learning-baseer LLIE methods (Li et 202 Liu et al.) 202 [Wang et al. (2019) proposed a network to enhance the mapping. EnlightenGAN (Jiang et al.) 2021 proposed an attention-guided network to make the generated results indistinguishable from real normal-light images. By formulating light enhancement as a task of image-specific curve esti- mation that can be trained with non-reference losses, Zero-DCC (Guo et al. 2020) and Zero-DCE++ (Li et al. b02 obtain good brightness enhancement. iZhao et al. (2021) treated underexposed im- age enhancement as image feature transformation between the underexposed image and its paired enhanced version. Liu et al. 1b) proposed a Retinex model-inspired unrolling method, in which the network structure is obtained by neural architecture search. [ffifi et al. (2021) proposed a coarse- to-fine network for exposure correction. Ma et al. (2022) proposed self-calibrated illumination learning framework using unsupervised losses. Wu et al. (2022) combined the Retinex model with a deep unfolding network, which unfolds optimization problem into a learnable network. Xuu et al. (2022) proposed to exploit the Signal-to-Noise Ratio (SNR)-awar) -aware Transformer and convolutional models for LLIE. In this method, long-range attention is used for the low SNR regions while the short-range attention (convolutional layers) is for other regions.
underexposed photos by estimating an
Different from these works, our network takes the challenging joint Juminance enhancement, noise removal, and high resolution constraint of UHD low-light images into account in the Fourier domain, endowing new insights on UHD LLIE and achieving better performance.
Low-Light Image Enhancement Datasets. LOL (Wei et al. 2018) dataset contains pairs of low- lnorhal-light images saved in RGB format, in which the low-light images are collected by changing the exposure time and ISO. Due to the small size, it only covers small fraction of the noise and darkness levels. MIT-Adobe FiveK (Bychkovsky et 201 dataset includes paired low-/high- quality images, where the high-quality images are retouched by five experts. The low-quality images are treated as low-light images in some LLIE methods. However, this dataset originally collected for global tone adjustment, and thus, it ignores noise in its collection. Based on the MIT-Adobe FiveK dataset, a multi- exposure dataset. Exposure-Erree Errors, is rendered to emulate a wide range of exposure errors (Afifi et al. 2021). Similar to the MIT Adobe FiveK dataset, the Exposure-Error dataset also neglects the noise issue. SID (Chen et al. 2018) is a RAW data dataset.
The images of the SID dataset have two different sensor patterns (i.e., Bayer pattern and APS-C X-Trans pattern). Due to the specific data pattern, the deep models trained on this dataset are not versatile as they require the Raw data with the same pattern as input. Besides, a long-exposure reference image corresponds to multiple short-exposure images, leading to limited scene diversity.
Unlike existing LLIE datasets that either omit noise, capture limited numbers of images, require specific sensor patterns, or exclude UHD images, we propose a real UHD LLIE dataset that con- tains www.nooosoooomaalllaau image pairs with diverse darkness and noise levels captured in different scenarios. A comprehensive comparison is presented in Sec.
Image Decomposition based Enhancement. There are some image decomposition- en- hancement methods. For LLIE, Xu et al. (2000) proposed frequency-based decomposition-an and enhancement model, which suppresses noises in the low-frequency layer and enhances the details in the high-freyuency layer. Yang et al. (2020a) proposed a band representation-basse model. This method consists two stages: recursive band learning and band recomposition. WWil al. (2018) also dccomposed low-light image into an illumination component and a reflectance component according to the Retinex model and then separately enhance the components. Image decomposition was also used in shadow removal, in which two networks are used to predict shadow parameters and matter layer (Le & Samaras 2019).
semi-supervised
In addition, assuming that phase preserves high- semantics while the amplitude contains low- level features, (Guo et al. 2022) proposed a FPNet that consists of two stages for image de-raining. The first state is to restore the amplitude of rainy images. The second state then refines the phase of
13

---

Published as a conference paper at ICLR 2023
the restored rainy images. To solve the limitations of ResBlock that may overlook the low- frequency information and fails to model the long-distance information, (Mao et al. 2021) proposed a Residual Fast Fourier Transform with Convolution Block for image deblurring. The Block contains spatial residual stream and FFT stream. (Pham et al., 2021) proposed a complex valued neural network with Fourier transform for image denoising. The complex valued network first converts the noisy image to complex value via Fourier transform, then estimates complex filter which is applied to the converted noisy image for approaching the complex value of the ground truth image.
Although these methods decompose an image or use Fourier transform in the networks, our design has different motivations. Our motivations are inspired by the uniqueness of the Fourier domain for UHD low-light image enhancement as presented in Figure 2 i.e., luminance and noise can be "decomposed" to a certain extent in the Fourier domain and HR image and its LR versions share similar amplitude patterns. We also design the specific blocks to use these characteristics and solve the high-resolution constraint issue, which have not been explored in previous works.
HDR Imaging, There are some high dynamic range (HDR) reconstruction works (Salih al. 2012 Wang & Yoon| (202T) that are related to our low-light image enhancement. The HDR reconstruction also can be grouped into multi-image HDR reconstruction and single-image reconstruction. The multi-image methods require to fuse the multiple bracketed exposure low dynamic range (LDR) images. To mitigate artifacts caused by image fusion, several technologies (Srikantha & Sidibe methods, deep learning has achieved impressive per- formance. In addition to learning end-to-end LDR-tt-HDR networks (Kalantari & Ramamoorthi
2012) have been proposed. For single-imag
2017a Wu et al. 2018 Yang et al. 2018 Zhang & Lalonde 2017 Hu et al. | Eilertsen et al,
2017) some methods either synthesize multiple LDR images with different exposures (Endo al.
2017 or model the inverse process of the image formation pipeline (Liu et al. (222)) Besides, some works also focus on HDR reconstruction and denoising. For example, (Hu et al.) (022a)) proposes joint multi-scale denoising and tone- mapping framework, which prevents the cone-mapping operator from overwhelming the denoising operator. (Chen et al., (202b) takes the noise and quantization into consideration for designing the HDR reconstruction network.
B ANALYSIS OF UHD-LL DATASET
We first show the intensity histograms and the SNR distribution of our UHD-LL dataset in Figure 8 The SNR is computed using the same algorithm as the recent LLIE method (Xu et al. 2022). As shown in Figure and Figure b)), when splitting the training and testing sets, we make the pixel intensity distributions of the training and testing sets consistent to guarantee the rationality of the dataset split. We also plot the SNR distribution of the dataset show the noise levels in Figure c). The SNR distribution suggests the wide and challenging SNR ranges of our dataset. We show more samples and amplify the resolution of our UHD-LL data in Figure 9
Light
Light
UHD
Ground Truth
Ground Truth
Probaiity
Prooabilty
Intonsity
Intersity
SNR
(a) Intensity distribution (train)
(b) Intensity distribution (test)
(c) SNR (whole)
Figure 8: Distribution analysis of the UHDD-L dataset. Pixel intensity histograms of images for low-noise (green) and normal-clear (blue) from the training and test partitions are shown in (a) and (b), suggesting their consistent pixel intensity distributions. (c) The whole datasct covers a wide range of SNR distribution ranging from to 30 and the SNR values center at the range of [0,10], indicating the challenging noise levels.
14

---

Published as a conference paper at ICLR 2023
Figure 9: Samples of the UHD-LL dataset. The UHD--L dataset contains 2,150 pairs of 4K UHD low- noise/normal-clar data, covering real noises, diverse darkness levels, and large variety of scenes.
C FURTHER ANALYSIS OF MOTIVATION
Recall that in Sec. 2.1 we discussed two observations that serve the motivation to design net- work. In particular, (a) Swapping the amplitude of a low-noise image with that of its corresponding normal-clear image produces a normal-noise image and low-clear image, and (b) The amplitude patterns of an HR normal-clear image and its LR versions are similar and are different from the corresponding HR low-noise counterpart.
We first show more motivation cases in Figures T0 and
12 These visual results suggest the
same tendency the motivations shown the main paper.
Amplitude
compusitional
normal-clear
Iow-clear
normal-clcar
nermal-clear
normal-clear
Iow-nolse
Amplitude
Ampuitude
Amplitude
Amplitude
low-nnosee
normal-noioe
(a)
(b)
Figure 10: Examples of our motivations.
To further analyze our first motivation, we compare the luminance and noise of real normal-clear and low-noise images and compositional low-clear and normal-noise images. To compare the lumi- nance similarity, we compute the average luminance. For real noise level measurement, there is no corresponding metric. Thus, we use the recent multi-scale Transformer MUSIQ (Ke et al. 2021] 1) for image quality assessment. MUSIQ is not sensitive to luminance changes. Moreover, can be used to measure the noise level as its training dataset contains noisy data and it shows rate-offthe. art performance for assessing the quality of natural images. A large MUSIQ value reflects better image quality with less noise and artifacts. We select 50 images from the UHD-LL dataset and compare the average scores. We present the quantitative results in Table
15

---

Published as a conference paper at ICLR 2023
Amplitude
Phase
pompaet sene
NK Honnsamnic
Eall resoluties
normal-elear
Amplitude
Phase
Amplitade
Anglitude
Amplinade
Amplinud
low-nuise
normal-nuise
Figure Examples our motivations.
Amplinskc
Phose
composttonal
Eall resolution
normall-llaa
normal-clene
normal-clear
low-nolse
Amslitndd:
Phass
Amplitade
Amplicude
Amplisude
Amplinude
low-noise
normall-nose
Figure 12: Examples of motivations.
Table 6: Quantitative results to support motivation (a). The mcasurement metrics the average luminance value/MUSIQ value.
Real
Compositional
normal clear -ormal-clear
low-note noise
low-clear
normal- noise
UHD-LL dataset
147.25/61.22
20.50/32.75
23.25/55.50
125.00/33.55
As presented in T the real normal-clear images have similar luminance values with the com- positional normal-noise images while they have similar high MUSIQ values with the compositional low-elear images. Similarly, the real low-noise images have similar luminance values with the com- positional low-clear images while they have similar low MUSIQ values with the compositional normal-noise images. The results further suggest that luminance and noise can be decomposed to certain extent in the Fourier domain. Specifically, luminance would manifest as amplitude while noise is closely related phase.
For the second motivation, it is difficult to quantify the similarity of amplitude spectrum of different sizes as cannot be directly interpolated. The full-reference metrics cannot be used in this situation. Hence, we show more visual examples in Figure (o),,, Figure [b),,, and Figure 12 b). The extra examples support our motivation.
D VISUALIZATION IN THE NETWORK
We show the changes of amplitude and phase in our proposed UHDFour network in Figure 13 shown, the amplitude and phase of our final result are similar with those of ground truth. Moreover, the amplitude and phase of the low-resolution output js are also similar with those of its corre- sponding ground truth ys. wish emphasize that although noise is related to phase, cannot be explicitly represented in phase imagery format phase represents the initial position of the wave. Only the combination of amplitude and phase can express a complete image. Moreover, the fea- ture domain, such relevance is more difficult to be represented in imagery format. Thus, suggest to see the similarity between the final result and ground truth, instead of the intermediate features and phase.
As
16

---

Published as a conference paper at ICLR 2023
As 2e10
DEEK
1A
LENM
ITANN
Figure 13: Visualization of the amplitude and phase changes in the proposed UHDFour network. We show the amplitude and phase components yellow boxes. For visualization, show the amplitude and phase imagery format with common transformations. Note that phase periodic and cannot accurately represented in imagery format. is the corresponding ground truth. ys is the 8x downsampled ground truth.
E MORE RESULTS ON RELEASED MODELS
We present more comparisons between state of the arts for restoring UHD low-light images in Fig- ures 14 15 and T6 This is similar to Figurefoor the main paper where we compare methods using their original released models. As shown, all existing models cannot handle the UHD low-light images well. Since we cannot infer the full-resolution results of SNR-Aware Xu et al. 2022) on UHD images, despite using a GPU with 48G memory, some obvious borders appear in its results due to the stitching strategy. The phenomenon also indicates the commonly usedd stitching strategy in previous UHD data processing methods is inapplicable to the challenging UHD low-light image enhancement task.
F MORE RESULTS ON RETRAINED MODELS ON UHD-LL
We provide more visual comparisons of our method with retrained state-of-the-art methods on the UHD-LL dataset in Figures 17 and 18
As the results shown, for UHD low-light image enhancement, the retrained models on the UHD-LL dataset still cannot achieve satisfactory results. Noise and artifacts can still be found their results. The results suggest that joint luminance enhancement and noise removal in the spatial domain is difficult. Our solution effectively handles this challenging problem by embedding Fourier transform in cascaded network, in which luminance and noise can be decomposed to certain extent and are processed separately.
G MORE RESULTS ON RETRAINED MODELS ON LOL-V1 AND LOL-v2 DATASETS
We also provide more visual comparisons our method with models that were pre-trained or fine-tuned on the LOL-v] and LOL-v2 datasets in Figures 19 20 and 21]
As for the low-light images LOL-v1 and LOL-v2 datasets, even though the mild noise and low- resolution images prohibit from showing the full potential of our method in removing noise and processing high-resolution images, our method still achieves satisfactory performance. The results suggest the potential of our solution in different cireumstances.
H ABLATION STUDY
We show some visual results of the ablated models in Figure 22 Without the Fourier branch (#1), the ablated model cannot effectively enhance luminance and remove noise. Although the result of
17

---

Published as a conference paper at ICLR 2023
Tabble 7: The computational and time of FFT/IFFT operations for processing different scales of features.
compute FLOPs (in M) and running time (in second).
FFT
IFFT
Scales
FLOPs
Running time
FLOPs
Running time
8x +
35.22
2.62x 10
35.22
5,46x 10
16x
7.77
8.10x10
7.77
1.39 10-4
32x
1.68
6,34x10-5
1.68
1.17x10-4
#2 looks better than #1, the Spatial branch (#2) also affects the final result. Directly replacing the FouSpa Block with the Residual Block of comparable parameters (#3) cannot obtain a satisfactory result, suggesting the good performance of the FouSpa Block not because of the use of more parameters. Removing the Amplitude Modulation (#5) results in the visually unpleasing result. The Phase Guidance (#6) and Spatial branch (#7) in the Adjustment Block also contribute to the good performance of the full model. Directly replacing the Adjustment Block with the Residual Block of comparable parameters (#10) still cannot obtain satisfactory result. The introduction of the estimation of low-resolution result (#11) also leads to clear result. The visual comparisons further show the significance of the proposed FouSpa Block and Adjustment Block in dealing with the intricate issue joint luminance enhancement noise removal while remaining efficient.
Additionally, we list the computational and time costs of FFT/IFFT operations for processing differ- ent scales of features. In our network, we the feature channels 16 and use three different scales 8x, 16x, and 32x downsampled original resolution (i.e., 3840 2160). The results are presented Table 7 The FFT and IFFT operations have same computational cost. The difference in running time may be because of the different optimization strategies used in PyTorch.
18

---

Published as a conference paper at ICLR 2023
Corr
Corr
Corr
Corr
Input
DRBN
Zero-DCE
Zero-DCE++
Corr
Core
Corr
Core
NPE
SRIE
RUAS-LOL
RUAS-MITSK
Corr
Corn
Corn
Corn
RUAS-DarkFaad
Zhao al.-MITSK
Zhao al.-LOL
EnlightenGAN
Corn
Corn
Corr
Corn
Afifi al.
SCI-easy
SCI-medium
SCI-difficlt
Corr
Corr
Corr
Corr
ANR-Aware-LOOL resice
SNR- TOR-Aware--LLN LOLvI
SNR Aware-
LOLVI
WAR-Aware-LOLYY
Cofr
Corr
Carr
Corr
S.R--Aarr-SSyn..... SNR-Aware- SSanarechyatttehh
URetinex-Ne Net
GT
Figure 14: Visual comparison
released state
arts restoring UHD low-light image. The
compared methods include DRBN Yang et al., (2020a), Zero-DCE (Guo et al. 2020). Zero-DCE++ (Li al.
2021b), NPE (Wang et al. (013). SRIE (Fu et al. [2066). RUAS-LOL. (Liu al.
202lb RUAS-MITSK (Liu
al. 2021b), RUAS-DarkFas Face Liu al. 2021b). Zhao al.-MIT5K (Zhao 2021). SCI-easy (Ma et all
(02T), Zhao et al.-OL
(Zhao al. 2021]. EnlightenGAN (Jiagg et 2021), Afifi al. (Afii all
2022).
SCI-medium (Ma et al., 2022). SCI-dificuulrMa et al.) 2022). SNR-Aware-LO
rreise Xu et al.
2022).
SNR-Aware-LoO Liare-LOLvlleecee
(Xu et al. 2022). SNR-Aware. LOLV2restse (Xu et al. (022), SNR- Aware-
LOLv2atteh (Xu 2022)) SNR MON--Aware- Sware-Syn.esise. el. 2022). SNR- SNR-Aware Aware-SSn.tttee (Xu at 2022)
"RRetine-Neetiee al. 2022
19

---

Published as a conference paper at ICLR 2023
Input
DRBN
Zero-DCU
Zero-DCE++
NPE
SRIE
RUAS-LOL
RUAS-MITSK
TUAS-DarkFaad
Zhao al.-MITSK
Zhao al.-LOL
EnlightenGAN
Afif et al.
SCI-casy
SCI-medium
SCI-diffiult
Aeee--war--LLL ressse
ANR-Awaree LOLV
SNR Aware-LOLv2..m
AwRreware- - LOLv
SNN-Aware-Syn....
URetinex-Ne
GT
Figure 15: Visual comparison
released state arts restoring UHD low-light image. The
compared methods include DRBN Yang et al., (2020a), Zero-DCE (Guo et al. 2020). Zero-DCE++ (Li al.
2021b), NPE (Wang et al. (013). SRIE (Fu et al. (2066), RUAS-LOL. (Liu al.
2021b RUAS-MITSK (Liu
al. 2021b), RUAS-DarkFas Face (iu al. 2021b). Zhao al.-MIT5K (Zhao 2021). SCI-easy (Ma et all
(02T), Zhao et al.-LL
(Zhao al. 2021]. EnlightenGAN (Jang et 2021), Afifi al. (Afii all
2022).
SCI-medium (Ma et al., 2022). SCI-dificuulrMa et al.) 2022). SNR-Aware-LOO
reeise Xu et al.
2022). MIRR-Awrr--LOLy
(Xu et al. 2022). SNR-Aware- LOLV2restse (Xu et al. (022), SNR- Aware-
LOLv2atteh (Xu 2022)) SNR MON--Aware- Sware-Syn.esise. el. 2022). SNR- SNR-Aware Awwre-Synatteeee... (Xu at 2022)
PRReinex-Net(VWie al. 2022
20

---

Published as a conference paper at ICLR 2023
Input
DRBN
Zero-De DCE
Zero-DCE++
NPE
SRIE
RUAS-LOI.
RUAS-MITSK
RUAS-DarkFace
Zhao al.-MITSK
Zhao al.-LOL
EnlightenGAN
et al.
SCI-easy
SCI-medium
SCI-diffiult
atitch LOLvan-LLLLeeeeeee
TOR-Aware-LOOL-//
SNR-Aware Sware-Syn.cited
URetinex-Ne Net
GT
Figure 16: Visual comparison the released state of the arts for restoring UHD low-light image. The compared methods include DRBN Yang et al., 2020a), Zero-DCB (Guo et al. 2020). Zero-DCE++ (Li et al.
2021b), NPE (Wang et al.) al.]|2011 (203). SRIE (Fu al. 2016). RUAS-LOL (Liu al. 2021b). RUAS- (Liu
al. 2021b). RUAS-DarkFace (Lu al. (021b). Zhao et al.-MITSK (Zhao et (022)) Zhao et aL.-LOL
(Zhao et al. 2021). EnlightenGANN (Jing el al. 20021), Afifi et al. (Afii et al.) 2021). SCI-easy (Ma el all
2022). ) SCI-medium (Ma et al., (022). SCI-diftcuul((Ma et al. 2022). SNR-Aware-LOO LOLy resize [X ot all
[Xu al. 2022), SNR-Aware LOL (Xu al. 2022), Aware LOLv2stach (Xu ct. (022). SNR-Aware- Awwre-Waar-aaaae SyDreiize al. 2022), SNR-Aware Synnre-SS,tttttt et 2022)
2022
AOR-Awarr-LOLv
URetinex-Net al. 2022).
21

---

Published as a conference paper at ICLR 2023
Input
Zero-DCE
Zero-DCE++
RUAS
Zhao al.
Afifi al.
SCI
SNR-Aware
Uformer
Restormer
UHDFour (Ours)
GT
Figure 17: Visual comparison the retrained state of the on the UHD-LI LL dataset. The compared methods
include Zero-DCE (Guo et al. 2020), Zero-Do crro-DCE++ (Li al. 20211), RUAS (Liu et al. 20216), Zhao et al.
ZZhoo et al. 2021)
Afifi et al.
Afii et al. 2021), SCI (Ma et al 2022
SNR-Awar (Xu et al 2022) Uformer
Wang et al.
2022 and Restormer (Zamir al. 2022
22

---

Published as a conference paper at ICLR 2023
Input
Zero-DCB
+ero-DCE++
RUAS
Zhao al.
Afifi al.
SCI
SNR-Awar
Uformer
Restormer
UHDFour (Ours)
GT
Figure 18: Visual comparison the retrained state of the on the UHD-LL LL dataset. The compared methods
include Zero-DCE (Guo et al. 2020), Zero-De Prro-DCE++ (Li ct al. 20211). RUAS (Liu et al. 20216), Zhao et al.
ZZhao et al. 2021)
Afifi et al.
Afifi et al. 2021), SCI (Ma et al. 2022
SNR-Awar (Xu et al 2022) Uformer
Wang et al.
2022 and Restormer (aamir al. 2022
23

---

Published as a conference paper at ICLR 2023
Input
Retinex-
Zero-DCE
AGLLNet
Zhao et al.
RUAS
SCI
URetinex-Net
UHDFour (Ours)
GT
Figure 19: Visual comparison on the LOL-vi dataset. The compared methods include Retinex-Net Wei et al.
(201)) Zero-DEE (Guo et al. 2020), AGL.NNet Lv et al. 2021), Zhao et al. (Zhao al. 2021 RUAS (Liu
202Tb), and URetinex-N Net Wu
2022
24

---

Published as a conference paper at ICLR 2023
Input
Retinex-Ne
Zero-DCE
AGLLNet
Zhao et al.
RUAS
SCI
URetinex-Net
UHDFour (Ours)
GT
Figure 20: Visual comparison on the LOL-vi dataset. The compared methods include Retinex-Net Wei et al.
(018). 8) Zero-DCE (Guo al. 2020), AGL Ly et al. (202T) Zhao al.
(Zhao al. 2021 RDAS (Liu
2021b), and URetinex Net Wu al. 2022
25

---

Published as a conference paper at ICLR 2023
Input
Retinex-No Net
Zero-DCE
AGLLNet
Zhao et al.
RUAS
SCI
UHDFoour (Ours)
GT
Figure 21: Visual comparison the LOL-v2 dataset. The compared methods include Retinex-Net Wei et al.
(2018), Zero-DCE (Guo et al. 2020), AGLLNet Ly 2021), Zhao
(Zhao
2021 and RUAS
Liu et al. 2021b
26

---

Published as a conference paper at ICLR 2023
#5
#6
input
(full model)
GT
Figure 22: Visual results of ablated models.
27


Reviews:
[{"review_id": "tHZsp7OcXsX", "reviewer": "Reviewer_ohoL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To tackle the problem of joint luminance enhancement and denoising in low-light imaging, the author proposed 1) a new network architecture, UHDFour, that utilizes authors' observations on noise and signal luminance's relation in the amplitude and phase images in the frequency domain, and 2) a new dataset, UHD-LL, that contains high-resolution 4K low-light / normal light pairs.", "summary_of_the_review": "Overall, this is a solid paper with interesting insights and new datasets. If the authors and properly address the concerns in the weakness section, I'm happy to upgrade the rating from marginally accept to accept.", "strength_and_weaknesses": "### Strength \n1. A reasonably sized, high-resolution new dataset for joint luminance enhancement and denoising is provided.\n1. The proposed network architecture cleverly mixes and matches the amplitude and phase of the low light input and normal light GT to constrain the denoising network.\n### Weakness\n1. The importance of the Fourier branch (FB) compared to a more conventional spatial branch (SB) in FourSpa is less than expected and should be further validated. Swapping FB with SB brings about a $0.6~dB$ difference in PSNR, as ablation studies #1 and #2 show, which made me wonder if 2xSB (i.e. increase the size of the SB) would have similar performance as FB+SB in FourSpa. Moreover, for the adj. block, it's necessary to show the performance of SB only and SB increase to the comparable size of SB+AM+PG. Lastly, there should be a Fourier-free architecture where there are only SBs (of the equivalent size of SB + FB + AM + PG + SB, e.g. #9) in both FourSpa and Adj. Block. The author should consider adding those in the final version.\n1. What are the computational and time costs of FFT/IFFT-related operations? It will be useful if the author can add this to the ablation study table.\n1. Even for consumer audio/video market, \"HDR\" and \"4K\" are oftentimes mentioned together. Moreover, HDR is common for low-light scenes as light sources can often be seen in night-time photography. However, HDR scenes are not considered in this paper. I suggest the author mention such drawbacks of the proposed dataset in discussion and/or indication of the dynamic range of the proposed datasets (i.e. the specs of the cameras used for the data capture).\n1. Also, since an important aspect of this paper is luminance enhancement, it's natural to discuss HDR tone-mapping. A simple search on [joint HDR and deoising](https://scholar.google.com/scholar?hl=en&q=joint+hdr+denoising) leads to many recent papers on the topics. I suggest the authors consider including such papers in the related work section.\n1. Denoising research is at a stage where one has to \"pixel peep\" to tell the difference. Please provide zoom-in insets of the denoising performance in the result figures.\n1. The proposed UHDFour method is only tested on one external dataset, LOL. It's reasonable to include at least another state-of-the-art dataset\n1. For the dataset comparison table, the authors should add SID datasets. The argument that SID only works for particular Bayer pattern is not valid. SID data's Bayer or Fuji pattern raw can be demosaiced to RGB raw, meanwhile All the authors' data also undergoes some demosaicing too.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_t0pNNpqwJR", "reviewer": "Reviewer_Fqm7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propsoe the UHDFour solution for UHD enhancement. This paper introduce a new dataset for 4K UHD, called UHD LLIE. Also this paper designed a module utilize the frequency domain.", "summary_of_the_review": "I think this paper is complete and has enough novelty.", "strength_and_weaknesses": "Strength:\n1. This paper is well written and easy to understand.\n2. The performance is great.\n3. The novelty is enough for ICLR. The feature modeling in frequency domain and the dataset.\n\nWeakness.\n1. I'm little confused about table 2. Why the performance of proposed method is not shown in Table 2?\n2. I suggest authors also show the Amplitude and Phase in the network. People can see how the network improve the Amplitude and Phase.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OHKbaqPXc6", "reviewer": "Reviewer_yq2M", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the Ultra-High-Definition (UHD) low-light image enhancement, which is a new task. Different from previous low-light image enhancement, the task presented in the paper faces more challenging and practical issues such as dealing with the intricate issue of joint luminance enhancement and noise removal while remaining efficient. The paper first shows the limitations of previous methods and datasets for processing the UHD low-light images.  Then, the paper proposes to embed Fourier transform into a cascaded network to address these issues. The designs are interesting and insightful, which are based on the unique characteristics in the Fourier domain. In comparison to spatial domain, these characteristics of Fourier domain make the task easy to deal with. Besides these designs, the paper also presents the first real UHD low-light image enhancement with paired data. The paper also provides some techniques about how to capture the dataset of this kind. The proposed solution achieves the best performance on UHD low-light image enhancement and fast inference speed. Also, the solution can be extended and used for processing existing low-light image dataset.", "summary_of_the_review": "The paper presents a novel solution and a new dataset. Moreover, the analysis and motivations of this work provide new insights into this challenging task. The experiments are sufficient, and the results are convincing. These are the important factors in my rating. ", "strength_and_weaknesses": "Strengths:\n\n+ New task. UHD low-light image enhancement task is becoming more and more important, especially on mobile devices. The reviewer is happy to see there is a paper focusing on this issue and proposing a new solution and the first dataset.\n+ Novel solution. Facing challenging issues such as joint luminance enhancement and noise removal under the ultra-high-resolution constraint, this paper proposes an interesting and insightful solution. Different from previous spatial methods, this method tasks full use of the unique characteristics of Fourier domain for this task. In the Fourier domain, the amplitude and phase of a low-light image can be separately processed to avoid amplifying noise when enhancing luminance. Moreover, the proposed method can be scalable to UHD images by implementing amplitude and phase enhancement under the low-resolution regime and then adjusting the high-resolution scale with few computations. The solution is novel and interesting.\n+ New dataset. This paper contributes the first UHD low-light image enhancement dataset with paired data. The dataset not only provides sufficient and diverse data but also presents some techniques which show how to minimize geometric and photometric misalignment when capturing the paired low-light/normal-light data. The data collection details provide some insights into the pair data collection.\n+ Systematical analysis. This paper systematically analyzes the performance of existing low-light image enhancement methods for processing UHD images. The baselines cover almost all state-of-the-art methods, and the experiments include quantitative and quantitative results. From the results, it is clear to see the advantages and disadvantages of previous methods. \n+ Outstanding performance. The proposed method in the paper achieves state-of-the-art performance for UHD low-light image enhancement with the fastest inference speed. Besides, the proposed method can be slightly modified and then achieves state-of-the-art performance on existing low-light image enhancement dataset despite it is not designed for data of this sort.\n+ Sufficient experiment and analysis. The paper provides sufficient comparison experiments to show the advantages of the proposed method in the main paper and appendix. In addition, the ablation studies and motivation analysis are convincing.\n\nWeaknesses:\n\n+ The paper analyzes the performance of existing low-light image enhancement for processing UHD low-light images. The baselines are all deep learning-based methods. The reviewer wonders whether traditional methods still cannot solve this challenging issue either. It would be good if some traditional methods could be included for comparison in the final version.\n+ For the Sec. 4.1 Benchmarking existing models, the quantitative scores in terms of non-reference metrics could be removed when the paired data is available. It is almost common that non-reference metrics are not much reliable, especially for low-light images. \n+ In Figures 6-8 and Tables 2-4, for each method, the corresponding reference should be provided to improve the readability of the paper. It is the same for the figures in the appendix. \n+ The reviewer wonders whether the main idea in Fourier domain can be extended to other UHD image related low-level tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "1QylgiySaBX", "reviewer": "Reviewer_aXaB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method (UHDFour) for enhancing ultra high-resolution low-light noisy images. The proposed method is based on two observations. The first observation is that most luminance information exists on amplitudes while noise exists on phases of the Fourier domain. This observation drives the proposed UHDFour method to process amplitudes and phases separately for luminance enhancement and noise removal. The second observation is that a high-resolution image and its corresponding low-resolution version have similar amplitude patterns, which results in a cascaded network design that first processes a low-resolution image and then processes a high-resolution one. The authors construct a new ultra high-resolution image dataset (UHD-LL). The authors conduct experiments to justify the motivations and the effectiveness of proposed method.", "summary_of_the_review": "I am on the positive side of this paper but with some questions listed in the above section. I am willing to raise my rating if the authors can address them.", "strength_and_weaknesses": "Strong points.\n1. A new method is proposed for ultra high-resolution low-light noisy image enhancement with notable performance advantages against existing methods.\n2. The network design has a clear motivation with some interesting experiments to justify the observations.\n3. A new ultra high-resolution dataset of 2150 low-noise/normal-clear paired 4K images is constructed.\n4. The paper is written well and easy to follow.\n5. Code and dataset will be released.\n\nWeak points.\n1. The first observation is justified via visual illustrations and quantitative experiments while the second one is illustrated by visual comparisons. It would be better if the authors can provide some interpretations. Can authors explain from the FFT why these phenomena happen? The second observation only mentions that HR images and their corresponding LR images share similar amplitude patterns, but how about the phases? Do they have similar patterns? If not, would the noise removal in LR version affect that in HR images?\n\n2. In the 3rd paragraph, Introduction, there are some vague descriptions when talking about existing methods, e.g., \"existing methods are mainly...\" and \"some studies...\". I suggest adding references here so that readers can have direct references.\n\n3. The LRNet/FouSpa block designs seem not to be aligned with the observations. For the FouSpa block, it is mentioned that \"we observe that luminance and noise can be decomposed in the Fourier domain\", but the design (Fig. 4) shows that the Fourier branch only separately processes the A and P, then combines them together using IFFT. There is still a Spatial branch that works with the Fourier branch. It is not clear why there should still have a spatial branch and how the Fourier branch enhances luminance in A and removes noise in P. Regarding the LRNet design (Fig. 3), it is not clear why FFT and Conv layers are paralleled. Further, do A_r and P_r have any supervision signals? Would the authors show some intermediate visualizations of A_r and P_r?\n\n4. Regarding the dataset acquisition, it seems that images are taken in normal-light scenes, and low-light images are captured with under-exposures. In this case, it seems the noise would not be very severe (e.g., in Figure 5 it is hard to see noise). This makes this paper more related to under-exposure image enhancement but not low-light image enhancement. It would be better if the authors could provide more justifications for this issue. The dataset split is 2000 for training and 115 pairs for test (The train/test ratio is 20:1), which I think is extreme. It would be better if the authors could justify this.\n\n5. A minor issue is that \"Table 5\" on page 8 should be \"Table 4\".\n\n6. In Section A of the Appendix, I think it would be better if the authors can discuss how FFT is used in other low-level vision tasks, e.g., image deraining, denoising, and deblurring, and highlight the differences to the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%