DMFourLLIE: Dual-Stage and Multi-Branch Fourier Network for Low-Light Image Enhancement

Authors: tongshun  zhang
pingping  liu
ming  zhao
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 04:48:38

Abstract: In the Fourier frequency domain, luminance information is primarily encoded in the amplitude component, while spatial structure information is significantly contained within the phase component. Existing low-light image enhancement techniques using Fourier transform have mainly focused on amplifying the amplitude component and simply replicating the phase component, an approach that often leads to color distortions and noise issues. In this paper, we propose a Dual-Stage Multi-Branch Fourier Low-Light Image Enhancement (DMFourLLIE) framework to address these limitations by emphasizing the phase component's role in preserving image structure and detail. The first stage integrates structural information from infrared images to enhance the phase component and employs a luminance-attention mechanism in the luminance-chrominance color space to precisely control amplitude enhancement. The second stage combines multi-scale and Fourier convolutional branches for robust image reconstruction, effectively recovering spatial structures and textures. This dual-branch joint optimization process ensures that complex image information is retained, overcoming the limitations of previous methods that neglected the interplay between amplitude and phase. Extensive experiments across multiple datasets demonstrate that DMFourLLIE outperforms current state-of-the-art methods in low-light image enhancement.

Main paper: 
 fourier frequency information: 
 Firstly, we provide a brief introduction to Fourier frequency information. Fourier frequency information refers to the representation of an input image 𝑥 in the Fourier space 𝑋 through a transform function F . The input image 𝑥 has a shape of 𝐻 × 𝑊 , where 𝐻 represents the height and 𝑊 represents the width. F is expressed as:EQUATIONwhere ℎ, 𝑤 are the coordinates in the spatial space and 𝑢, 𝑣 are the coordinates in the Fourier space, 𝑗 is the imaginary unit, the inverse process F is denoted as F -1 . Each complex component 𝑋 (𝑢, 𝑣) can be represented by the amplitude component A (𝑋 (𝑢, 𝑣)) and phase component P (𝑋 (𝑢, 𝑣)). These two components are expressed as:EQUATIONEQUATIONwhere 𝑅(𝑥) and 𝐼 (𝑥) represent the real and imaginary part of 𝑋 (𝑢, 𝑣) respectively. In our method, the Fourier transform and inverse procedure is computed independently on each channel of feature maps.As shown in Fig . 4 , the low-light input image 𝐼 𝑙𝑜𝑤 is fed into the Fourier reconstruction phase, resulting in the output of the first stage, 𝐼 𝑠1 . This stage consists of six identical components with skip connections, where the first component is visualized in detail.The LA-Net and pretrained models of Infrared Branch are only activated in the first component, with their outputs being passed and processed in the subsequent five components. The branches are described as follows:Fourier Branch: The input image is first transformed into the Fourier space to obtain the amplitude component 𝐴 𝑖𝑛 and the phase component 𝑃 𝑖𝑛 . Once transformed into the Fourier space, the assumption of spatial invariance no longer holds [8] . Therefore, we apply a 1×1 convolutional layer and LeakyReLU activation function in this space to extract the amplitude component, to avoid information loss. The amplitude and phase components are each extracted through a 1 × 1 convolutional layer with LeakyReLU activation, resulting in 𝐴 𝑜𝑢𝑡 and 𝑃 𝑜𝑢𝑡 . Unlike existing methods [14, 37] , which only enhance the amplitude component and adopt a strategy of replicating the phase component, our analysis in Fig . 2 emphasizes the importance of phase information for the structure and content of images, indicating that merely replicating the phase component is not the best strategy.Luminance Branch: To address the potential overflow issue in the Fourier domain space, we learn a luminance attention map through the luminance branch to precisely guide the enhancement level of the amplitude component in different areas, correctly enhancing underexposed areas while avoiding over-enhancement of correctly exposed areas. According to the study by Bread [10] , among various color spaces, the luminance component (Y) in the YCbCr color space performs best and is least affected by noise. Inspired by this finding, we predict the luminance attention map 𝐿 𝑎𝑡𝑡 through LA-Net (Luminance Attention Net) based on the luminance component 𝑌 𝑙𝑜𝑤 from the YCbCr color space. LA-Net follows the classic U-Net architecture with an encoder-decoder structure. This process and the learning target can be represented as:EQUATIONwhere 𝑌 𝑙𝑜𝑤 and 𝑌 𝑔𝑡 represent the luminance components (Y channel) extracted from the input low-light image and the ground truth image, respectively. In the predicted 𝐿 𝑎𝑡𝑡 , brighter areas correspond to darker areas in the input low-light image, and vice versa. Then, 𝐿 𝑎𝑡𝑡 is transformed into the Fourier space to obtain the amplitude component 𝐴 𝑙𝑢𝑚 , and features are extracted through two 1 × 1 convolutional layers with LeakyReLU activation, resulting in Ã𝑙𝑢𝑚 .Luminance Augment: To address the issue of varying brightness across different regions and the potential overflow problem during the transfer of amplitude components in the Fourier space, we propose a Luminance Augment operation. This operation leverages 𝐿 𝑎𝑡𝑡 to make 𝐴 𝑜𝑢𝑡 pay more attention to darker areas while reducing focus on brighter areas. Specifically, Ã𝑙𝑢𝑚 undergoes a sigmoid operation and is then element-wise multiplied by 𝐴𝑜𝑢𝑡, and through a residual connection, we obtain the enhanced amplitude component 𝐴 𝑎𝑢𝑔 . The process can be described as follows:EQUATIONthrough the aforementioned operation, 𝐴 𝑎𝑢𝑔 effectively adjusts the attention towards enhancing different areas, thereby more accurately amplifying the amplitude component.Infrared Branch: The input 𝐼 𝑙𝑜𝑤 is processed by a pre-trained RGB-to-TIR image translation model [15, 19] , which focuses on edge preservation. This model, trained on the VIPER [32] , FLIR-ADAS, and STheReO [50] datasets, generates infrared images from low-light scenes, improving infrared image performance in dark conditions. This process retains spatial texture information and generates the corresponding infrared image 𝐼 𝑖𝑛𝑓 from 𝐼 𝑙𝑜𝑤 . Infrared images, unaffected by illumination, color, and distortion, enhance object outlines. Since contour information primarily resides in the phase components of the Fourier domain, we extract the phase component 𝑃 𝑖𝑛𝑓 of 𝐼 𝑖𝑛𝑓 using FFT. Features are then extracted through two 1 × 1 convolutional layers with LeakyReLU activation, resulting in P𝑖𝑛𝑓 .Infrared Augment: The phase component in the Fourier space is crucial for the structure and content of an image. Considering the rich structural information of infrared images and their high robustness in dark scene tasks, we are inspired by the transposed attention in Restormer [51] and design an Infrared Augment operation. This operation achieves phase attention maps 𝑀 𝑝ℎ𝑎 through cross-modal fusion of phase component features of infrared and visible light images in the Fourier space. The phase attention map 𝑀 𝑝ℎ𝑎 is described as follows:EQUATIONthen, 𝑃 𝑜𝑢𝑡 is multiplied by 𝑀 𝑝ℎ𝑎 and added through a residual connection, resulting in the enhanced phase component 𝑃 𝑎𝑢𝑔 . This process is as follows:EQUATIONto this end, we obtain the enhanced amplitude component 𝐴 𝑎𝑢𝑔 and phase component 𝑃 𝑎𝑢𝑔 , which are then transformed back into the spatial domain through the inverse Fourier transform to yield the output 𝐼 𝑠1 :EQUATIONWhile the Fourier space conveys global information, it lacks the enhancement of spatial details and may introduce additional noise and feature loss during the Fourier transform and feature transfer processes. To address this, we design the Spatial and Texture Multi-scale Convolution Branch: The structure of the Multiscale Convolution is depicted below Fig. 5 . In this branch, multiple convolutional layers with varying kernel sizes are utilized to capture a wide range of spatial features across multiple scales and enhance the overall representation of spatial structures. To reduce computational cost, we downscale the dimension from (C=64) to (C/4=16), and gradually increase the kernel size by 2 from 3 × 3 to 9 × 9. After applying a sigmoid operation to the reduced features, they are multiplied with the output features of different convolutional kernels, then concatenated, restoring the dimension to (C=64). The features fused with multi-scale spatial information are processed through a 1 × 1 convolutional layer and combined with the input via a residual connection.Fourier Convolution Branch: Fast Fourier Convolution (FFC) [4] has been proven to handle high-resolution restoration cases with strong periodic texture, achieving robust enhancement at the resolution level. It consists of two branches: 1) a local branch that uses regular convolution, and 2) a global branch that convolves features after a fast Fourier transform. The outputs of both branches are then merged to obtain a larger receptive field and local invariance during the restoration process [36] . However, such a powerful model is unable to learn a reasonable overall spatial structure [5] . Therefore, we pair it with the Multi-scale Convolution to form a dual-path structure, complementing it to achieve precise overall spatial structure through the final feature fusion.To further improve image quality from both qualitative and quantitative aspects by considering perceptual information and regional differences, our loss function consists of four parts:EQUATIONwhere 𝜆 denotes the loss weights, we empirically set 𝜆 1 , 𝜆 2 , 𝜆 3 , 𝜆 4 = [0.5, 1, 0.2, 0.1]. L 𝑠1 = ∥𝐼 𝑠1 -𝐺𝑇 ∥ 2 represents the 𝑙 2 loss between the preliminary enhanced result 𝐼 𝑠1 and ground truth image 𝐺𝑇 . Similar to L 𝑠1 , L 𝑠2 = ∥𝐼 𝑠2 -𝐺𝑇 ∥ 2 represents the loss between the final output 𝐼 𝑠2 and 𝐺𝑇 . L 𝑝𝑒𝑟 is the perceptual loss between the 𝐼 𝑠2 and 𝐺𝑇 , which constrains the features extracted from VGG [34] to obtain better visual results. Additionally, to acquire the luminance attention map for correctly guiding enhancement of luminance in image regions by 𝐿𝐴-𝑁 𝑒𝑡, we adopt 𝑙 2 error to measure the prediction error:EQUATIONwhere 𝐿𝐴-𝑁 𝑒𝑡 (𝐼 𝑙𝑜𝑤 ) and 𝐿 𝑎𝑡𝑡 represent the predicted and expected luminance attention maps, respectively.Methods LOL-v2-Real LOL-v2-Syn LSRW-Huawei #Param #FLOPs PSNR ↑ SSIM ↑ LPIPS ↓ PSNR ↑ SSIM ↑ LPIPS ↓ PSNR ↑ SSIM ↑ LPIPS ↓ (M) (G)NPE [38] DMFourLLIE adopts an end-to-end training strategy to jointly optimize the network parameters of both stages. We implement DMFourLLIE in Pytorch. During training, images are randomly cropped to 256 × 256 and augmented with random flipping. An ADAM optimizer with an initial learning rate of 4.0 × 10 -4 is used to optimize the network, along with a multi-step scheduler. The batch size is set to 8 and the total number of training iterations is set to 1.0 × 10 5 . Training takes approximately 18 hours on an NVIDIA 3090 GPU. 
comparison with current methods: 
 In this paper, our DMFourLLIE is compared to thirteen state-ofthe-art LLIE methods, including traditional approaches LIME [11] , Table 4 : Ablation experiments on the loss function.NPE [38] and SRIE [6] , Fourier-based methods FourLLIE [37] , FEC-Net [14] and UHDFour [20] . Deep learning-based methods involved in the comparison are Kind [57] , Kind++ [55] , MIRNet [52] , SGM [47] , HDMNet [22] , SNR-Aware [44] , Bread [10] , SNR-SKF [42] and Retinexformer [2] . All deep learning models are trained on the same datasets using their original public codes, ensuring a fair evaluation.Quantitative results on LOL-v2-Real, LOL-v2-synthesis and LSRW-Huawei datasets. In this work, we use Peak Signalto-Noise Ratio (PSNR), Structural Similarity Index (SSIM) [40] and Learned Perceptual Image Patch Similarity (LPIPS) [54] as our evaluation metrics. Generally, higher PSNR and SSIM as well as lower LPIPS indicate higher similarity between two images.As shown in Tab. 1, we evaluate our approach using the LOL-v2-Real, LOL-v2-synthesis, and LSRW-Huawei datasets. Compared to the current state-of-the-art methods, it is evident that DMFourLLIE achieves nearly the best performance across the PSNR, SSIM, and LPIPS metrics. Furthermore, we also have advantages in terms of model parameters and computational efficiency.Visual Quality Comparisons. Due to space limitations, we select the most recent and superior methods FECNet [14] , SNR-SKF [42] , FourLLIE [37] , UHDFour [20] and Retinexformer [2] for an intuitive comparison with our DMFourLLIE. Fig. 6 shows the comparison results on LSRW-Huawei, LOL-v2-real, and LOL-v2-Synthesis datasets. In LSRW-Huawei, FourLLIE causes overbrightening leading to image overexposure, while SNR-SKF and Retinexformer do not achieve fully brightened visualization results. UHDFour exhibits an issue of insufficient brightness. FECNet, even with better brightening effects, lacks the clarity in motorcycle detail and overall image color (especially the floor) compared to our results, which are closer to ground truth (GT). In LOL-v2-real, FourL-LIE and Retinexformer evidently introduce unavoidable blotchiness, greatly affecting the visualization results. FECNet and SNR-SKF show noticeable noise issues, whereas our method produces the clearest and most natural brightening results. In LOL-v2-synthesis, FECNet, SNR-SKF, FourLLIE, and Retinexformer suffer from insufficient brightness or overexposure issues, as well as some degree of color deviation. UHDFour introduces noise leading to blurry images, while our method achieves visually similar results to the GT.Quantitative results on LIME, VV, DICM, NPE and MEF datasets. We measure the no-reference Natural Image Quality Evaluator (NIQE [28] ) scores on five non-paired datasets. Lower NIQE scores indicate images with higher naturalness quality. Tab. 2 shows the NIQE evaluation results. It can be seen that our DMFourLLIE outperforms most of the existing LLIE methods.We elaborate on the effectiveness of the second stage from three perspectives: (1) As shown qualitatively in Fig. 8 and quantitatively through mean square error metrics, the second stage demonstrates its effectiveness and outperforms other methods; (2) By replacing the second stage of FourLLIE with the second stage of our DM-FourLLIE, as reflected in Tab.5, all performance metrics indicate improvements compared to the FourLLIE baseline; (3) Due to the inevitable information loss introduced during Fourier transform and its inverse in the first stage (refer to Fig. 9 , owing to discretization), and after multiple transformations and network inferences, the refinement provided by the second stage is essential. Moreover, to further validate the limitations in the design of the first-stage loss functions in FECNet and FourLLIE, we present the results of adding L 𝑎𝑚𝑝 in the last column of Tab. 4 (following the design of FECNet and FourLLIE, where L 𝑎𝑚𝑝 is defined as the difference between the enhanced amplitude component 𝐴 𝑎𝑢𝑔 and the ground truth amplitude component 𝐴 𝑔𝑡 ). We found that the performance actually decreases after adding L 𝑎𝑚𝑝 , which again supports our motivational hypothesis that the strategy of solely boosting the amplitude component leads to using degraded images as the learning target for the network.We examined the effects of different models on infrared information extraction, and the impact of incorporating depth and edge maps. In Tab.6, we provide a comprehensive analysis of the impact of different models on infrared information extraction, including the use of depth maps and edge maps. It is well-known that infrared images have significant advantages in extremely low-light conditions. In contrast, depth maps and edge maps are often susceptible to interference from factors such as lighting and color noise. This validates our motivation for using infrared images to enhance the structural integrity of the phase component. The infrared extraction model employed by DMFourLLIE achieves optimal overall performance.On the Dark Face dataset [48] , we conducted object detection experiments under low-light conditions to evaluate the preprocessing effects of the latest Fourier-based methods on advanced visual understanding tasks. The Dark Face dataset consists of 6000 lowlight images with real-world annotations. For testing, we randomly selected 200 images from the dataset and validated using the official YOLOv5 model pre-trained on the COCO [23] dataset. Fig. 10 presents the visual comparison results, where our method outperforms other methods significantly in terms of detection accuracy and recall rate. Notably, in the results of the second row, DMFourL-LIE successfully detects the backpack, while in the fourth row, the detection rate of bicycles is significantly higher than other methods, with other methods showing instances of false detections.Furthermore, through visual analysis, it is evident that DMFourL-LIE surpasses other methods in terms of brightness enhancement, naturalness, and clarity. This is particularly noticeable when observing the faces in the second and fifth rows. In contrast, both FourLLIE and UHDFour methods exhibit blurriness and noise issues. Therefore, as shown in Tab. 7, we also compared image quality metrics (NIQE [28] and UICM [30] ) on the Dark Face dataset to further support the advantages of our method. 
conclusion: 
 In this paper, we introduce the Dual-Stage Multi-Branch Fourier Low-Light Image Enhancement (DMFourLLIE) framework, a novel structure designed to enhance the expressiveness of Fourier frequency domain information and improve the spatial and textural details of images. The first stage of our framework utilizes a distinctive multi-branch structure that integrates infrared and brightness priors, thereby refining the expressiveness and accuracy of frequency domain information. This integration significantly bolsters the phase component's capacity to preserve image structure while enabling precise amplification of the amplitude component, tailored to regional luminance variations. In the second stage, DMFourLLIE employs a multi-scale spatial perception module in conjunction with the innovative application of fast Fourier convolution. This dual-branch approach markedly enhances the representation of spatial structures and the delineation of subtle texture details in the enhanced images, leading to superior image quality. 
: 
 and FourLLIE [37] , which exhibit color distortion and noise. Our method, in contrast, enhances brightness and preserves color fidelity, eliminating noise. (b) shows the superior noise suppression and color accuracy of our approach, particularly in highlighted areas.Due to environmental constraints and hardware limitations, images often suffer from issues like diminished visibility, uneven exposure, loss of detail, and color inaccuracies. These challenges significantly hinder the performance of computer vision applications, affecting critical tasks such as autonomous driving [35] , pedestrian recognition [56] , and object detection [13] . The primary aim of low-light image enhancement (LLIE) techniques is to restore images to a state of normal lighting, preserving as much texture detail as possible, thereby enhancing the efficiency of subsequent vision processing tasks. With the advancement of neural networks, traditional non-learning methods are becoming less prevalent. Recent years have seen the emergence of innovative LLIE methods [9, 25, 27, 39, 53, 60, 61] in computer vision, yet a balance between method performance and computational demand remains elusive. High-performing models [44, 52] often come with substantial computational costs, whereas Fourier-based strategies [14, 37] present a cost-effective solution by leveraging Fourier frequency information for image brightening.Researches have validated the efficacy of Fourier domain-based low-level image enhancement techniques across various tasks, including enhancement under low-light conditions [14, 37] , denoising [49, 62] , and super-resolution [8] . In LLIE tasks, since luminance information is predominantly encoded in the amplitude component, common strategies [14, 37] involve amplifying the amplitude component and duplicating the phase component. However, the efficacy of such strategies warrants scrutiny. Through rigorous experimentation, we have arrived at the following two critical insights:(1) As demonstrated in Fig. 2 (a), directly replacing the amplitude component of a low-light image with that of a normally illuminated image results in the generated image 𝐼 𝑎 suffering from compromised brightness and texture, thereby introducing noise. Therefore, this indicates significant limitations in previous methods [14, 37] , where they directly approximate the amplitude component between the low-light input and the ground truth image. This approach leads to the degraded image 𝐼 𝑎 being used as the learning target for the network, making the subsequent optimization challenging. We employ difference maps and histogram difference curves to visually demonstrate the disparity between 𝐼 𝑎 and 𝐼 𝑔𝑡 . Furthermore, enhancing the amplitude component may result in overflow of inherently brighter regions in the original image, introducing distorted patches in the visual output (highlighted by the red boxes in 𝐼 𝑎 ). As consistently demonstrated in our results shown in Fig. 1 , although FECNet [14] and FourLLIE [37] attempt to mitigate degradation by restoring the phase component and spatial information in the second stage, merely recovering these aspects does not fully address the distortions caused by the first stage.(2) As depicted in Fig. 2 (b), the distorted image 𝐼 𝑏 demonstrates that the phase component also contains a significant amount of image information, which likewise requires meticulous handling during the process of frequency domain information transfer. The phase component retains more detailed information of the image, sometimes even more critical than the amplitude component. Furthermore, swapping the phase components between two images while keeping their amplitude components unchanged results in the content of the images being exchanged along with the phase information, underscoring the significance of phase information for the structure and content of images.Building upon the insights garnered, we propose a novel framework, DMFourLLIE (Dual-Stage Multi-Branch Fourier Low-Light Image Enhancement), which employs a two-stage design rooted in Fourier-based methods [14, 20, 37] , as illustrated in Fig. 3 . The inaugural stage, detailed in Fig. 4 , is dedicated to Fourier reconstruction. Within the Fourier frequency domain, our aim is to concurrently augment the expressiveness of both amplitude and phase components. To bolster the structural integrity of the phase component in low-light images, we integrate infrared prior information. For the amplitude component, we employ a luminance attention map to transition the luminance space, thereby enhancing the accuracy of amplification across various brightness levels. Progressing to the second stage, depicted in Fig. 5 , our focus shifts towards the reconstruction of spatial structures and textures. Here, we introduce a dual-pathway architecture that merges multi-scale spatial convolution branches with branches based on Fourier convolution [4] . This configuration is strategically designed to refine spatial structure representation while capturing intricate texture details.Overall, the main contributions of this paper can be summarized as follows: 18, 31] and Retinex-based [6, 21, 43] methods. These approaches, however, fall short in accurately capturing scene dynamics and perceptual cues, leading to suboptimal brightness and color fidelity in real-world scenarios, along with exacerbated noise in extremely low-light conditions. The advent of deep learning catalyzed the development of a plethora of learning-based LLIE strategies [9, 25, 27, 53, 60] . Notably, [7] introduced an unsupervised model that derives adaptive priors from pairs of low-light images.Similarly, [42] unveiled a semantic-aware framework that imbues models with a rich tapestry of semantic knowledge. Furthermore, studies [14, 37] have validated the efficacy of Fourier space manipulation for enhancing illumination in low-light scenes, achieving nuanced improvements with minimal model complexity.Infrared and Visible Image Fusion. In computer vision, combining infrared and visible light imagery [3, 58] enhances visual representation in various tasks. However, the lack of annotated visible-infrared image pairs hinders the development of infraredbased deep learning models. This challenge has led to methods [16, 29] for training models in visible-to-infrared image translation. [15, 19] introduced a translation model focused on edge preservation to maintain spatial textures. Unlike traditional shared encoder designs, [59] explored a novel approach for feature extraction and fusion. Inspired by the unique properties of infrared imagery and previous research, we investigate the application of infrared in the Fourier domain for illumination enhancement.Fourier-based LLIE Methods. The exploration of Fourier frequency information for LLIE is still in its early stages. [8] investigated Fourier loss for super-resolution enhancement, while [62] used Fourier transform for image fusion to combine spatial-frequency signals. [49] tackled dehazing by exploring both frequency and spatial domains. Additionally, [14] highlighted the role of amplitude in encoding brightness and phase in capturing structural details, proposing a network for exposure correction. FourLLIE [37] showed that global feature extraction via Fourier analysis can be achieved without increasing model parameters. UHDFour [20] noted the similarity of amplitude components across image resolutions and proposed an illumination technique for ultra-high-definition content. Despite these advancements, the isolated manipulation of amplitude and phase components has limitations, highlighting the need for further exploration of Fourier frequency information. 
 
 Appendix:  


Decision: Luminance and structure features are separately encoded in Fourier frequency domain. This paper argued that existing works did not well exploit those structure features of phase component and then propose a Dual-Stage Multi-Branch Fourier Low-Light Image Enhancement (DMFourLLIE) framework. Experimental results show its effectiveness and superiority.  

Pros: 1. The method proposed is novel and technically sound. 2. The authors provide sufficient experiment (including the rebuttal) to support the performance of their method. 3. The paper writing is clear and easy to read.

Cons: The authors still suggest to revise their work including their feedback during rebuttal (e.g. The impact of the pre-trained model for infrared information extraction may require further exploration). 

All reviewers agree the novelty and effectiveness of this work. After rebuttal, all reviewers’ concerns have been well addressed. Therefore, I will recommend to accept this work.

Reviews: review: paper_summary: paper_summary: In the Fourier frequency domain, luminance information is primarily encoded in the amplitude component, while spatial structure information is in the phase component. Previous low-light image enhancement methods have focused on amplifying the amplitude and replicating the phase. However, these approaches are questioned for their effectiveness. This paper reveals the crucial role of the phase component in preserving image structure and detail and proposes the Dual-Stage Multi-Branch Fourier Low-Light Image Enhancement (DMFourLLIE) framework to enhance the expressiveness of Fourier frequency information. The first stage integrates structural information from infrared images to enhance the phase component and uses a luminance-attention mechanism to control amplitude enhancement in different brightness regions. The second stage combines multi-scale and Fourier convolutional branches for robust image reconstruction. Extensive experiments show that DMFourLLIE outperforms current state-of-the-art methods.
strength_weakness: strength_weakness: This article demonstrates a somewhat degree of novelty. The theoretical description and network framework is well-articulated, the experimental results are robust, the format is standardized, and the logical is coherent.
limitations: limitations: 1. The abstract is lengthy and overly describes the existing problems, while the description of the proposed method is a bit simple, resulting in insufficient logical details.
2. The details of Figures 1-2 are already described in the text, while their captions also accordingly describe a large amount of details, which can lead to duplication.
3. As shown in Figure 1, the result images generated by the first and second stages of the proposed method appear subjectively consistent, suggesting that the second stage might be unnecessary. To demonstrate the necessity of the second stage, it is essential to conduct both subjective and objective ablation experiments. These experiments should treat the first and second stages as a unified process rather than discussing their branches separately, and provide evidence illustrating the importance of the second stage.
4. I wonder about the importance of Section 2.2 in this article.
5. This article focuses on describing the Fourier transform theory. However, in the third Section, the authors do not describe enough how to apply this theory to the enhancement task.
6. The format of labeling maximum values in all tables should be consistent.
paper_summary: paper_summary: In the manuscript, the authors propose a dual-Stage multi-Branch Fourier low-light image enhancement (DMFourLLIE) framework, which can enhance the expressiveness of Fourier frequency domain information and improve the spatial and textural details of images. Some experimental results verified the performance of the proposed method.
strength_weakness: strength_weakness: 1.The paper appears technically sound in terms of the proposed method, model, and experimental evaluations. 
2. For almost each component of the proposed method, there is ablation study correspondingly.
3.The object detection experiments under low-light conditions prove the effectiveness of the results on down-streaming tasks.
4.The paper is well organized and the figures are concise.
limitations: limitations: 1.I suggest to discuss some parameters in the experiment part, such as the loss weights.
2.If possible, the authors should share their code to encourage reproducibility.
paper_summary: paper_summary: The authors propose an LLIE framework based on Fourier theory, addressing the limitations of previous Fourier-based methods and introducing solutions termed DMFourLLIE. They augment DMFourLLIE with infrared and luminance information to aid in low-light image recovery in Fourier space. Additionally, they propose a dual-branches network, comprising multi-scale spatial and Fourier branches, for further refinement.

Overall, this paper presents a clear Fourier-based LLIE method with solid performance. Although there may be a lack of detailed analysis regarding the core theory of Fourier information in low-level tasks, such as the intricate correlation between phase components and spatial images, this is a common limitation in the field that the authors have attempted to address. A borderline acceptance is recommended. If the authors provide a more detailed analysis, the rating could be elevated.
strength_weakness: strength_weakness: 1. The paper is well-organized and easy to understand.
2. Experimental settings are rational and comprehensive. Furthermore, the authors demonstrate the effectiveness of DMFourLLIE in high-level tasks.
3. Detailed analyses of the motivations behind each module and their implementations are provided.
limitations: limitations: The impact of the pre-trained model for infrared information extraction may require further exploration. This includes investigating the use of different models, determining the proportion of this model to the total network size, and addressing scenarios with extremely low-light conditions where extracting infrared information might be challenging. Additionally, the potential integration of other information such as depth maps could be explored.
paper_summary: paper_summary: In this paper, the authors propose a framework named DMFourLLIE that incorporates Fourier frequency domain information and spatial texture for low-light image enhancement (LLIE). The framework contains two stages. In the first stage, DMFourLLIE utilizes a multi-branch structure that integrates infrared and brightness priors for refining the image in the frequency domain. In the second stage, DMFourLLIE employs multi-scale convolutions and Fourier convolutions to restore the image details. With the help of infrared and luminance priors within the Fourier domain, the proposed method achieves promising performance with a low amount of parameters and computational cost. This method has certain innovations but lacks a clear interpretation of the framework design and the effectiveness of the framework component.
strength_weakness: strength_weakness: 1. Extensive experiments conducted on widely used LLIE datasets show that the proposed method achieves superior performance and has advantages in terms of model parameters and computational efficiency. In addition, the images enhanced by the proposed method perform better in object detection tasks.
2. DMFourLLIE innovatively proposes to incorporate infrared and luminance priors in the Fourier space and effectively improves the image enhancement ability in the frequency domain.
limitations: limitations: 1. The effectiveness of infrared prior information lacks sufficient analysis. The authors emphasize the purpose of using infrared priors is to “bolster the structural integrity of the phase component”. But in terms of extracting structural information, what is the advantage of the RGB-to-TIR model compared to the traditional edge detector such as the Canny edge detector used in [1]? In addition, as shown in Figure 8, there is no obvious structural difference between the images enhanced by the whole framework and the framework w/o Infrared Branch. How to explain the result?

2. The motivation for employing the Spatial and Texture Reconstruction Stage is not convincing. The authors mention that the Fourier Reconstruction Stage “may introduce additional noise and feature loss” so the Spatial and Texture Reconstruction Stage is needed in Section 3.3, and the comparisons are shown in Figure 6. However, there is no discussion about the degradation introduced by the Fourier Reconstruction Stage in Figure 6. The rationality of network design would be enhanced by adding relevant explanations.

3. The design of the Fourier Reconstruction Stage needs further explanation. As each component can output the enhanced image, why six identical components are needed in this stage? How does the number of components impact the enhancement results?

4. Some descriptions of the experimental settings are not clear. For example, in Table 3 there are settings named “w/o Luminance Branch” and “w/o Luminance Augment”. However, these two modules are tightly connected in sequential order. How can the network work if only one of these modules is excluded? The same unclear expression appears in the settings “w/o Infrared Branch” and “w/o Infrared Augment”.

5. Some figures need to be modified. For example, in Figure 6, there are some confusing coordinate axes around the heatmaps. 

[1] Xu, Xiaogang, et al. Low-light image enhancement via structure modeling and guidance. CVPR, 2023.