Exploring in Extremely Dark: Low-Light Video Enhancement with Real Events

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 17:39:11

Abstract: Due to the limitations of sensor, traditional cameras struggle to capture details within extremely dark areas of videos. The absence of such details can significantly impact the effectiveness of low-light video enhancement. In contrast, event cameras offer a visual representation with higher dynamic range, facilitating the capture of motion information even in exceptionally dark conditions. Motivated by this advantage, we propose the Real-Event Embedded Network for low-light video enhancement. To better utilize events for enhancing extremely dark regions, we propose an Event-Image Fusion module, which can identify these dark regions and enhance them significantly. To ensure temporal stability of the video and restore details within extremely dark areas, we design unsupervised temporal consistency loss and detail contrast loss. Alongside the supervised loss, these loss functions collectively contribute to the semi-supervised training of the network on unpaired real data. Experimental results on synthetic and real data demonstrate the superiority of the proposed method compared to the state-of-the-art methods. Our codes will be publicly available.

Main paper: 
 introduction: 
 Low-light video enhancement is a critical task in the field of computer vision, which aims to enhance the visibility and visual quality of videos captured under low-light conditions. Due to the constraints of low-light environments and the limitations of camera sensor, these videos often suffer from noticeable noise, insufficient information and blurriness, leading to the loss of details and a decrease in the overall quality. Despite significant progress in this field, low-light video enhancement remains a formidable challenge.Many deep learning methods [1, 4, 11, 21, 34, 43] have been proposed to enhance low-light videos. However, most of these approaches rely on mining information directly from a video to enhance it. Due to the limitations of cameras, details in extremely dark areas of the videos are inevitably lost and remain missing even after the enhancement.Event camera is an emerging sensor characterized by its large dynamic range and rapid response. Events recorded alongside lowlight videos, even in extremely dark areas, can fully capture motion information, which is lacking in traditional cameras. This feature makes event cameras highly suitable for low-light video enhancement.However, employing event cameras for low-light video enhancement faces several distinct challenges: 1) Fusion complexities: The information in the extremely dark regions of the video is limited, making it challenging to fuse event data with these areas and achieve effective enhancement. 2) Data acquisition difficulties: Constructing a dataset consisting of paired real events, normal-light videos and low-light videos is difficult. Most existing video-event paired datasets [6, 14] only consist one type of videos while also suffering from alignment issues. While event simulators [10, 27] can be used to generate synthetic events, the inherent disparities between synthetic and genuine data can compromise the inferential performance on real data.In this paper, we propose the Real-Event Embedded Network (REN), a low-light video enhancement model that integrates event data based on the Retinex [13] theory. We tackle the challenges by the following strategies: To effectively fuse misaligned real events and images, we present Event-Image Fusion (EIF) module. It can adaptively identify extremely dark regions and utilize event information to enhance them. Furthermore, EIF module establishes spatial alignment between misaligned image and event features. To address the challenge of lacking genuine paired data, we propose unsupervised temporal consistency loss and detail contrast loss, which offer supervision at both temporal and spatial levels. They ensure the temporal consistency of input and output sequence, and spatially supervise detail reconstruction in extremely dark regions respectively. Together with the supervised loss, these loss functions collectively contribute to the semi-supervised training of the REN. The overall network is trained end-to-end with partially labeled datasets, which consist of paired low-light and normal-light videos with synthetic events, as well as solely low-light videos alongside real events.In summary, the contributions of our work are as follows:• We propose the Real-Event Embedded Network to leverage real events for addressing low-light enhancement challenges.To the best of our knowledge, it is the first exploration of real events in this field. • We propose the Event-Image Fusion module, which can identify extremely dark regions and utilize event to enhance them. • We design unsupervised temporal consistency loss and detail contrast loss, aiming to maintain temporal stability of videos and restore details in extremely dark regions.• Extensive experiments on both real and synthetic datasets demonstrate that our approach outperforms state-of-the-art methods in low-light video and image enhancement. 
related work: 
 Low-Light Video Enhancement This line of work primarily includes direct enhancement and Retinex-based enhancement. In the former, Liu et al. [18] employ neural networks to simulate event information from videos and fuse them. Zhang et al. [40] extract optical flow from individual images to enforce the temporal stability. Chhirolya et al. [2] propose a self-cross dilated attention module to learn inter-frame relationships in static videos. SGLLIE [43] extracts enhancement factors and recursively enhances videos. MBLLEN [21] and SMOID [11] employ 3D convolutions to harness the temporal information within videos. As for the Retinex-based enhancement, LAN [4] and SDSDNet [34] extract illumination from videos and enhance them, with LAN designing an adaptive brightness enhancement method capable of achieving one-to-multiple enhancement. In contrast to previous Retinex-based approaches, our focus lies on the reflectance, using event data to complement missing information in extremely dark regions.Event-Based Video Reconstruction Event cameras have found widespread applications in the field of image and video reconstruction, including super-resolution [8, 20] , high dynamic range [25, 39] , deblurring [30, 36] and video frame interpolation [12, 31] . Considering the distinct modalities of image and event features, these works primarily focus on the fusion. In addition to simply summing or concatenating events and images [26, 33] , HDRev [39] propose a multimodal representation alignment strategy to learn a shared latent space and a fusion module tailored to complement two types of signals in different regions. EvIntSR [8] constructs latent frames separately from events and images, followed by their fusion. EFNet [29] propose an Event-Image Cross-modal Attention fusion module, which allows attending to the event features via a channel-level attention mechanism. Very recently, Liang et al. [16] propose a low-light video enhancement method with hybrid inputs of events and frames, using synthetic events to train their network. The differences between these methods and our work are: 1) We employ real event for low-light video enhancement; 2) Our work focuses on enhancing extremely dark regions using event data, while other approaches treat all regions uniformly; 3) Our approach can deal with the misalignment image and event features through spatial-level attention. 
proposed method 3.1 formulation: 
 Event Representation An event data 𝑒 = (𝑝, 𝑡, 𝜎) is triggered at time 𝑡 and at pixel 𝑝 = (𝑥, 𝑦) when the log intensity change Δ𝐿 between 𝑡 and 𝑡 -𝛿𝑡 exceeds the dispatched threshold 𝜃 , 𝜎 ∈ {+1, -1} is the polarity that indicates the increase or decrease of intensity changes.EQUATIONwhere 𝑅 (𝑝 ) 𝑡 denotes the instantaneous intensity at time 𝑡 and pixel 𝑝.As event streams are sparse points and can not input into network, we represent the events as 3D voxel grid [44] . By discretizing duration Δ𝑡 = 𝑡 𝐾 -1 -𝑡 0 spanned by 𝐾 events into 𝐶 temporal bins, each event 𝑒 𝑘 = (𝑝 𝑘 , 𝑡 𝑘 , 𝜎 𝑘 ) distributes its polarity 𝜎 𝑘 to the two closest voxels:EQUATIONwhere 𝑡 𝑘 = 𝐶 -1 Δ𝑡 (𝑡 𝑘 -𝑡 0 ) is the normalized timestamp. Problem Statement Classic Retinex-based model assumes that image can be decomposed into reflectance and illumination, similar to [3] , we employ a regularization-free method to represent the image 𝐼 : where 𝜙 𝑅 , 𝜙 𝐿 denote the modules used to estimate reflectance and illumination respectively, and 𝜙 𝑠 denotes the module used to synthesize the reflectance and illumination. Previous methods primarily focus on accurately estimating these two components and their fusion, often directly assuming the estimated reflectance is same as the ground truth [3, 4] . However, in extremely dark regions, the reflectance tends to lack fine details, owing to the limitations of conventional camera sensors. Different from prior work, our approach enhances the reflectance using event data, the enhanced image 𝑌 𝑡 can be defined as:EQUATIONEQUATIONand the objective function is as follows:minEQUATION)where 𝐼 𝑛 , 𝐼 𝑙 and 𝐸 denote normal-light image, low-light image and event respectively; 𝜙 𝐸 denotes the reflectance estimated from events and images; and ∥ • ∥ 𝐹 represents the Frobenius norm. ∈ 𝑅 𝐻 ×𝑊 ×𝐶 respectively through a Residual Block [9] . Then we estimate the reflectance 𝜙 𝐸 (𝐼 𝑡 , 𝐸) and illumination 𝜙 𝐿 (𝐼 𝑡 ) from them, and finally synthesize them into normal light frames 𝑌 𝑡 .We use a dual-branch structure to estimate the reflectance 𝜙 𝐸 (𝐼 𝑡 , 𝐸), with a branch dedicated to extract image features and the other to extract the event features respectively. Every branch is structured as a hierarchy with four stages. Each stage of the image branch consists of an image encoder Γ 𝑖 𝐼 and an Event-Image Fusion module (EIF, will be detailed in Sec. 3.3) . Each stage of event branch only consists of an event encoder Γ 𝑖 𝐸 . For the 𝑖-th stage, the image feature 𝑓 will be considered as the reflectance 𝜙 𝐸 (𝐼 𝑡 , 𝐸).𝐹 𝑖 -1 𝐼 ∈ 𝑅 𝐻 2 𝑖 -1 × 𝑊 2 𝑖 -1 ×2 𝑖 -1 𝐶 and event feature 𝐹 𝑖 -1 𝐸 ∈ (a) Low-light Frame 𝐼 (b) Luminance 𝐺 (c) SNRmap 𝑆 (d) Event 𝐸 (e) 𝑀𝑎𝑠𝑘 (f) Our result𝑅 𝐻 2 𝑖 -1 × 𝑊 2 𝑖 -1 ×2 𝑖 -1 𝐶 will first be processed as 𝐹 𝑖 𝐼 ∈ 𝑅 𝐻 2 𝑖 × 𝑊 2 𝑖 ×2 𝑖 𝐶 and 𝐹 𝑖 𝐸 ∈ 𝑅 𝐻 2 𝑖 × 𝑊 2 𝑖 ×2 𝑖 𝐶 throughConsidering that the illumination requires more global features, we first encode 𝐹 (0) 𝐼 into the latent space, followed by a U-Net [28] to estimate the Illumination 𝜙 𝐿 (𝐼 𝑡 ).The estimated illumination 𝜙 𝐿 (𝐼 𝑡 ) and reflectance 𝜙 𝐸 (𝐼 𝑡 , 𝐸) are first aligned by convolutional layers, after which the aligned features are fed into the decoder 𝜙 𝑠 to synthesize them. In addition, we also employ residual connections between image encoders and the decoder to improve the performance. In the end, the output of the decoder is processed through a convolutional layer to obtain the enhanced frame 𝑌 𝑡 . The fusion of image features and event features is the key to eventbased low-light enhancement, requiring the matching of relevant features from both the image and the event. Due to the limited information in the ultra-dark regions and interference from other areas, matching is very challenging. Generate Dark Region Mask We design a patch-based extremely dark region mask to identify these regions, so the constraints imposed by the two conditions mentioned above can be relaxed when matching with events. As the extremely dark regions are included in the patch, information within the patch can be used for matching with event features, reducing the difficulty of matching, meanwhile avoiding interference from information outside the patch.Due to the extremely low brightness and Signal-to-Noise Ratio (SNR) in the ultra-dark regions, we use SNR distribution and luminance distribution to generate the mask. We calculate the SNR map using the method similar to [37] , and regard the grayscale image as luminance map. Specifically, given the frames 𝐼 𝑡 +𝑗,𝑗 ∈ [-𝑘,𝑘 ] , we calculate the luminance map 𝐺 and SNR map 𝑆 of 𝐼 𝑡 , and then count their distribution.EQUATIONwhere 𝐺𝑟𝑎𝑦 denotes the process of obtaining a grayscale image, 𝑑𝑒𝑛𝑜𝑖𝑠𝑒 represents a non-learning-based denoising operation. For the 𝑖-th stage, 𝐹 𝑖 𝐼 ∈ 𝑅 ℎ×𝑤×𝑐 , we first resize 𝐺 and 𝑆 to (ℎ, 𝑤), then divide them into non-overlapping patches 𝐺 𝑗,𝑗 ∈ [1,2,...,𝑚] and 𝑆 𝑗,𝑗 ∈ [1,2,...,𝑚] , using a window of size (𝑤𝑠, 𝑤𝑠). Then 𝑚𝑎𝑠𝑘 𝑖 is get through:EQUATIONwhere 𝑚𝑒𝑎𝑛 denotes taking the average value, 𝑚𝑎𝑠𝑘 𝑖 𝑗 denotes the 𝑗-th patch of 𝑚𝑎𝑠𝑘 𝑖 , 𝜃 𝑙 and 𝜃 𝑠 are thresholds based on luminance distribution and SNR distribution respectively. We set 𝜃 𝑙 and 𝜃 𝑠 as the last 10% of luminance distribution and the last 30% of SNR distribution respectively. The extremely dark region feature of images can be defined as 𝐹 𝑖 𝐼𝑑 = 𝐹 𝑖 𝐼 × 𝑚𝑎𝑠𝑘 𝑖 . Difference from previous mask-based method [37] , we integrate both luminance and SNR rather than solely focusing on one, and employ adaptive thresholds, which grant it superior generalization capability. Additionally, our mask operates at the patch level rather than the pixel level, which contributes more effectively to the fusion of images and events.Event-Image Attention Most real datasets suffer from alignment issues between videos and events, we utilize a novel Event-Image Attention fusion to fuse them, and establish spatial mapping relationships to faciliate the alignment. Specifically, given the image feature 𝐹 𝑖 𝐼 ∈ 𝑅 ℎ×𝑤×𝑐 , dark regions feature 𝐹 𝑖 𝐼𝑑 ∈ 𝑅 ℎ×𝑤×𝑐 and event feature 𝐹 𝑖 𝐸 ∈ 𝑅 ℎ×𝑤×𝑐 , we use 1 × 1 conv to get queries 𝑄 𝑖𝑑 from 𝐹 𝑖 𝐼𝑑 , 𝑄 𝑖 from 𝐹 𝑖 𝐼 , keys 𝐾 𝑒 and values 𝑉 𝑒 from 𝐹 𝑖 𝐸 respectively. Then we divide them into patches using a window of size (𝑤𝑠, 𝑤𝑠). The shape of features are changed from (𝑐, ℎ, 𝑤) to (ℎ/𝑤𝑠 ×𝑤/𝑤𝑠, 𝑤𝑠 ×𝑤𝑠 ×𝑐), where each patch represents features from different spatial positions. The aggregated event feature 𝐹 𝑖 𝐸 𝑓 is written as:EQUATION𝑑 𝑘 denotes the dimension of the features. It makes event feature focusing on extremely dark areas. To establish the spatial mapping relationship, we calculate an attention similarity between 𝐹 𝑖 𝐼 and 𝐹 𝑖 𝐸 :EQUATIONwhich serves as a measure of the similarity between every image patch and event patch. For misaligned image and event features, we consider an event patch with the highest similarity to an image patch to be one that matches the image patch. Consequently, we set the maximum value of each row in 𝑊 𝑝 to 1 while the remaining values to 0, and utilize it to establish a spatial mapping between two types of patches via:EQUATIONwhere 𝐹 𝑖 𝐸 𝑓 represents the event features aligned with the image features. Finally, the output of EIF is:EQUATIONAn event stream 𝐸 𝑡 -Δ𝑡 ∼𝑡 +Δ𝑡 records dynamic information between 𝑡 -Δ𝑡 and 𝑡 + Δ𝑡. When Δ𝑡 → 0, it indicates a very small motion and reflects the motion trend. To better keep the temporal stability of output frames, we design the temporal consistency Loss 𝐿 𝑡 , which estimates the motion trend at time 𝑡 from frame 𝐼 𝑡 and compares it with the input. For the synthetic data, the input events are aligned with the frames, and the temporal consistency loss is:EQUATIONFor the real data, the input events and the frames are misaligned, the temporal consistency loss is:EQUATION)where v denotes a U-Net [28] structure used to extract motion trend from frames.The detailed information in extremely dark areas is lost while relatively complete in other areas. To restore complete details in the enhanced frame, we encourage greater discrepancies between input frames and output frames in extremely dark regions, while reducing discrepancies in other areas. The loss function is defined as follows:EQUATION)where 𝑚𝑎𝑠𝑘 identifies the extremely dark region. Furthermore, we also employ the 𝐿 𝑐𝑙𝑖𝑝 in [17] to preserve the semantic integrity of the enhanced results. 15) where Φ 𝑖𝑚𝑎𝑔𝑒 and Φ 𝑡𝑒𝑥𝑡 denote the image encoder and text encoder in CLIP model respectively, 𝑤 is the weight. We set 𝑇 𝑛 = "normal light image" and 𝑇 𝑝 = "low light image".𝐿 𝑐𝑙𝑖𝑝 = 𝑒 𝑐𝑜𝑠 (Φ 𝑖𝑚𝑎𝑔𝑒 ( 𝑌 𝑡 ),Φ 𝑡𝑒𝑥𝑡 (𝑇 𝑛 ) ) 𝑖 ∈ {𝑛,𝑝 } 𝑒 𝑐𝑜𝑠 (Φ 𝑖𝑚𝑎𝑔𝑒 ( 𝑌 𝑡 ),Φ 𝑡𝑒𝑥𝑡 (𝑇 𝑖 ) ) + 𝑤 • ∥ Φ 𝑖𝑚𝑎𝑔𝑒 ( 𝑌 𝑡 ) -Φ 𝑙 𝑖𝑚𝑎𝑔𝑒 (𝐼 𝑡 ) ∥ 2 (We propose to train the REN over a partially labeled dataset, composed of the synthetic dataset 𝐷 𝑠 with groundtruth and the real dataset 𝐷 𝑟 without groundtruth. We employ 𝐿 𝑠 𝑡 , 𝐿 𝑒𝑟𝑟𝑜𝑟 for 𝐷 𝑠 , and 𝐿 𝑟 𝑡 , 𝐿 𝑐𝑜𝑛 , 𝐿 𝑐𝑙𝑖𝑝 for 𝐷 𝑟 . 𝐿 𝑒𝑟𝑟𝑜𝑟 is the reconstruction loss according to Eq. 5:EQUATION)Thus the overall function is as follows:EQUATIONwhere 𝛼, 𝛽, 𝛾 and 𝛿 denote the balancing parameters. 
experiments 4.1 experimental settings: 
 Dataset The proposed Real-Event Embedded Network is trained in a semi-supervised strategy, where one synthetic dataset (SDSD dataset [34] ) is provided for training with the groundtruth and one real dataset (DSEC dataset [6] ) for unsupervised training. In addition, we use the ViViD++ dataset [14, 15] for cross-dataset evaluation. SDSD: Based on the SDSD dataset [34] , we build a syntheic dataset composed of synthetic events as well as real low-light videos and normal-light videos. We employ V2E [10] to simulate events from low-light videos. The detailed configuration of V2E can be found in the supplementary material.DSEC: DSEC dataset [6] consists of real videos and events captured by Prophesee Gen3.1M camera. We select low-light videos along with events from it to train and test our method. We choose 10 training videos with a total of 2948 frames, along with 19 test videos totaling 570 frames.ViViD++: Similar to the DSEC dataset, ViViD++ dataset consists of real videos and events captured by DVXplorer camera. We select low-light videos along with events from it for cross-dataset evaluation. Due to ViViD++ is collected from real-world scenarios, it reflects the effectiveness of the method in real-world settings.Implementation Details Our network is implemented using Pytorch on NVIDIA Geforce RTX 3090. During training, we randomly crop the samples into 320 × 320 patches and augment the data using rotation and horizontal flipping. We optimize the network by AdamW optimizer [19] with the momentum terms of (0.9, 0.999). We set the learning rate to 4e-4 and use the cosine decay strategy to decrease it. The total epoch of training is set to 100 and the balancing parameters 𝛼, 𝛽, 𝛾, 𝛿 and 𝑤 are all set to 1. To validate the effectiveness of exploiting real event, REN is trained respectively over two different datasets: SDSD and SDSD + DSEC, and final networks are respectively denoted as REN 𝑠𝑑 and REN. Specifically, REN 𝑠𝑑 is trained only over syntheic SDSD dataset, while REN is trained over syntheic SDSD dataset and real DSEC dataset.To comprehensively evaluate the effectiveness of our proposed method, we compare REN with SOTA methods using the SDSD dataset and DSEC dataset.Evaluation on SDSD dataset. We train and test the SOTA methods on SDSD dataset for quantitative evaluations, the testset of SDSD dataset includes 12 indoor video pairs and 13 outdoor video pairs.We adopt three evaluation metrics: Peak Singal-to-Noise Ratio (PSNR), Structural Similarity (SSIM) [35] and Learned Perceptual Image Patch Similarity (LPIPS) [42] . PSNR quantifies image quality by comparing signal power to noise. SSIM assesses the image similarity by considering luminance, contrast and structure. LPIPS measures the perceptual image patch similarity using deep learning.The results are shown in Table 1 , the * in table means unsupervised methods. As these methods can be trained on SDSD dataset or SDSD+DSEC datasets, which is similar to ours, we only present the best results of these methods in the table, and the full results are presented in the supplementary materials. As shown in the table, REN 𝑠𝑑 outperforms all supervised methods and REN achieves the best performance. Furthermore, the application of real events and our semi-supervised training strategy is shown to be very effective in improving the performance of the model.Evaluation on DSEC dataset. We also compare different methods on the DSEC real dataset. For supervised methods, due to the lack of groundtruth in the DSEC dataset, it is not feasible to them. Therefore, to compare them with our method, we generate results using models trained on the SDSD dataset, which is consistent with the REN 𝑠𝑑 . For the unsupervised methods (marked with * ), we train them on SDSD+DSEC datasets and compare them with REN.We employ four no-reference evaluation metrics: Natural Image Quality Evaluator (NIQE) [24] , Integrated Local NIQE (ILNIQE) [41] , Perceptual Index (PI) [22] and Perception based Image Quality Evaluator (PIQE) [32] . NIQE assesses the image naturalness based on statistical properties. ILNIQE is an opinion-unaware blind image quality assessment method. PI evaluates the image quality based on human perceptual assessment. PIQE is based on the perceptual characteristics of the human visual system. The evaluation results are shown in Table 2 , under the same training dataset conditions, REN 𝑠𝑑 outperforms all supervised methods, while REN outperforms all unsupervised methods among all the metrics.We also integrate event to two recently best low-light video enhancement methods, SDSDNet [34] and LAN [4] , for a more fair comparison. They are represented as SDSDNet + Event and LAN + Event respectively. Specifically, we incorporate the EIF module into them and train them on SDSD dataset. As shown in Table 3 , both methods improve the performance after integrated with events, and our method achieves the best results.We perform qualitative assessments on the SDSD and DSEC datasets, and cross-dataset evaluations on ViViD++ dataset. Fig. 4 presents the results obtained on the SDSD dataset, similar to Table 1 , we present the best results of unsupervised methods. In comparison to the groundtruth, the CLIP-LIT produces results with weak enhancement and low brightness. The PairLIE generates unrealistic outcomes. NeRCo, SCI and LAN suffer from significant color deviation, which adversely impairs the visual quality of the images. Fig. 5 presents the visual results obtained on the DSEC dataset, which comprises low-quality videos with numerous extremely dark 6 shows the visual results obtained on the ViViD++ dataset. These results are generated using the models presented in Table 2 . The PairLIE enhances the brightness but generates unrealistic color tones. The images enhanced by NeRCo and SDSDNet exhibit noticeable artifacts. CLIP-LIT and SCI fail to enhance the brightness adequately, while the result enhanced by LAN appears relatively blurry. The image enhanced by REN 𝑠𝑑 lacks sufficient clarity and exhibits blurriness in the details. In contrast, REN generates more realistic and clearer result. Fig. 7 shows the visual results of SDSDNet and LAN as well as their integration with event, it also shows the result of our method. Upon the integration with event, both SDSDNet and LAN exhibit noticeable improvements in detail recovery and visual effects, demonstrating the efficacy of event and EIF module. Meanwhile, REN 𝑠𝑑 and REN achieve the most favorable visual results.After a comprehensive evaluation of the comparative results of various methods across the three datasets, our proposed approach demonstrates outstanding visual performance in global brightness and detail restoration. More experimental results can be found in the supplementary material. (e) CLIP-LIT [17] (f) NeRCo [38] (g) REN 𝑠𝑑 (h) REN (f) CLIP-LIT [17] (g) SDSDNet [34] (h) LAN [4] (i) REN 𝑠𝑑 (j) REN The proposed REN improves the performance of low-light video enhancement. To find out what contributes to the remarkable effectiveness of our approach, we do an ablation study and show the results in Table 4 and Fig. 8 .The effectiveness of Events. To validate the effectiveness of events, we train the model only using videos (Only Video). The PSNR value of this model decreases sharply by 2.91dB, which indicates the significant impact of events. As shown in Fig. 8 , events significantly contribute to the recovery of details. The effectiveness of real events (REN 𝑠𝑑 , W/o Real Events) is already demonstrated in Table 1 and Table 2 , where REN outperforms REN 𝑠𝑑 on all datasets, and as shown in Fig. 8 , REN achieves superio results on real data compared to REN 𝑠𝑑 (W/o Real Events).The effectiveness of EIF Module. The EIF module consists of two parts, and concurrently establishes the mapping relationship between images and events, so we build three modified models for ablation study. [34] and LAN [4] as well as their integration with event. Table 5 : The results of the user study. "REN" means that our result is preferred, "Other" means some other approach is preferred, "Same" means that the users have no preference.Other Same REN MBLLEN [21] 15% 5% 80% SGLLIE [43] 5% 0% 95% SMID [1] 0% 0% 100% SDSDNet [34] 5% 5% 90% LAN [4] 15% 25% 60%events. The PSNR value of these models drop 1.88dB, 2.74dB and 1.4dB respectively. As shown in Fig. 8 , the models W/o DRM and W/o EIA demonstrate unsatisfactory performance in the recovery of details within extremely dark regions. Due to the inability to align with real events and images, the model W/o 𝑊 𝑝 fails to fully leverage event information and recover details on real data. The result shows the effectiveness of our EIF module. More detailed discussions regarding the network architecture can be found in the supplementary material. The effectiveness of temporal consistency loss and detail contrast loss. To validate the effectiveness of the proposed 𝐿 𝑡 and 𝐿 𝑐𝑜𝑛 , we build three modified strategies: 1) W/o 𝐿 𝑡 and 𝐿 𝑐𝑜𝑛 , which means training the model only using 𝐿 𝑐𝑙𝑖𝑝 and 𝐿 𝑒𝑟𝑟𝑜𝑟 . 2) W/o 𝐿 𝑡 , which means training the model without 𝐿 𝑡 . 3) W/o 𝐿 𝑐𝑜𝑛 , which means training the model without 𝐿 𝑐𝑜𝑛 . The PSNR value of these models drop 0.42dB, 0.23dB, 0.12dB respectively, which shows the effectiveness of our proposed 𝐿 𝑡 and 𝐿 𝑐𝑜𝑛 .We conduct user studies with 20 participants to compare the subjective visual quality of REN and other low-light video enhancement methods, including MBLLEN [21] , SGLLIE [43] , SMID [1] , SDSD-Net [34] , LAN [4] . We randomly select 10 enhanced videos from the enhancement results of ViViD++ [14, 15] dataset to compare the performance of different models.Each participant underwent five sets of test. In each set, participants are required to randomly select one method from the five options excluding REN and then choose a video randomly. They are instructed to compare the enhancement results of the selected method with REN on that particular video, and then they should decide which is better. The quantitative results of the user studies are shown in Table 5 , indicating that our method is more appealing to users compared to other approaches. 
conclusion: 
 In this work, to address the issue of information loss in extremely dark regions on low-light video enhancement, we propose the Real-Event Embedded Network, which leverages events to enhance such regions in low-light videos. We also design unsupervised temporal consistency loss and detail contrast loss, aiming to maintain temporal stability of videos and restore details in extremely dark regions respectively. These two loss functions, along with the supervised loss, are jointly employed in a semi-supervised manner to train the network on real data. Extensive experiments demonstrate that our method achieves SOTA performance. 
 
 Appendix:  


Decision: Most reviewers find the presented method has technical strengths, with the proposed event-image fusion attention module. The experimental results demonstrate convincing improvement over prior work, with comprehensive results on different datasets and ablation studies. The authors are expected to supplement more experimental results to further address some questions. The AC recommends acceptance to this paper.

Reviews: review: paper_summary: paper_summary: The paper proposes a novel network to use event signals guide low-light video enhancement.
Details contained in event signals are extracted by segmenting extreme dark areas and integrated into contrast enhanced RGB images.
Multiple loss functions are adopted to facilitate more robust results.
Comprehensive experiments are carried out to prove advantages brought about by the proposed method.
strength_weakness: strength_weakness: - Thorough theoretical analysis on image formation and pipeline design. Both dark region mask estimation and event-image attention module have alignments with intuition as well as detailed reasoning to support design choices. 
- Comprehensive experiment results presented. Including both synthetic and real-world datasets, detailed ablation studies, and user study. All showing convincing performance improvements compared to existing methods.
limitations: limitations: There is one minor concern regarding the generalization ability of the proposed method. With a large number of parameters and obvious performance gain, it is natural to ask about how well the proposed model can generalize to unseen data. A cross-dataset test experiment could be helpful.
paper_summary: paper_summary: This paper presents an event-based low-light video enhancement method. An event-image fusion module is proposed to primarily utilize the given events to enhance the dark regions of RGB images. More deeply, novel event-image attention is designed to establish a relationship between the event features and dark-region image features. Furthermore, since there is no real-world normal-light gt of low-light videos with real-world events, the paper proposes to train the method with a supervised strategy on the synthetic event data and an unsupervised strategy on the real event data. An unsupervised detail contrast loss is proposed to primarily enhance the dark regions of input images. The proposed method achieves state-of-the-art performance on the synthetic test set and the real-world test set.
strength_weakness: strength_weakness: + This paper presents a novel event-image attention for enhancing dark-region features. Additionally, an unsupervised detail contrast loss is proposed to improve the dark-region low-light enhancement;
+ This paper is well-written, well-organized and easy to comprehend.
+ The proposed method achieves state-of-the-art performance both on quantiative and qualitative evaluations.
limitations: limitations: - Missing a necessary baseline. Although authors mention the paper ([ICCV2023] Coherent Event Guided Low-Light Video Enhancement) in the "Related Work", they do not compare the proposed method with this competitive baseline;
paper_summary: paper_summary: - This paper proposes an Event-Image Fusion module, which can identify these dark regions and enhance them significantly.
- To ensure temporal stability of the video and restore details within extremely dark areas, this paper design unsupervised temporal consistency loss and detail contrast loss.
strength_weakness: strength_weakness: - To ensure temporal stability of the video and restore details within extremely dark areas, this paper design unsupervised temporal consistency loss and detail contrast loss.
limitations: limitations: 1. Figure 2 has the problem that the data streamlines are too thin and the fonts are messy, which is not easy for readers to read and understand. Please ask the author to optimize the fonts and data streamlines.
2. Basis for the selection of loss function hyperparameters? Did the experiment compare? With all parameters set to 1, what is the point of introducing parameters? In addition, what is the basis for hyperparameter selection?
3. What is the meaning of the second line of formula 5? In this paper, the reflection estimation obtained from image combined with event camera is restricted to only using image to get reflection estimation. What is the specific meaning of adding event camera? Why add it?
4. In Method 3.4, how to use U-Net to extract motion trends in line 450? The real data time is not aligned, how to extract the movement trend, how to maintain the time consistency? Please add the specific formula and explain the specific principle.
5. The contrast method lacks the latest low-light image enhancement method; As shown in the experimental results in Figure 4, the enhanced floor tile details of the proposed method have fuzzy problems, which are obviously inferior to the SCI algorithm. Similarly, the tower crane in Figure 5 has the same problem. We think it is caused by less information on the edge of the event camera. How should the author solve and improve the above problems?
6. Please add a brief introduction to the comparison method and explain why the above method was chosen.
7. In Figure 2, why do the two modules proposed in this paper need to be repeated four times? What is the basis of the experiment? Please supplement the experimental description.
paper_summary: paper_summary: The paper presents a novel approach to enhance low-light video using event cameras, which provide high dynamic range and can capture motion information even in darkness. The authors propose the Real-Event Embedded Network (REN), which integrates real event data to enhance low-light videos. The approach includes an Event-Image Fusion (EIF) module that identifies and enhances dark regions using event information. The REN employs both supervised and unsupervised loss functions to ensure temporal stability and detail restoration, demonstrating superior performance on both synthetic and real datasets compared to existing methods.
strength_weakness: strength_weakness: 1.  The use of real events to enhance low-light video is innovative and effective based on the authors' presentation.

2. The design of unsupervised temporal consistency loss and detail contrast loss ensures the temporal and spatial quality of the enhanced videos.

3. The paper includes thorough experimental validation on both synthetic and real datasets, demonstrating the superiority of their method over state-of-the-art approaches and the applicability of the proposed method in the real-world scenario.
limitations: limitations: 1. I notice the real-event, low-light and ground-truth data used in this paper is paired, which may cause challenges in obtaining more data and the broader applicability and generalization of this method may be undermined.

2. I am not familiar with real-event cameras, but could the proposed method, though trained with real-events, be used in the data captured by traditional cameras? I look forward to the authors' reply to clear my confusions.

3. To better demonstrate the applicability of the proposed method, some downstream vision tasks, like detection and segmentation, can be conducted on the enhanced results.