LuSh-NeRF: Lighting up and Sharpening NeRFs for Low-light Scenes
Authors: Zefan Qu
Ke Xu
Gerhard Petrus Hancke
Rynson W. H. Lau
Venue: NeurIPS 2024 poster (2024)
Keywords: Deep learning
Neural Radiance Field
Low light
Primary area: machine_vision
TL;DR: A novel method that can reconstruct a clear and sharp NeRF from a group of hand-held captured low-light images and a new proposed dataset.

Abstract: Neural Radiance Fields (NeRFs) have shown remarkable performances in producing novel-view images from high-quality scene images. However, hand-held low-light photography challenges NeRFs as the captured images may simultaneously suffer from low visibility, noise, and camera shakes.
While existing NeRF methods may handle either low light or motion, directly combining them or incorporating additional image-based enhancement methods does not work as these degradation factors are highly coupled.
We observe that noise in low-light images is always sharp regardless of camera shakes, which implies an implicit order of these degradation factors within the image formation process.
This inspires us to explore such an order to decouple and remove these degradation factors while training the NeRF.
To this end, we propose in this paper a novel model, named LuSh-NeRF, which can reconstruct a clean and sharp NeRF from a group of hand-held low-light images.
The key idea of LuSh-NeRF is to sequentially model noise and blur in the images via multi-view feature consistency and frequency information of NeRF, respectively.
Specifically, LuSh-NeRF includes a novel Scene-Noise Decomposition (SND) module for decoupling the noise from the scene representation and a novel Camera Trajectory Prediction (CTP) module for the estimation of camera motions based on low-frequency scene information.
To facilitate training and evaluations, we construct a new dataset containing both synthetic and real images.
Experiments show that LuSh-NeRF outperforms existing approaches. Our code and dataset can be found here: https://github.com/quzefan/LuSh-NeRF.

LuSh-NeRF: Lighting up and Sharpening NeRFs for Low-light Scenes
Zefan Qu
Ke Xu
Gerhard Petrus Hancke Rynson W.H. Lau
Department of Computer Science City University of Hong Kong aitty..du....
zefanqu2- camy. cityu. edu. hk, kkangwingogmail. com, hancke, Rynson Lau)@cityu. edu. hk
Abstract
Neural Radiance Fields (NeRFs) have shown remarkable performances pro- ducing novel-view images from high-quality scene images. However, hand-held low-light photography challenges NeRFs as the captured images may simultane- ously suffer from low visibility, noise, and camera shakes. While existing NeRF methods may handle either low light or motion, directly combining them or in- corporating additional image-based enhancement methods does not work as these degradation factors are highly coupled. We observe that noise in low-light images is always sharp regardless of camera shakes, which implies an implicit order of these degradation factors within the image formation process, This inspires us to explore such an order decouple and remove these degradation factors while training the NeRF. To this end, we propose in this paper novel model, named LuSh-Neba which can reconstruct a clean and sharp NeRF from a group of hand-held low-light images. The key idea of LuSh-NeRF is to sequentially model noise and blur in the images via multi-view feature consistency and frequency information of NeRF, re- spectively. Specifically, LuSh-NeRF includes novel Scene-Noist Decomposition (SND) module for decoupling the noise from scene representation and novel Camera Trajectory Prediction (CTP) module for the estimation of camera motions based on low-frequency scene information. To facilitate training and evaluations, construct new dataset containing both synthetic and real images. Experiments show that LuSh-NeRF outperforms existing approaches. Our code and dataset can be found here: https://gthubb.coqquufaaa/////// NeRF.
Introduction
Neural Radiance Fields (NeRFs) [31, 3, have achieved notable success in modeling 3D scene information via implicit functions learned from set of 2D images with known camera poses. Since optimizing NeRFs by measuring the colorimetric errors of training views essentially requires bright and sharp training images, hand-held low-light photography, which is prevalent in our daily life (e.g., in nighttime scenes), cannot be used directly for training NeRFs produce visually pleasing novel view images (Fig. 1(b)), due to the co-existence of low visibility, noise, and camera motion blur in the captured images.
straightforward solution is to incorporate existing low-light enhancemen//ceblriing methods (e.g., [57, 12, 67, 59, 13, 45]) to preprocess the captured images before using them for training the NeRFs. However, it raises two problems. First, as these methods are typically image- they do not consider the multi-view consistency. Second, These enhancement methods may introduce
Joint corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) A Low-light Scene
(b) NeRF [31]
(c) LEDNet [67|+NeRF [31]
(d) Restormer [59]+LL.NeRF [47] (c) PairLIE [13]+DP-NeRF] [19]
(f) Ours
Figure 1: Given hand-held captured low-light scene (a), while (a combination of) existing low-light enhancement/NeRF methods may not produce visually pleasing novel-view images ((b)-(e), our LuSh-NeRF can produce bright and sharp results (f).
additional artifacts (e.g., overexposure and unnatural color contrast). We note that there are some NeRF methods proposed for handling low-light scenes [47, 6] and scene motions [26, 48, 19, 7]. However, while the former assumes no camera motions occur during the capture, the latter cannot handle low-light scenes. Directly applying them our problem does not work. A visual example is shown in Fig. ((c) to (e)), where existing methods struggle to render the desired results.
We observe that in the captured low-light images, noise always appears sharp regardless of the camera shakes, due the independent sensor noise generation within the collection and transformation of photons into electronic signals in the camera Image Signal Processor (ISP). This implies an implicit order of low visibility, sensor noise, and blur, which inspires us to model such an implicit order to decouple and remove those degradation factors for NeRF's training in an unsupervised manner.
In this paper, we propose novel method, called LuSh-NeRF, Light up and Sharpen NeRF by sequentially modeling the degradation factors. Specifically, the brightness of training images is first scaled up to provide more contextual information (which simultaneously amplifies the noise in the images). We then propose two novel modules, i.e., Scene-Noise Decomposition (SND) module and Camera Trajectory Prediction (CTP) module, handle the noise and camera shake problems. The SND module learns to decouple noise from the implicit scene representation by explicitly learning noise field through leveraging the multi-view feature consistency of NeRF. The CTP module then estimates the camera trajectories for sharpening image details based on the low- frequency information of denoised scene images rendered by SND. The two modules are optimized in an iterative manner such that the denoised scene representation of the SND module provides more information for predicting trajectories the CTP module, while the results with sharp details from the CTP module in turn facilitate the denoising process in the SND module. To facilitate model training and evaluations, we construct a new dataset consisting of five synthesized scenes and five real scenes. As shown in Fig. 1(f), our LuSh-NeRF can render bright and sharp novel-view results.
In summary, this work has the following main contributions:
We propose the first method (LuSh-NeRF) reconstruct NeRF from hand-held low- light photographs, by decoupling and removing degradation factors through modeling their implicit orders.
Our LuSh-NeRF contains two novel modules, a novel SND module for noise removal from the implicit scene representation and novel CTP module for handling camera motions.
We construct the first dataset for training and evaluations. Experiments show that LuSh- NeRF outperforms existing methods.

---

2
Related Works
Neural Radiance Field (NeRF). NeRF [31] has fundamentally changed the way of modeling 3D scenes by learning coordinate-bard based implicit neural representations from a set of 2D observations. has gained widespread popularity in computer vision and graphics tasks pertaining to neural rendering, leveraging coordinate-basee implicit neural representations (INR). It has gained significant popularity in computer vision and graphics tasks, with many NeRF variants proposed for, e.g., accelerating the training and rendering of NeRF [14, 18, 40, 41], handling dynamic scenes [21, 34, 39, 44, 61] and digital humans body [1, 34, 36, 37, 10] or human head [53, 68, 16] modeling, and the manipulation [4, 38, 28, 23] or generation [8, 15, 25, 33] of scene contents. Many variants of NeRF are also gradually expanding into multiple research areas. For example. [14, 18, 40, 41] are proposed to accelerate the NeRF training procedure, [21, 34, 39, 44, 61] are applied to render dynamic scenarios, [4, 38, 28, 23] are focused on the NeRF relighting methods, [l, 34, 36, 37, 53] are expanded to the non-rigid object rendering, [8. 15, 25, 33] are used for the generation models. These methods typically require high-quality (i.e., bright and sharp) 2D images for optimizing NeRFs
NeRF from Low-light Images. Recently, there are some methods [47, 30, 6] proposed to relax such constraints by learning to reconstruct NeRFs from low-light images. Wang ef al. [47] reconstructs a normally illuminated scene with multiple low-light images in an unsupervised manner. Cui et al. [6] can change the luminance of the scene by extending the transmittance function in NeRF. However, these methods do not consider the camera shakes that often occur in hand-held low-light imaging.
NeRF from Blurry Images. Rendering scenes with multiple blurry images is frequent and challeng- ing the real world. Ma et al. [26] firstly propose the deblurring problem in NeRF and simulate the blurring process with a deformable kernel. Most works [19, 48, 20, 17] model the camera motion trajectory in 3D space to handle the blur in normal-light scenes. Peng et al. [35] accelerate the deblurring process with an efficient rendering scheme. Huang et al. [17] utilize a blur generator to model the camera imaging process for the all-in-focus photos. However, these methods cannot handle hand-held low-light photographs as they do not jointly model noise and camera motions under low-light conditions.
Low-light Image Enhancement. Deep learning-based methods have been shown to be more effective in this task. Some Retinex-based methods [52, 65, 64, 54] and end-to-end methods [9, 11, 27, 32, 50, 56, 66, 58, 46, 22] are capable of achieving good enhanced results. Dong et [l.[9] abandon the bayer-filter to recover the low light color from raw camera data. Dudhane et [ merge sequential images taken in the same scene to enhance the image quality. Xu et al. [5.] take an SNR prior to guide the feature learning and formulate the network with a new self-attention model. However, there are few methods that can address the blur present in practical low-light images owing to the long exposure time. Recently, Zhou el al.[67] propose a large dataset for the low-light deblurring task, and some works [67, 62] propose solutions to enhance low-light blurred images.
Low-light Deblurring. There are few methods that can address the blur present in practical low-light images owing to the long exposure time. Recently, Zhou et al.[67] propose a large dataset for the low-light deblurring task, and some works [67, 62] propose solutions to enhance low-light blurred images. Zhou er al. [67] propose a highly coupled encoder- decoder architecture to solve the joint artifact. Zhang et al. [62] delves into the multiple degradations and proposes an all-in-one fashion restoration network. While our work addresses similar problem, we aim for NeRF reconstruction, which is more challenging.
3 Proposed Method
In this section, we will introduce the details of the proposed method LuSh-NeRF, which can recon- struct a normally illuminated, sharp, and clean scenario from set of hand-held captured low-light 2D images. Specifically, we first propose the SND module to separate noise from the scene information in the pre-processed image (Sec. 3.2), and then use the CTP module to make an accurate prediction of the kernel from the low-frequency information the image (Sec. 3.3). Our network is supervised by multiple loss functions (Sec. 3.4). The overall structure of the network is shown in Fig. 2.
3

---

Secuorie Nobe Deconpesiton
Rays Algament
Training Image 1?
Propecao
Exestnctioon
Frequency Riter
Rigid
Rgdd
Scenario
Weights
Feepencc Moul
Camero Trajector Prediction
Gradiett Dersch
The pipeline of our proposed LuSh-NeRF. It contains two novel modules: (a) SND module: Decompose the noise in each view from the origin training image with a Noise NeRF architecture. and utilize the multi- viel consistency characteristic in 3D scenario to separate the scene information and noise better; (b) CTP module: minimize the interference of noise in low-light images on blur kernel predictions, the high frequency domain of the low light regions which are severely affected by noise are abandoned. In the rendering stage, we discard the Noise Estimator and Blur Kernel, and only use the Scenario-NeRF to render the enhanced scene.
Figure 2:
3.1 Problem Statement
Preliminary of Neural Radiance Field. NeRF utilizes volume rendering [29] to synthesize 3D scenes by simulating multiple camera rays rtt)
td emitted into the scene, where represents the ray origin, is the sample distance, and d denotes the ray direction. The values at various sample points along these rays are encoded within an implicit neural network. Given 3D coordinate x (xr, y, =) and querying view direction d (0, Φ), NeRF estimates the function F: (x, d) + (c, through an MI.P network, where e and a represent the RGB color value and the volume density, respectively.
After obtaining the information each sample point on a ray, the pixel value ((r) ray r can be computed as:
N C(r) = Σ C.C. Σ T. (1-exp(-ooo.) C, T; = exp 1=1
N
[-1 -
-
(I)
where 5 is the distance between two subsequent sample points ray. T. is the accumulated transmittance value that denotes the radiance decay rate of sampled points. Problem Modeling. For images taken in prac- tical low-light scenarios, degradation can be shown in the ample in Fig. 3: (1) Low value intensity. The pixel value intensities in the image are gener- ally low, which leads to increased difficulty in NeRF training [47], resulting poor results shown in Fig. (b); (2) Noise. Due to the sen- sor noise generated within the collection and transformation of photons, the image pixel val- ues are distorted randomly and unpredictably; and (3) Camera motion blur. Long exposure times inevitably lead to blur in the photo due to the camera motion. When trying to address one of the three defects, the others interfere significantly. Therefore, how to reconstruct a properly illuminated, clear, and sharp NeRF scene from this poor-quality image is very challenging problem.
modeled into 3 categories,
Figure 3: Different degradations in the real low light images. (a) Low intensity (b) Noise (c) Blur.

---

In this work, we attempt to optimize the NeRF by decoupling and removing the above defects in the training stage. Modeling and analyzing noise and blur are difficult at low pixel intensity values. Thus, the low visibility of the image should first be raised. Considering that noise in low-light photo mainly generated by the imaging process and would not be affected by the camera motions, the implicit order of the defect decomposition should be low visibility, noise, and blur.
Specifically, given multiple training images Lu, boisy,bbbb blurry from different views, NeRF aims to render a scenario with Att.clear. We first scale up the brightness of Lil,noisy,bbure Muuryy and get Jhl.noisy,bburry for more contextual information, but this process simultaneously amplifies the noise in the image. LuSh-NeRF then decomposes the image into scene and noise information with 3D multi-view consistency, removing the noise from the image to obtain Int.elear.Muurry. Finally, the image without noise interference is utilized to model the blur phenomenon and restore shheeppar The enhancement process for LuSh-NeRF as follows:
hhlcclear.eharp R(LuS)-NeRF) Sharpen Sharpen(ScaleUp( Lu, moisy, Murry) Noise),
Jat,clear. biurru
where Noise is the noise of the different views predicted by the SND module. Sharpen(-) denotes the CTP module to ease the camera motion blur phenomenon, and R-)) is the NeRF rendering function.
During training, due to the lack of ground truth, J.l.noisg.Murry (i.e., Scale (oaeeeeUUU,,niiyp.p.e nois,,y, is used as the supervised signal. The inverse of Eq. should be taken to simulate the low-quality image as:
h Moiry = Sharpen - (R(LuSh- NeRF)) + Noise,
After training, the correct-light (denoted as hl), sharp and clear NeRF model can be obtained directly by discarding the Sharpen function and Noise. the following subsections, we introduce how we estimate the Noise under each view with the SND module and how we accurately predict the Sharpen function with the CTP module.
3.2 Scenario-Noise Decomposition
Directly training NeRF with Ini,noisy sllnciisy.burry would still produce defective rendering results, as scaling up the low-light images can amplify noise and distort colors significantly, causing more severe multi-view inconsistency.
We note that while the noise is generally sharp and stochastic and has different values in different views, scenario information across views tends to be consistent. This makes it possible to decouple noise and scenario information. To co-optimize with the base network Scenario-NeRF (S-NeRF), we propose new network, named Noise-Estimate structure as S-NeRF, compute the noise of each rendering coordinate. Since the noise is not consistent, N-Estimator discards the volume rendering calculation and directly takes the values of the intermediate sampling points of the input rays as the output. The formula for obtaining the noisy pixel (r) is:
(N-Estimator), which has the same network
Croiey())
+ CN-Eetimator (T) CTP
Σ w.c,) +n
where
ng ML Estimator (Pmid, d),
where Pmid is the intermediate sampling points coordinate, and d the view direction. The C-P(-) function refers to the Camera Trajectory Prediction module, N is the number of sampled points on a ray, and n y is the noise value rendered by N-Estimator. The volume rendering computation of S-NeRF gradually forces S-NeRF to learn the consistency scenario information, and the noise that does not share this feature is decomposed from the scene representation by the N-Estimator during the optimization process to fit the final noisy training images.
To more accurately decouple the scene information and noise, LuSh-NeRF utilizes the Rays Alignment supervision in the SND module. Specifically, An image matching method [43] is applied to the images rendered by S-NeRF after sharpening in each viewpoint to obtain a dense pixel matching matrix M and certainty matrix C, which represent the coordinate pairs of matching pixels in different views and the confidence level of each matching. The current input ray is then taken as
5

---

anchor_roo -ry, and the coordinates of its rendered pixels are used to find the matching pixels under other viewpoints in M. The confidence of the matching pairs in C' should be higher than the preset threshold 0 to ensure that accurate matches can be obtained.
With these matching pixel coordinates, we get the corresponding aligned rays align_ray, under different viewpoints view;, Ideally, the RGB colors computed by anchor_ray and align_ray: should be close identical owing the scenario consistency. So consistency loss Leonsisteney is proposed to reduce the distance within these rendering pixels in S-NeRF to force the decoupling of scene information and noise, as:
K
K
Leonsistency
Σ
Cs Ce-NeRF(ri)
Σ j=1
CC-NNeRF(ry)
(5)
K
K
where K is the number of training views. Constrained by Leemsisteney the information extracted by S-NeRF has a higher viewpoint consistency, leading to a better decomposition of the scene information and noise.
3.3 Camera Trajectory Prediction
Motivated by [19]. the CTP module employs the same thoughts to predict the rigid camera motions with all rays in each view. However, unlike the properly lighted blurry data used in [19], the practical low light conditions present substantial interferences, which could cause notable errors in the CTP network predictions, as demonstrated in the results of Subsec 4.1.
When the SND module is not fully converged, noise still exists in the SND- END-processed scenario. The kernel prediction using rays with strong noise interference may interrupt the CTP network, and directly lead to the failure of the whole NeRF training. Hence, it is necessary to filter the rays for CTP module optimization. Since the nigh-frequency and low-intensity regions in the low-light images more disturbed by noise, corrupting the information in the original raw image [55], the CTP module utilizes Discrete Fourier Transform (DFT) to obtain the image frequency map as:
M N
Yu, r) (FT(2(m,
Σ
x(m,n) e 2-22x
(6)
===
The noisy high-frequency regions of the image are filtered by the low-pass filter H of radius r:
1, if Y Hompuss (u, v) H(u, v) Yuu, v),, where H(u = o, if of
Sr
The retained low-frequency image regions are then converted back to the image via the DFT function to obtain the image informative region mask M Maskrowpss (m, n2) as:
Maskawwuss (m, n) Bi(DFT (Vuwpass (u, w)] T))
where Bi(., 7') is a binarization function with threshold T. . Maskooppss (m, n) classifies the rays into rayalear and raynoisy- which are used to render the clear and noise-dominated pixels in the training images separately. The gradient of rayyoiss to the rigid motion prediction network is detached during NeRF optimization to reduce the impact of noise yet to be removed by the SND module on blur kernel prediction as:
(9) where the Detach(-) function is used to detach the gradient from the variables. The role of the low-pass frequency filter shown in Fig. 7, where the CTP Mask reduces the interference of noisy regions on the blur kernel prediction compared to the mask obtained by directly using the RGB intensity as the threshold.
rays = [rayslear: Detach(rayhooisy)) )}
3.4 Training& & Rendering
After employing the SND and CTP modules as in Eq. 3, we have the simulated RGB values Öhdl,noiey, blurry in the image Int.noisy, We can then optimize the network with the MSE loss Leonsturction as:
Leonstruction
Σ |htt, (r:) - Chi (r:))l2.
(10)

---

Centre
es
Service
then changed to alth Service
ith Service
iith Service
rth Service
light 16 [3I]
[13]-DP-NGRR [19]
[9]+LLNERF [47]
LuSb-NeRF(Oury
Figure 4: Qualitative results of different methods real scenes. Our LuSh-NeRF render
cleaner and sharper results for real low-light scenes with camera motions.
All the modules in the network are trained in an end-to-end manner with the following training losses:
LLuSh-NeRFF
Leonstruction + B Leonsistencys
(11)
where and ,3 are the hyper- paremeerreeters to balance the impact of two losses. When rendering scenario, only the S-NeRF network in LuSh-NeRF is utilized with the volume rendering method to produce images of different views.
4 Experiments
Our Dataset. Since we are the first to reconstruct NeRF with hand-held low-light photographs, we build a new dataset based on the low-light image deblur dataset [67] for training and evaluation. Specifically, our dataset consists of synthetic and 5 real scenes, for the quantitative and generalization evaluations. We use the COLMAP [42] method scenarios. Each scenario contains 20-25 images, at 1 x 640 resolution. The brightness of images each scenario is extremely low, where the intensities of most pixels are below 50. 80% of the images contain camera shake problems, which pose significant challenge for NeRF reconstruction. Refer to Appendix A.3 for more details of our dataset.
estimate the camera pose of each image in the
Implementation Details. We have implemented our LuSh-NeRF based on the official code of Deblur-NeRF [26], and with the same Rigid Blurring Kernel network in [19]. The S-NeRF shares the same structure as NeRF [31], while the the depth and width of N-Estimator are set to half of those of the S-NeRF. The number of camera motions & and the frequency filter radius in the CTP module are set to 4 and 30. The number of aligned rays K and certainty threshold in the SND module are set to and 0.8. Before training, the input images are up-scaled by gamma adjustment and histogram equalization. The batch size is set to 1,024 rays, with 64 fine and coarse sampled points. and B are set to and during the first 60K iterations for better rendering results, to avoid the inaccuracy matching matrix M interfering with the SND module. The two

---

Imput
LEDNNe [67 1ONNNNRFI 1311
Restormer [5RR+ [47]
MPRNet |6ON-L [47]
PairL [13]-DP-NoRFI [9]
U'eeinee [5o]-DP-NoRF [19]
LuSh-NNRR (Ours)
Ground Truch
Tnput
LEDNee [67] NeRF [31]
Restormer 159 (47)
MPRNet LNER [47]
PaizLIE [13 NeDF- 1191
URetinex 154 NDRF 1191
LuSh-NeRE (Ours)
Ground Truth
Input
LEDNet N6RF
Restormer NLL (47)
MPRNet [60]-LLNeRF
PaisLIE D13)-DP-NeRFI [19]
URetinex DS-RRDP-NeRFI [19]
LuSh-NeRI (Ours)
Ground Truch
Qualitative results of different methods on our synthetic scenes. Our method yields the most natural restoration results while sharpening the image.
Figure 5:
1 and 1 10-2 in the last 40K iterations. All the experiments in this paper are performed PC with an 19-139 CPU and a single NVIDIA RTX3090 GPU.
4.1 Main Results
Evaluation Methods and Metrics. Since we are the first to tackle the NeRF reconstruction from low-
light images captured with camera motions, we design several baseline methods by combining existing
state-of-ther of-the--rt image- based and NeRF-based methods: (1) Low-light image enhancement [54, 49] +
Image deblurring method [60, 59] + NeRF [31]; (2) Joint low-light image enhancement and deblurring method [67] + NeRF [3 ] (3) Low-light image enhancement [13, 54] + deblurring NeRF [19]; and
(4) Image deblurring [59, 60] + low-light NeRF enhancement [47]. All the selected models have been shown to achieve SOTA performances their individual tasks. PSNR, SSIM, and LPIPS [63] metrics are used to evaluate the performance difference of the rendered images between LuSh-NeRF and the combination of other otate-of-the-art methods. Due to the significant variations in color contrast ratio and image brightness of the restored low light images by different methods, we consider LPIPS (Learned Perceptual Image Patch Similarity), which can reasonably assess perceptual image quality and visual differences rather than the pixel-level difference, as the key metric.
Visual Comparisons. Fig. 4 presents qualitative comparison of the rendering results from the above three combined methods with those of our LuSh-NeRF in our real low-light scenarios. We can see that the results from LEDNet [67] are still dark, which significantly limits the learning capability of the NcRF, While PairLIE [13] combined with DP-NeRF [19] can reconstruct clear rendering results, they rather unnatural. The color contrast is severely shifted, resulting in unrealistic images with some lost details and incomplete blur removal. In addition, we can also observe that first deblurring on low-light images and then applying the LLE method, in Restormer [59] + LLNeRF [47], is less effective, and there is still substantial blurring in the results. In contrast, our approach can recover detailed scenes with natural colors, resulting in sharp and clean novel view visualization.
Fig. 5 presents qualitative comparison between existing methods and our LuSh-NeRF on our synthetic scenes. All of the above methods suffer serious performance degradation when the image luminance is low and the camera shake severe (e.g., the bottom of the first example in Fig. 5), where LuSh-NeRF can still reconstruct satisfactory scene by decoupling the defects of the scene. As we can see from
8

---

Methods
PSNK
"Dorm"
SSIM
LPIPSL
PSNRT
SSIM
"Psseer"
LPIPS:
PSNR
"Plane"
SSIM
LPIPST
NeRT [31]
0.5159
0.4061
L.Former [49] + MPRNet [60] NoRF [S]]
19.01
0.5558
0.4571
14.67
0.5765
0.2785
18.99
0.5258
0.5290
URetinexNet
Restormer
NeRF [31]
17.47
0.5240
0.4820
12.38
0.3483
0.6347
17.06
0.5205
0.5256
LEDNet [67] NeRF [S1]
18.14
0.6025
0.3530
21.09
0.71440
0,3009
15.21
0,5869
0,1377
Restormer [59] + I.I. N.L.N [47]
17.85
0.5541
0.5000
14.03
0.5462
0,4182
14.94
0.5250
0.52600
MPRNet -LLNRRF
17.27
0.3493
0.4942
0.4744
0,4334
0.5420
PR [13] + DR-NeRF [19]
14.08
0.4574
0.3651
13.34
0,4482
0.2995
13.11
0.5048
04316
URetinexNet [54] DP-NeRF [19]
16.55
0.4884
0.3762
14.12
0.4199
0.5829
16.43
0.5211
0.4472
LuSb-NeRI (Ours)
19.06
0.5354
0.3491
18.12
0.6331
0,2265
19.34
0.5275
0.3852
Method
PSNK
"Sukaru
SSIM
LPIPS
PSNRT
SSIM
"Hall"
LPIPST
PSNR
Averuge
SSIM
LPIPST
NcRE [3T]
0.055!
0.7186
8.19
0.1667
0.5048
0.1683
F.rroreer [49] MPRNet [60] + NeRF [S]]
16.04
0.5204
0.4192
22.02
0.7153
0.3026
18.14
0.5787
0.3973
URetinexNet
Restormer [59] NeRF [31]
16.29
0.5152
0.4347
20.39
0.7045
0.2834
16.72
0.5225
0,4721
LEDNet [67] NeRF [S1]
18.24
0.6127
0.2772
19.33
0.7279
0,2570
18.40
0.6488
0.3204
Restormer [59] + NeRF [47]
15.22
0.5016
0.4199
20.04
0.71226
0.3452
16.42
0.56799
0.4419
MPRNet LLNNRR
14.25
0.3923
0.5213
0.4958
0,4661
Pairt [13] DP-NeRF [19]
13.86
0.5137
0.3157
19.65
0.6055
0.2600
14.81
0.5059
0.3344
URetinexNer [54] DP-NeRF [19]
16.38
0.5286
0.3550
20.21
0.6328
0,2616
16.74
0.5182
0.4046
LuSb-NeRF (Ours)
18.94
0.5884
0.2562
21.09
0.6421
0,2400
19.31
0,5853
0.2914
Table 1: Quantitative comparison of different methods on our synthetic scenes. The best and the
second performances of each scenario are marked in the table.
the given visualizations on realistic and synthesized data, higher PSNR and SSIM metrics of the methods (e.g., LEDNet NeRF) do entirely represent better performances the NeRF results in our proposed task. This demonstrates that the perceptual metric, LPIPS, is better at assessing the merits of the reconstructed scenario. Refer Appendix A.5 for more visual results.
Quantitative Comparison. We conduct a quantitative comparison of our method against various combinations of SOTA approaches data originates from the training and testing sets of LOL-Blur dataset [67], while the weight of LEDNet [67] is also specifically trained on the synthesized dataset of LOL-Blur. This condition does not constitute a fair comparison. Nonetheless, we report the results of using "LEDNet [67] + NeRF [31]" as reference.
synthesized data in Tab. 1. Note that our synthesized
Our method achieves better performances than multiple combinations of existing methods on all synthesized data. In terms of the LPIPS metric all the scenarios, our method outperforms the rendering results of LEDNet [67] + NeRF [31] in an unsupervised optimization manner, which proves that the SND and CTP modules in LuSh-NeRF fully utilize the 3D information of the scene to aid in the recovery the irreversible quality degradation the single image.
Ablation Study. Fig. 6 demonstrates the effect of the various components of LuSh-NeRF on a realistic scenario. After the images are preprocessed to enhance the luminance, the noise in the images is significantly amplified (as shown in the white box). Directly using blur kernel prediction [19] on these images is heavily interfered by noise, which prevents recovery from camera shakes (Fig. 6(b)). In addition, incorrect kernel predictions have a negative impact as they make it difficult to align the multi-view images rendered by S-NeRF and the SND module cannot obtain accurate viewpoint consistency supervision (Fig. 6(c)). When the CTP module is utilized, the blur of the image is alleviated, but the noise problem still serious (Fig. 6(d)). First sharpening the image with CTP
(s) Input (Proprocessed & Original)
(b) NoRF + Blur Kemel
(c) NeRF+ Blur Kemeel SND
(d)
(e) LaSisNeRF (Shum Demoise)
(1) (Denaise Sharp)
Figure 6: Ablation study of LuSh-NeRF on a real scenario.

---

RGB Intensity Mask
SpaleUp Input
CTPMask
RGB Intensity Mask
Sp Input
CTP Mask
Figure 7: The RGB intensity mask and CTP mask comparison. Two masks are generated with the same threshold T. White points represent and black points represent in each mask.
and then removing noise with SND somewhat suppress the noise problem, but the noise may be reconstructed to the various viewpoints and is not completely removed (Fig. 6(e)). LuSh-NeRF can render sharp and clean scenario images in novel views after the training process (Fig. 6(f)).
We show in the Fig. 7 the difference between the two masks obtained by directly performing an RGB intensity threshold and performing frequency filtering before taking threshold. (1) The RGB intensities in many noise-dominant regions are also high (c.g., the sky and the grass), which are harmful to the blur estimation but not excluded in the mask produced by thresholding. (2) The CTP module uses a frequency filter to identify the low-frequency-dominnte and then obtains the desired mask based on the RGB intensity values. The gradients of rays in high-frequency and dark regions (regions more severely affected by noise) are detached during the Blur Kernel optimization process, which ensures better blur modeling.
regions of the image,
Conclusion and Limitation
In this paper, we have proposed the NeRF reconstruction task from low-light images with camera motions, and a novel unsupervised reconstruction network LuSh-NeRF. LuSh-NeRF models the NeRF restoration process based on the characteristics of three kinds of defects in the scene, i.e., low-light, noise, and camera blur, and successfully reconstructs normal-light, clean, and sharp NeRF network from poor-quality low-light images, by exploiting multi-view consistency and frequency-doman information. To facilitate the training and evaluation of the new proposed task, we construct new dataset which contains 5 synthetic and real scenes low light images with hand-held camera motions. Extensive experiments on our proposed dataset demonstrate that our method is capable of achieving satisfactory NeRF reconstruction results, despite the defects in the input images.
The LuSh-NeRF network also has some limitations. One limitation that it needs to optimize two NeRF networks at the same time and render multiple rays for the blur problem when sharpening images during training, which results in slower optimization process, even though this would not interfere with the rendering speed. Besides, noise that is relatively similar across views may be difficult remove due the reliance on viewpoint consistency. (Refer to Appendix A.8 for details.) As future work, would like to address these issues.
10

---

References
[1] ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu. controllable neural 3d portraits. CVPR, 2022.
Rignerf: Fully
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. ICCV, 2021.
[3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
[4] Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold- Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv:2008.03882 03824. 2020.
[5] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view sterco. In ICCV, 2021.
[6] Ziteng Cui, Lin Gu, Xiao Sun. Xianzheng Ma. Yu Qiao, and Tatsuya Harada. Aleth-nerf: Illumination adaptive nerf with concealing field assumption. In AAAI, 2024.
[7] Peng Dai, Yinda Zhang, Xin Yu, Xiaoyang Lyu, and Xiaojuan Qi. Hybrid neural rendering for large-scale scenes with motion blur. In CVPR, 2023.
[8] Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong. Gram: Generative radiance manifolds for 3d-aware image generation. In CVPR, 2022.
Xingho Dong, Wanyan Xu, Zhihui Miao, Lan Ma, Chao Zhang, Jiewen Yang. Zhe Jin, Andrew Beng Jin Teoh, and Jiajun Shen. Abandoning the bayer-filter to see the dark. In CVPR, 2022.
[10] Zheng Dong, Ke Xu, Yaoan Gao, Qilin Sun, Hujun Bao, Weiwei Xu, and Rynson W. H. Lau. Sailor: Synergizing radiance and occupancy fields for live human performance capture. ACM TOG, 2023.
[11] Akshay Dudhane, Syed Waqas Zamir, Salman Khan, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Burst image restoration and enhancement. In CVPR, 2022.
[12] Huiyuan Fu, Wenkai Zheng, Xiangyu Meng. Xin Wang, Chuanming Wang, and Huadong Ma. You not need additional priors or regularizers in retinex-based low-light image enhancement. In CVPR, 2023.
[13] Zhenqi Fu, Yan Yang. Xiaotong Tu, Yue Huang. Xinghao Ding, and Kai- Kuang Ma. Learning a simple low-light image enhancer from paired low-light instances. In CVPR, 2023.
[14] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamic Shotton, and Julien Valentin. Fastnerf: Higl-fidelity neural rendering 200sps. In ICCV, 2021.
[15] Jiatao Gu, Lingjie Liu. Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. arXiv:2110.08985, 2021.
[16] Andrew Hou, Feng Liu, Zhiyuan Ren, Michel Sarkis, Ning Bi, Yiying Tong, and Xiaoming Liu. Infamous- nerf: Improving face modeling using semantically-allee arXiv preprint arXiv:2312.166 161197 2023.
hypernetworks with neural radiance fields.
[17] Xin Huang, Qi Zhang, Ying Feng, Hongdong Li, and Qing Wang. Inverting the imaging process by learning an implicit camera model. CVPR, 2023.
[18] Christoph Lassner and Michael Zollhofer. Pulsar: Efficient sphere-based neural rendering. In CVPR, 2021.
[19] Dogyoon Lee, Minhyeok Lee, Chajin Shin, and Sangyoun Lee. Dp-nerf: Deblurred neural radiance field with physical scene priors. In CVPR, 2023.
[20] Dongwoo Lee, Jeongtaek Oh, Jaesung Rim, Sunghyun Cho, and Kyoung Mu Lee. Exblurf: Efficient radiance fields for extreme motion blurred images. In ICCV, 2023.
[21] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Neweombe, et al. Neural 3d vidco synthesis from multi-view video. In CVPR, 2022.
[22] Yiyu Li, Ke Xu, Gerhard Petrus Hancke, and Rynson W.H. Lau. Color shift estimation corcrceention for image enhancement. In CVPR, 2024.

---

[23] Zhengqin Li. Jia Shi, Sai Bi, Rui Zhu, Kalyan Sunkavalli, Miloš Hašan. Zexiang Xu, Ravi Ramamoorthi. and Manmohan Chandraker. Physically-basse editing of indoor scene lighting from single image. In ECCV, 2022.
[24] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In NeurlPS, 2020.
[25] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun- Yan Zhu, and Bryan Russell, Editing conditional radiance fields. In ICCV, 2021.
[26] Li Ma, Xiaoyu Li, Jing Liao, Qi Zhang. Xuan Wang, Jue Wang, and Pedro V Sander. Deblur-nerf: Neural radiance fields from blurry images. In CVPR, 2022.
[27] Long Ma, Tengyu Ma. Risheng Liu, Xin Fan, and Zhongxuan Luo. Toward fast, flexible, and robust low-light image enhancement. In CVPR, 2022.
[28] Ricardo Martin- Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021.
[29] Nelson Max. Optical models for direct volume rendering. IEEE TVCG, 1995.
[30] Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul Srinivasan, and Jonathan T Barron. Nerf in the dark: High dynamic range view synthesis from noisy raw images. In CVPR, 2022.
[31] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 2021.
[32] Sean Moran, Pierre Marza, Steven McDonagh, Sarah Parisot, and Gregory Slabaugh. Deeplpf: Deep local parametric filters for image enhancement. In CVPR, 2020.
[33] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. In CVPR, 2021.
[34] Keunhong Park, Utkarsh Sinha, Jonathan Barron, Sofien Bouaziz, Dan Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In CVPR, 2021.
[35] Cheng Peng and Rama Chellappa. Pdrf: progressively deblurring radiance field for fast scene reconstruction from blurry images. In AAAI, 2023.
[36] Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodies. In ICCV, 2021.
[37] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. CVPR, 2021.
[38] Julien Philip, Sébastien Morgenthaler, Michaël Gharbi, and George Drettakis. Free-viewpoint indoor neural relighting from multi-view stereo. ACM TOG, 2021.
[39] Albert Pumarola, Enric Corona. Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2021.
[40] Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf: Decomposed radiance fields. In CVPR, 2021.
[41] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mips. In ICCV, 2021.
[42] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. ECCV, 2016.
[43] Xuelun Shen, Wei Yin, Matthia Müller, Zijun Li, Kaixuan Wang. Xiaozhi Chen, Cheng Wang, et al. Gim: Learning generalizable image matcher from internet videos. In /CLR, 2023.
[44] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In ICCV, 2021.
12

---

[45] Di Wang. Long Ma. Risheng Liu, and Xin Fan. Semantic-aware texture-structur feature collaboration for underwater image enhancement. In 2022 International Conference on Robotics and Automation (ICRA). pages 4592-4598. IEEE, 2022.
[46] Haoyuan Wang, Ke Xu, and Rynson WH Lau. Local color distributions prior for image enhancement. In ECCV, 2022.
[47] Haoyuan Wang. Xiaogang Xu, Xu, and Rynson WH Lau. Lighting up nerf via unsupervised decompo- sition and enhancement. In ICCV, 2023.
[48] Peng Wang, Lingzhe Zhao, Ruijie Ma, and Peidong Liu. Bad-nerf: Bundle adjusted deblur neural radiance fields. In CVPR, 2023.
[49] Tao Wang, Kaihao Zhang, Tianrun Shen, Wenhan Luo, Bjorn Stenger, and Tong Lu, Ultra-high wiwhhhhhhhhiiition low-light image enhancement: A benchmark and transformer-based method. In AAAI, 2023.
[50] Yufei Wang, Renjie Wan, Wenhan Yang, Haoliang Li, Lap-Pui Chau, and Alex Kot. Low-light image enhancement with normalizing flow. In AAAI, 2022.
[51] Zirui Wang. Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-; Neural radiance fields without known camera parameters. arXiv preprint urXiv:2102.07064 2021.
[52] Chen Wei, Wenjing Wang. Wenhan Yang, and Jiaying Liu. Deep retinex decomposition for low-light enhancement. arXiv:1808.04560. 2018.
[53] Chung-Yr Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher Shiizerran..z Humannerf: Free-viewpoint rendering of moving people from monocular video. In CVPR, 2022.
[54] Wenhui Wu, Jian Weng, Pingping Zhang, Xu Wang, Wenhan Yang, and Jianmin Jiang. Uretinex-net: Retinex-based deep unfolding network for low-light image enhancement. In CVPR, 2022.
[55] Ke Xu, Xin Yang. Baocai Yin. and Rynson WH Lau. Learning to restore low-light images via decomposiioo--mddeemnaneeeet In CVPR, 2020.
[56] Xiaogang Xu, Ruixing Wang, Chi-Wing Fu, and Jiaya Jia. Sur-aware low-light image enhancement. In CVPR. 2022.
[57] Xiaogang Xu, Ruixing Wang, and Jiangbo Lu. Low-light image enhancement via structure modeling and guidance. CVPR, 2023.
[58] Xin Yang, Ke Xu, Yibing Song, Qiang Zhang, Xiaopeng Wei, and Rynson WH Lau. Image correction via deep reciprocating hdr transformation. In CVPR, 2018.
[59] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2022.
[60] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In CVPR, 2021.
[61] Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Jingyi Yu, and Lan Xu. Editable free-viewpoint vidco using a layered neural representation. ACM TOG, 2021.
[62] Jinghao Zhang, Jie Huang, Mingde Yao, Zizheng Yang, Hu Yu, Man Zhou, and Feng Zhao. Ingredient- oriented multi-degradation learning for image restoration. In CVPR, 2023.
[63] Richard Zhang, Phillip Isola, Alexei Efros, Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
[64] Yonghua Zhang, Xiaojie Guo, Jiayi Ma, Wei Liu, and Jiawan Zhang. Beyond brightening low-light images. JJCV, 2021.
[65] Yonghua Zhang, Jiawan Zhang, and Xiaojic Guo. Kindling the darkness: A practical low-light image enhancer. In ACM multimedia, 2019.
[66] Chuanjun Zheng, Daming Shi, and Wentian Shi. Adaptive unfolding total variation network for low-light image enhancement. In ICCV, 2021.
[67] Shangchen Zhou, Chongyi Li. and Chen Change Loy. Lednet: Joint low-light enhancement and deblurring in the dark. In ECCV. 2022.
[68] Yiyu Zhuang, Hao Zhu, Xusen Sun, and Xun Cao. Mofanerf: Morphable facial neural radiance field. In ECCV, pages 088-285,,20 2022.
13

---

A Appendix
A.1 Broader Impacts
While do not foresee method causing any direct negative societal impact, it may be leveraged to help create fake images using image generation techniques. The human information in the captured images may have the risk of a leak that raises privacy concerns. We urge the readers to limit the usage of this work to legal use cases.
A.2 Details of the RBK module
The Rigid Blurring Kernel (RBK) module proposed in the DP-NeRF [19] utilized in our CTP module. It mimies the scene blur kernel by simulating the 3D camera motions, which contains the following two main parts:
The blurring process of an image is modeled by the Ray Rigid Transfor- mation (RRT). The RRT is formulated as ray transformation derived from the deformation of rigid camera motion, defined as the dense SE(3) field for scene approximated by the MLPs as follows:
Ray Rigid Transformation:
S. - vr) (1.); ((E())))) where vieWiigg
(12)
where I, is the latent code for each view through the embedding layer in [1], R, & are three MLP networks, viewlng the training image set. The S₂ (Fs) v.) E R is the matrix which will be used for the RRT modeling as follows:
RRT rays:g
Trriid_rrasffa formsformrra.... (rs:v,)).
(13)
RRT are the orgin ray and the transformed rays in scene s, q E {1, ...,...) k.. k s where rays and rays:: hyper-parameter that controls the number of camera motions contributing to the blur in each scene 8. The blurry RGB value at rays can be acquired by weighting sums of the NeRF volume rendering values Ce. and Casy from rays and raysg
RRT
Coarse Composition Weights: This part is responsible for computing the weights of cach ray obtained by the RRT part, which is given by the following equation:
Wo o (W(E(L))) where
Σ
Mari 1,
(14)
t=0
where m, is the final weight for each ray in RRT. Finally, blurry color C, for scene can be computed by the weighted sum operation as shown below:
C₂ mo;OCss
Σ
maig Manyaaig
(15)
A.3 Details of the Proposed Dataset
In this work, we collect low-light blurry images from five synthetic and real videos in the LOL-Blur dataset [67] and calculate the camera pose for each image with COLMAP [42] toolbox. For synthetic data, we use ground truth images to estimate camera pose, while for realistic data, the images recovered from LEDNet pre-trained model are utilized to estimate pose since COLMAP hard to model the scenario with the origin low light images. Images with large shifts in each scenario are removed. The number of images for each scene and the training and evaluation viewpoint allocation are shown in Tab. 2.
Syuthesized Duraset
Realistic Dataset
Scenario
Dorm"
"Poster"
"Plane"
"Campus"
"Highway"
"Signhourd"
"Car"
"Neighborhood""
Collected Views
21
21
21
21
20
20
20
21
21
25
Training Views
18
18
21
Evaluation Views
3
Table The dataset split details for our proposed LOL-Blur BlRr dataset.
14

---

All images have a resolution of 1 1120 x 640. The mean value of all pixels in the images is low (less than 50) and camera motion blur is present in 80% of the images in each scene. As illustrated in Fig. 1, Fig. 4, Fig. 5 and Fig. 9, poor image quality in each scene leads to challenging NeRF reconstruction task.
A.4 Quantitative Ablation Studies
We have conducted detailed ablation studies on all the synthetic scenes in the Tab. 3.
PENRT
5S541
12915
15201
50531
L2755
35383
35031
12155
F5531
3510
L205
253RT
55534
121
15000
20101
17P5
1909
3435
20-15
18.00
-148
130
19/00
3330
2685
18:13
3999
21.36
1985
3113
13.38
7766
2834
1183
2013
3015
1920
2365
18.27
20.14
2715
302
3811
227
1558
Prppocco Birr KemzlAsK0
1809
3259
20.39
18.03
A119
150
1838
3191
18.55
2033
-723
2026
3351
2434
23.2
Eath (Shap (cheazio))
18:66
19:11
5383
18.1
2677
3380
2567
.141
A013
2365
3362
74
.214
Table 3: Ablation Study of different modules in our LuSh-NeRF. The best performances of each
scenario are marked in the table.
(I) Lines and 2 show that the ScaleUp preprocessing enhances the NeRF's reconstruction capabili- ties, resulting in an improved image.
(2) Lines and 4 show that CTP leverages frequency domain information to refine Blur Kernel predictions, boosting the perceptual quality of the reconstructed images. However, this process may decrease the PSNR and SSIM scores as it neglects noise interference.
(3) From Lines 2 and 5, the SND module can disentangle noise and scene information from the input noisy-blurry images, leading to substantial improvements on the PSNR and SSIM metrics. However, the SND module is not capable of resolving the blur problem, which leads to minor improvement in the image's 's perceptual quality (more important for the rendered images).
(4) The comparisons between Lines 4, 5, and 8 show that combining SND and CTP modules enhances image perceptual quality and outperforms using only CTP in terms of PSNR and SSIM scores.
(5) Lines 7 and 8 show that decoupling the noise in the scene first, and then modeling the scene blur is more robust restoration order, can effectively reduce the interference of noise in the deblurring process for obtaining better performances.
We have conducted the ablation experiments of the CTP module two synthetic scenes in the Tab. 4.
Dorm SSIM
'Poster" SSIM
Average SSIM
Scene
PSNR
LPIPS
PSNR
LPIPS.
PSNR1
LPIPS_
Threshold
18.99
0,5299
0.3630
18.05
0.6179
0.2598
18.52
0.57.9
0.3114
RGB Threshold(T= = 32)
19.18
0.5308
0.3580
17.86
0.5896
0.245
18.52
0,5602
0.3015
RGB Threshold(T = 48)
18.71
0.4874
0.4455
17.92
0.6057
0.2418
18.32
0.5466
0.3437
RGB Threshold(T
18.62
0.4804
0.4474
17.53
0,5947
0.2420
18.08
0.5376
0.3447
CTP Threshold(1 = 10, T= 48)
18.95
0,5052
0.3660
18.14
0.6306
0.2275
18.55
0.5679
0.2968
CTP Thresholdir= = 30, T= 32)
19.02
0,5310
0.3515
18.23
0.6318
0.2373
18.63
0.5814
0.2944
CTP Threshold( = 30. T (8urs))
19.06
0.5354
0.3491
18.12
0.6331
0.2265
18.59
0.5843
0.2878
CTP Threshold(r 50, T= 48)
19.10
0,5051
0.3634
17.78
0.6114
0.2384
18.44
0.5583
0.3009
Table 4: The ablation studies of the CTP module.
The experimental results show that the frequency filter provides more desirable mask, which is superior to the directly obtained mask with RGB intensity thresholds.
A.5 More Visual Results
We provide more visual comparisons between our proposed LuSh-NeRL network and the sota methods combinations, iee., LEDNet [67] NeRF, Restormer [59] I.LNRRF [47], MPRNet [60] +
LLNeRF [47], Lai [13] + DP-NeRF [19] and URetinexNet [54] + DP-NeRF [19] in Fig. 9.
15

---

A.6 Comparison with COLMAP-Free NeRF method
COLMAP-free NeRF methods robust low-light and blur phenomena would be much more helpful for the problem that we propose. However, existing colmap-fite free NeRF methods may not handle our task easily. The table below compares a COLMAP-Free NeRF [51].
Socoe COLLLEL
PSNK
DDrm
SSIM
Poster
PSNR
'Plane'
SNIM
LPIPS
Average
SSTM
PIPS
19.75
0.5399
0.4703
19.07
0.7048
0,3088
19.66
0.233
0.4986
19.49
0.3959
0.4200
Preprocess+NeRt NeRF -- [SI]
18.95
0.5423
0.4762
19.13
0.6935
0.3341
19.62
0.5243
0,5129
19.23
0.5867
0.4411
Table 5: Performance comparison with COLMAP-free NeRF method.
The results demonstrate that the existing COLMAP-free NeRF method cannot effectively handle low-light scenes with motion blur, due to the inaccurate poses optimized from the image directly.
A.7 The Generalizability of the GIM
The image-matching method used in LuSh-NeRF is GIM [43], which is a saagge-macching model pre-trained with abundant Internet videos. As mentioned Section 3.2, we also utilize pre-defined threshold to select those high-confidence matches in the results produced by GIM, avoiding low-quality matches that may affect the network's training negatively.
generalizable
We show some of the matches obtained by GIM on S-NeRF rendered results in Fig. 8.
Synthetie
Reclistie Scene
Figure 8: The visualization of matching results of GIM [43] on the S-NeRF rendering images.
A.8 Discussion
LuSh-NeRF may have some limitations certain data, as shown in Fig. 10. Although the noise in low light images random and fluctuates greatly, if the noise in certain areas maintains similar characteristics in multiple viewpoints, LuSh-NeRF may model it as scene information, resulting in errors in rendering results. Increasing the training image numbers and spanning the range of different views may alleviate this problem to some extent.
16

---

CARPARI ADUATE
ADUATE
ADUATE
Input
LEDNEr [67] NNRR
Restormer [59]+L.L NeRF [47]
MPRNet NELRF [47]
CARPARK
CARPARK
CARPARK ADUATE
CARPARK ADUATE
PaizL IE 113 199
L'Retiee (54 Deij--P-NNee NeRF 1191
LuSh-NeRE (Ours)
Ground Truth
N E
Input
LEDNet N6RF
Restormer NoRRE (47)
MPRNet [s0)-LLNGRF [47]
N
INE
P PINE
Pairt IE NeRF [19]
L'Retinex [Sa]-DP-NeRF) [19]
LeRF (Ours)
Ground Truth
Figure 9: Qualitative results of different methods the synthesized datasets.
Inpor:
Prepracess Imput:
Training View:
view;
Voows
Novel View Resalt:
URetineeNat DP-NoRF
MPRNar 1.NoRF
Prirl DP-NoRF
Ours
Figure 10: limitation of the proposed LuSh-NeRF, Noise that relatively similar across views may be difficult to remove due to the reliance on viewpoint consistency.
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have detailed the main contributions of this paper in the Abstract and Introduction, and summarized them at the end the Introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Section 5 and the visulization results in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have discussed it in the Subsection 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
We have provided the project link in the adstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppccceecccccccccccccclt S ssisssolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See Subsection 4.1 and 4.2.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper provides extensive information about the statistical significance of our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Subsection 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https http://neuippppe wttp::ppppppppppccttttececccccce http:////eehiiccccceeeeeeeeeeeee
Answer: [Yes]
Justification: We have followed the Ethics Guidelines to conduct our research and prepare our submission.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix A.I.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited the corresponding papers when using their codes, data, and models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will provide the documentation along with the new assets in project
page.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "9xoyNdmuUo", "reviewer": "Reviewer_4gW7", "summary": "This method proposes a solution for NeRF optimization under low light settings by resolving 3 different forms of degradation: low intensity, camera noise, and motion blur. Low intensity is effectively resolved by scaling up the image, camera noise is resolved by proposing a consistency loss between different views to locate the noise, and motion blur is handled by using a camera trajectory prediction module to predict the camera motion and thus the corresponding sharpening function. The experiments demonstrate state-of-the-art performance on novel view synthesis given low-light conditions. Qualitative ablations are provided to support the contribution of each component of the method. The authors will also make their code readily available as well as a dataset with 5 synthetic and 5 real scenes for evaluating low-light novel view synthesis.", "questions": "As mentioned above, please provide quantitative results to support the ablation studies and be sure to fix the errors in Table 1. I will be considering the rebuttal carefully as well as the thoughts from other reviewers in deciding my final rating.", "limitations": "Limitations and broader impact section are both available and appropriate. This work naturally does not have many immediately obvious negative societal impacts as it is simply trying to faithfully render novel-view images in low-light conditions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The experiments support that the model is state-of-the-art, as in most scenes the proposed method achieves the best performance for low-light novel view synthesis. \n\nThe model is well thought out and handles many types of degradations, including low intensity, noise, and motion blur. The SND and CTP modules are well designed and novel components of the method. \n\nThe paper presentation is clear and easy to follow with appropriate figures to aid the reader's understanding. \n\nThe code will be made publicly available and a dataset will be released to aid evaluations in the low-light novel view synthesis domain.", "weaknesses": "The ablations would be more convincing with some quantitative results to back up the qualitative results. \n\nThere appear to be some mistakes in Table 1: in some columns two numbers are underlined, in others the best and second best seem to be incorrectly reported. Please be careful to fix these mistakes. \n\nThere are some important citations missing for some parts of the related work, such as in line 71 when mentioning NeRF methods that model digital humans: \n\n1. HeadNeRF (CVPR 2022)\n2. MoFaNeRF (ECCV 2022)\n3. INFAMOUS-NeRF (arxiv, 2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1: Ablation studies with quantitative results.\n**R:** As suggested, we have conducted detailed ablation studies on all the synthetic scenes in the following table:\n| Scene                        | Dorm  |        |         | Poster |         |         | Plane |        |        | Sakura |         |        | Hall  |        |         | Average |         |         |\n|------------------------------|-------|--------|---------|--------|---------|---------|-------|--------|--------|--------|---------|--------|-------|--------|---------|---------|---------|---------|\n|                              | PSNR$\\uparrow$  | SSIM$\\uparrow$   | LPIPS$\\downarrow$   | PSNR$\\uparrow$   | SSIM$\\uparrow$    | LPIPS$\\downarrow$   | PSNR$\\uparrow$  | SSIM$\\uparrow$   | LPIPS$\\downarrow$  | PSNR$\\uparrow$   | SSIM$\\uparrow$    | LPIPS$\\downarrow$  | PSNR$\\uparrow$  | SSIM$\\uparrow$   | LPIPS$\\downarrow$   | PSNR$\\uparrow$    | SSIM$\\uparrow$    | LPIPS$\\downarrow$   |\n| NeRF                         | 6.02  | 0.0307 | 0.8030  | 11.25  | 0.5159  | 0.4061  | 5.53  | 0.0716 | 0.8418 | 7.54   | 0.0553  | 0.7186 | 8.19  | 0.1679 | 0.5048  | 7.706   | 0.1683 | 0.6549 |\n| ScaleUp                 | 19.75 | 0.5599 | 0.4705  | 19.07  | 0.7048  | 0.3088  | 19.66 | 0.5230  | 0.4986 | 18.73  | 0.5699  | 0.3666 | 21.34 | 0.7213 | 0.2831  | 19.71   | 0.6158 | 0.3855 |\n| ScaleUp+Blur Kernel          | 18.89 | 0.5259 | 0.4353  | 17.23  | 0.5900  | 0.2805  | 19.13 | 0.5193 | 0.4185 | 18.23  | 0.5482  | 0.2789 | 20.43 | 0.6353 | 0.2684  | 18.78  | 0.5637 | 0.3363 |\n| ScaleUp+CTP                  | 18.52 | 0.5205 | 0.3654  | 17.02  | 0.5915  | 0.2415  | 19.32 | 0.5144 | 0.4048 | 18.27  | 0.5514  | 0.2715 | 20.25 | 0.6411 | 0.2577  | 18.68  | 0.5638 | 0.3082 |\n| ScaleUp+SND                  | **20.18** | **0.5646** | 0.4490  | **21.37**  | **0.7542**  | 0.2534  | **20.13** | **0.5665** | 0.4873 | **19.16**  | **0.5889**  | 0.3568 | **21.67** | **0.7326** | 0.2801  | **20.50**  | **0.6414** | 0.3653 |\n| ScaleUp+Blur Kernel+SND      | 18.99 | 0.5299 | 0.3630  | 18.05  | 0.6179  | 0.2598  | 18.93 | 0.5191 | 0.3954 | 18.65  | 0.5530  | 0.2752 | 20.74 | 0.6381 | 0.2434  | 19.07  | 0.5716  | 0.3074 |\n| LuSh-NeRF (Sharp -> Denoise) | 18.66 | 0.5008 | 0.3514  | 17.38  | 0.5860   | 0.2600  | 19.13 | 0.5213 | 0.4076 | 18.24  | 0.5420  | 0.2589 | 20.72 | 0.6386 | 0.2667  | 18.83  | 0.5578 | 0.3089 |\n| LuSh-NeRF (Denoise -> Sharp) | 19.06 | 0.5354 | **0.3491**  | 18.12  | 0.6331  | **0.2265**  | 19.34 | 0.5275 | **0.3852** | 18.94  | 0.5884  | **0.2562** | 21.09 | 0.6421 | **0.2400**  | 19.31   | 0.5853  | **0.2914**  |\n\n(1) From Line 1 and 2, the ScaleUp preprocessing enhances the NeRF's reconstruction capabilities, resulting in improved image. \n\n(2) From Line 3 and 4, CTP leverages frequency domain information to refine Blur Kernel predictions, boosting the perceptual quality of reconstructed images. However, this process **diminishes** PSNR and SSIM results due to neglecting **noise interference**. \n\n(3) From Line 2 and 5, the SND module can **disentangle noise and scene** information from input noisy-blurry images, leading to substantial improvements in PSNR and SSIM metrics. However, the SND module is not capable of resolving the blur problem, which leads to an **minor improvement** in the image's perceptual quality (**more important** for the rendered images).\n\n(4) From Line 4, 5, 8, the combined application of SND and CTP modules enhances image **perceptual quality** and outperforms the sole use of CTP in terms of PSNR and SSIM and SND in terms of LPIPS. \n\n(5) From Line 7, 8, it can be concluded that **decoupling the noise first**, and **then modeling the scene blur** is a more robust restoration order, which can effectively reduce the interference of noise in the deblurring process, and obtain better performance metrics. \n\n### W2: Mistakes in Tab.1.\n**R:** Thanks for your careful reading, we will correct these mistakes **in the revision**.\n\n\n### W3: Missing Citations.\n**R:** Thanks for your valuable reviews. HeadNeRF [1] integrates NeRF to the parametric representation of the human head. MoFaNeRF [2] propose the first parametric model that maps free-view facial images into a vector space with NeRF. INFAMOUS-NeRF [3] propose a novel photometric surface constraint that improves the face rendering performance.\n\nThese inspiring works are of importance to the Human NeRF development. As suggested, we will **cite and discuss** these works in our revision.\n\n[1] Hong, Yang, et al. \"Headnerf: A real-time nerf-based parametric head model.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[2] Zhuang, Yiyu, et al. \"Mofanerf: Morphable facial neural radiance field.\" European conference on computer vision. Cham: Springer Nature Switzerland, 2022.\n\n[3] Hou, Andrew, et al. \"INFAMOUS-NeRF: ImproviNg FAce MOdeling Using Semantically-Aligned Hypernetworks with Neural Radiance Fields.\" arXiv preprint arXiv:2312.16197 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eL6gA9e2cW", "reviewer": "Reviewer_54cf", "summary": "The authors propose LuSh-NeRF, a model that reconstructs a clean and sharp NeRF from handheld low-light images by sequentially modeling noise and blur. LuSh-NeRF includes a Scene-Noise Decomposition (SND) module for noise removal and a Camera Trajectory Prediction (CTP) module for estimating camera motions based on low-frequency scene information. Experiments demonstrate that LuSh-NeRF outperforms existing methods in rendering bright and sharp novel-view images from low-light scenes.", "questions": "1. I have another small suggestion. Could the authors consider compressing the images, or perhaps saving them in PDF format? This might help reduce the memory size of the paper's PDF file.\n\n2. Could the author show some comparison with Aleth-NeRF [7] ? which is also a low-light NeRF method.", "limitations": "Please refer to the weakness part.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.The paper aims to address the challenge of using NeRFs in low-light conditions, where images often suffer from low visibility, noise, and camera shakes together.\n\n2. A new dataset containing synthetic and real images is constructed to facilitate training and evaluation.\n\n3. The method is grounded in a sound theoretical framework, leveraging multi-view consistency and frequency-domain information.", "weaknesses": "1. For me, the ideas and motivation behind this paper are quite good. What I am concern is whether the authors could consider adding some ablation studies, such as the role of various modules in LuSh-NeRF, this could let reader know which part is more effective.\n\n2. In the scenario described by the author, I think the colmap-free NeRF method of this task may be more effective. Low-light blurred images have a certain impact on colmap estimation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1: Ablation studies regarding the roles of proposed modules.\n**R:** Thanks for your positive feedback on our work, the **visualization** of the ablation experiments can be found in **Fig.6** in the main text. To better demonstrate the effectiveness of the different modules in LuSh-NeRF, we performed **quantitative ablation experiments** on all the synthetic datasets in the table below:\n| Scene | Dorm  |  |  | Poster |  |   | Plane |  |  | Sakura |  |   | Hall  |  | | Average |  |  |\n|------------------------------|-------|--------|---------|--------|---------|---------|-------|--------|--------|--------|---------|--------|-------|--------|---------|---------|---------|---------|\n|                              | PSNR$\\uparrow$  | SSIM$\\uparrow$   | LPIPS$\\downarrow$   | PSNR$\\uparrow$   | SSIM$\\uparrow$    | LPIPS$\\downarrow$   | PSNR$\\uparrow$  | SSIM$\\uparrow$   | LPIPS$\\downarrow$  | PSNR$\\uparrow$   | SSIM$\\uparrow$    | LPIPS$\\downarrow$  | PSNR$\\uparrow$  | SSIM$\\uparrow$   | LPIPS$\\downarrow$   | PSNR$\\uparrow$    | SSIM$\\uparrow$    | LPIPS$\\downarrow$   |\n| NeRF                         | 6.02  | 0.0307 | 0.8030  | 11.25  | 0.5159  | 0.4061  | 5.53  | 0.0716 | 0.8418 | 7.54   | 0.0553  | 0.7186 | 8.19  | 0.1679 | 0.5048  | 7.706   | 0.1683 | 0.6549 |\n| ScaleUp                 | 19.75 | 0.5599 | 0.4705  | 19.07  | 0.7048  | 0.3088  | 19.66 | 0.523  | 0.4986 | 18.73  | 0.5699  | 0.3666 | 21.34 | 0.7213 | 0.2831  | 19.71   | 0.6158 | 0.3855 |\n| ScaleUp+Blur Kernel          | 18.89 | 0.5259 | 0.4353  | 17.23  | 0.5900  | 0.2805  | 19.13 | 0.5193 | 0.4185 | 18.23  | 0.5482  | 0.2789 | 20.43 | 0.6353 | 0.2684  | 18.78  | 0.5637 | 0.3363 |\n| ScaleUp+CTP                  | 18.52 | 0.5205 | 0.3654  | 17.02  | 0.5915  | 0.2415  | 19.32 | 0.5144 | 0.4048 | 18.27  | 0.5514  | 0.2715 | 20.25 | 0.6411 | 0.2577  | 18.68  | 0.5638 | 0.3082 |\n| ScaleUp+SND                  | **20.18** | **0.5646** | 0.4490  | **21.37**  | **0.7542**  | 0.2534  | **20.13** | **0.5665** | 0.4873 | **19.16**  | **0.5889**  | 0.3568 | **21.67** | **0.7326** | 0.2801  | **20.50**  | **0.6414** | 0.3653 |\n| ScaleUp+Blur Kernel+SND      | 18.99 | 0.5299 | 0.3630  | 18.05  | 0.6179  | 0.2598  | 18.93 | 0.5191 | 0.3954 | 18.65  | 0.5530  | 0.2752 | 20.74 | 0.6381 | 0.2434  | 19.07  | 0.5716  | 0.3074 |\n| LuSh-NeRF (Sharp -> Denoise) | 18.66 | 0.5008 | 0.3514  | 17.38  | 0.5860   | 0.2600  | 19.13 | 0.5213 | 0.4076 | 18.24  | 0.5420  | 0.2589 | 20.72 | 0.6386 | 0.2667  | 18.83  | 0.5578 | 0.3089 |\n| LuSh-NeRF (Denoise -> Sharp) | 19.06 | 0.5354 | **0.3491**  | 18.12  | 0.6331  | **0.2265**  | 19.34 | 0.5275 | **0.3852** | 18.94  | 0.5884  | **0.2562** | 21.09 | 0.6421 | **0.2400**  | 19.31   | 0.5853  | **0.2914**  |\n\nThe specific analysis is as follows:\n\n(1) From Line 1 and 2, the ScaleUp preprocessing enhances the NeRF's reconstruction capabilities, resulting in improved rendering results. \n\n(2) From Line 3 and 4, CTP leverages frequency domain information to refine Blur Kernel predictions, boosting the perceptual quality of reconstructed images. However, this process **diminishes** PSNR and SSIM results due to neglecting **noise interference**. \n\n(3) From Line 2 and 5, the SND module can **disentangle noise and scene** information from input noisy-blurry images, leading to substantial improvements in PSNR and SSIM metrics. However, the SND module is not capable of resolving the blur problem, which leads to an **minor improvement** in the image's perceptual quality (**more important** for the rendered images).\n\n(4) From Line 4, 5, 8, the combined application of SND and CTP modules enhances image **perceptual quality** and outperforms **the sole use** of CTP in terms of PSNR and SSIM and SND in terms of LPIPS. \n\n(5) From Line 7, 8, it can be concluded that **decoupling the noise first**, and **then modeling the scene blur** is a more robust restoration order, which can effectively reduce the interference of noise in the deblurring process, and obtain better performance metrics.\n\n### W2: Colmap-free NeRF methods may be more effective?\n**R:** The authors highly agree with your ideas. Colmap-free NeRF methods robust to low-light and blur phenomena would be much more helpful for the problem we propose.  However, existing colmap-free NeRF methods may not handle our task easily.\n\nThe table below compares to a COLMAP-Free NeRF [1]. The results demonstrate the existing COLMAP-free NeRF method cannot effectively handle low-light scenes with motion blur, due to the inaccurate poses optimized from the image directly. Exploring colmap-free methods for our task can be interesting for future work.\n\n| Scene| \"Dorm\" | || \"Poster\" | | | \"Plane\" | | | Average | | |\n|--|--|--|--|--|--|--|--|--|--|--|--|--|\n|| PSNR$\\uparrow$ | SSIM$\\uparrow$| LPIPS$\\downarrow$ | PSNR$\\uparrow$ | SSIM$\\uparrow$| LPIPS$\\downarrow$ | PSNR$\\uparrow$  | SSIM$\\uparrow$ | LPIPS$\\downarrow$| PSNR$\\uparrow$ | SSIM$\\uparrow$| LPIPS$\\downarrow$|\n| Preprocess+NeRF(COLMAP) | **19.75** | **0.5599**| **0.4705** | 19.07| **0.7048** | **0.3088** | **19.66**   | 0.5230  | **0.4986** | **19.49** | **0.5959** | **0.4260** |\n| Preprocess+NeRF-\\-[1]  | 18.95  | 0.5423 | 0.4762 | **19.13** | 0.6935 | 0.3341 |19.62| **0.5243** | 0.5129 | 19.23 | 0.5867 | 0.4411 |\n\n[1] Wang Z, Wu S, Xie W, et al. NeRF--: Neural radiance fields without known camera parameters[J]. arXiv preprint arXiv:2102.07064, 2021.\n\n### Q1: Reduce the memory size of the paper.\n**R:** Thanks for the suggestion, we will compress the images and reduce the PDF size in our revision.\n\n### Q2: More comparison with Aleth-NeRF.\n**R:** As suggested, we have performed several experiments with Aleth-NeRF [7] on our proposed synthetic and realistic scenes. The visualization results of one comparison in this experiment are shown **in the Fig.1 in Rebuttal PDF**.\n\nAs Aleth-NeRF focuses on adjusting the luminance of the scene, it **cannot handle the blur** in low-light scenes. Will include more in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KqfukG1bWF", "reviewer": "Reviewer_4eod", "summary": "The paper proposed a method to train a NeRF with blurry (due to camera motion), low-light scene images. After training the method allows the recovery of enhanced, sharp images. To solve the problem two modules are proposed:\n\n1) A SND module for noise modeling, which includes both a noise-prediction NeRF (N-NeRF) and a consistency loss, which uses image matching to constrain the radiance to be multiview consistent\n\n2) A CTP module to model the blur induced by camera motion\n\nThe paper evaluates the method and baselines on a self-collected dataset.", "questions": "1) I think the $n_{N/2}$ refers to the N-NeRF output at the 3D coordinate which is the midpoint of the ray samples, is that right? If we don’t care about the 3D consistency of N-NeRF (since noise is not 3D consistent) why is the noise even modeled with a NeRF? Why not just optimize a tensor per view, or have the noise be a ray-conditioned network (instead of 3D coordinate conditioned)? \n\n2) I wonder how important it is to low-pass filter the images before thresholding for the CTP module, have the authors tried to just threshold the image without the low-pass filter?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "Originality: I appreciate the problem being tackled in the paper and think this is an underexplored problem. I think we are still far from being able to use NeRFs on casually captured phone videos, as evidenced by apps such as Luma, which although produce excellent quality require no (camera) motion blur and perfect lighting in the captures. I also appreciate the authors trying to assemble their own dataset although I do have some qualms about their claims (see weakness section). \n\nQuality: I think the experiments and baselines have been chosen well.", "weaknesses": "Quality: Even though I believe the dataset to be useful, I find it hard to attribute the dataset to the authors of the paper. If I understand correctly (please correct me if I am wrong), the original images were actually captured by prior work (Lednet), and the authors of LuSh-NeRF only ran an off-the-shelf SFM method to recover the camera parameters? Happy to be shown otherwise, but I do not think in this case the authors can claim the dataset as a contribution of the paper. \nOther than that although the results seem convincing, the contribution of the paper isn’t significant. \n\nClarity: I had serious issues understanding the paper, specifically the methods section. Authors introduce a lot of names, which I think would be better replaced by just mathematical notation. Some of the notation is also not properly explained, for example, $n_{N/2}$ in Eq. 5. (see also questions). \n\nIt is also unclear to me how exactly Equations 4 and 5 relate. If I understand correctly Eq 4. suggests that to render the low-light, blurry images the network prediction is first unsharpened and then noise is added, but in Eq 5. noise is directly added to the radiance predictions from the network?\n\nI think part of the CTP module also comes from the paper DP-NeRF, but I think the paper does not exactly mention what this module does. In the interest of making the paper self-sufficient, I think the authors should elaborate on what the module does/what it is exactly. I think what they do elaborate on (the detaching of certain rays) is just their contribution on top of the method from DP-NeRF, if I understand correctly. \n\n\nA few typos, there’s probably more I missed:\n\nL145: bracket not closed \n\nL183: bracket not opened", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1: Insufficient Dataset Contribution.\n**R:** Actually, we did **not just** take images from the LOL-Blur dataset and then ran an off-the-shelf SFM method to obtain the camera parameters. To build an effective dataset, we did the following works:\n\n (1) **Scene selection**: We went through the whole LOL-Blur dataset to select scenes **featuring different environments** (covering indoor/outdoor situations) with **different camera motions and lighting.**\n\n (2) **Image selection**: We manually select 20-25 (out of around 60) images per scene with relatively **high-quality** for the estimation of camera poses, and ensure their **luminance** is close to imitate the real shooting situation.\n\n (3) **Camera Pose Estimation**: Note that the estimation of camera pose by COLMAP for low-light blurry images are often **unreliable**. We first repeat the estimation of COLMAP **30 times for each scene** to select the optimal pose result, and then **manually tune** it with our baseline NeRF model to improve the accuracy.\n\nOur experimental results show that the resulting dataset helps us learn the **robust LuSh-NeRF** for handling low-light blurry scenes. We will clarify these in the revision.\n\n### W2: Unclear notation.\n**R:** $n_{\\frac{N}{2}}$ represents the noise value obtained by the N-NeRF MLP structure, and the $\\frac{N}{2}$ means the middle sampled points on each ray. \n\nWe thank this reviewer for pointing out this issue, and will **revise the Method section** to improve the clarity in the revision.\n\n### W3: Relation of Equations 4 and 5.\n**R:** We apologize for the confusion caused by Eq.5, as it omits the deblur process in LuSh-NeRF, and the revised Eq.5 should be:\n$ C_{noisy}(r) = CTP(C_{S-NeRF}(r)) + C_{N-NeRF}(r) = CTP(\\sum_{i=1}^{N} w_i c_i) + n_{\\frac{N}{2}},\n\\quad  where \\quad n_{\\frac{N}{2}} = MLP_{N-NeRF}(P_{mid}, d), $\n\nwhere the $P_{mid}$ and $d$ is the intermediate sampling points coordinate and the view direction. The $\\text{CTP}(\\cdot)$ function is the CTP module, which is illustrated in the Subsec. 3.3.\n\n### W4: Relation between CTP module and DP-NeRF.\n**R:** The CTP module adopts the Rigid Blurring Kernel (RBK) module of DP-NeRF. The RBK module uses two MLPs to model the Rigid Camera Motion parameters for each viewpoint in the dense SE(3) field, one MLP to calculate the weights of the rendered pixels obtained from each camera pose within the kernel, then these pixels are used to compute the final blurry pixel values via weighted sum operation.\n\nThe **difference** between the CTP and the RBK module lies in that, the **frequency-domain information** is introduced into the kernel estimation to help achieve **more accurate blur modeling** by exploiting the image regions which are less affected by the low light noise. We will clarity them in the revision.\n\n### Q1: Why is the noise modeled with a NeRF? Why not optimize ...\n**R:** Thanks for the question. We refer to it N-NeRF as it is a multilayer MLP network **similar to the S-NeRF**. We will revise its name to **\"Noise Estimator\"** in the revision to avoid any ambiguities.\n\nThe N-NeRF is **ray-conditioned** as the inputs are the view direction $d$ and the coordinates of the mid-point **(fixed for each ray)** during the whole training process.\n\nAs suggested, we try the **view-dependent inputs** $(x,y,N)$, i.e., a 2D coordinate $(x,y)$ with the view id $N$ as the inputs to the N-NeRF to optimize a **noise tensor per view**. The results are reported below and the performance difference is **quite small**, which shows that **both strategies work** in our task.\n\n|Scene|\"Dorm\"|||\"Poster\"|||\n|-|-|-|-|-|-|-|\n||PSNR$\\uparrow$|SSIM$\\uparrow$|LPIPS$\\downarrow$|PSNR$\\uparrow$|SSIM$\\uparrow$|LPIPS$\\downarrow$|\n|View-Independent Noise|**19.10**|0.5278|**0.3451**|18.08|**0.6348**|0.2343|\n|Ours|19.06|**0.5354**|0.3491|**18.12**|0.6331|**0.2265**|\n\n### Q2 Part1: The importance of the low-pass filter.\n**R:** We have shown in the **Fig. 2 in rebuttal PDF** the difference between the two masks obtained by directly performing a rgb intensity threshold and performing a Frequency filtering before taking a threshold.\n\n(1) The RGB intensity of many **noise-dominant** regions are also **high** (e.g., the sky and the grass), which are harmful to the blur estimation but **not excluded in the mask** produced by thresholding.\n\n(2) The CTP module uses a Frequency filter to identify the **low-frequency-dominated** regions of the image, and then obtains the desired mask based on the RGB intensity values. The gradients of rays in high-frequency and/or dark regions (regions more **severely affected by noise**) are detached during the blur kernel optimization process, which ensures better blur modeling.\n\n### Q2 Part2: Just threshold the image without the low-pass filter.\n**R:** We have conducted the ablation experiments of CTP module on two synthetic scenes in the following table.\n| Scene|\"Dorm\"||| \"Poster\" ||| Average |||\n|-|-|-|-|-|-|-|-|-|-|\n|| PSNR$\\uparrow$|SSIM$\\uparrow$|LPIPS$\\downarrow$|PSNR$\\uparrow$|SSIM$\\uparrow$|LPIPS$\\downarrow$| PSNR$\\uparrow$|SSIM$\\uparrow$|LPIPS$\\downarrow$|\n|No Threshold|18.99|0.5299|0.3630|18.05|0.6179|0.2598|18.52|0.5739|0.3114|\n|RGB Threshold(T=32)|**19.18**|0.5308|0.3580|17.86|0.5896|0.2450|18.52|0.5602|0.3015|\n|RGB Threshold(T=48)|18.71|0.4874|0.4455|17.92|0.6057|0.2418|18.32|0.5466|0.3437|\n|RGB Threshold(T=64)|18.62|0.4804|0.4474|17.53|0.5947|0.2420|18.08|0.5376|0.3447|\n|CTP Threshold(r=10, T=48)|18.95|0.5052|0.3660|18.14|0.6306|0.2275|18.55|0.5679|0.2968|\n|CTP Threshold(r=30, T=32)|19.02|0.5310|0.3515|**18.23**|0.6318|0.2373|**18.63**|0.5814|0.2944|\n|**CTP Threshold(r=30, T=48)**|19.06|**0.5354**|**0.3491**|18.12|**0.6331**|**0.2265**|18.59|**0.5843**|**0.2878**|\n|CTP Threshold(r=50, T=48)|19.10|0.5051|0.3634|17.78|0.6114|0.2384|18.44|0.5583|0.3009|\n\nThe experimental results show that the frequency filter provides a **more desirable** mask, which is superior to the directly obtaining mask with the RBG intensity thresholds.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DuPIYUVB7g", "reviewer": "Reviewer_m3Lt", "summary": "This work proposes a model to reconstruct a clean and sharp NeRF from a set of hand-held low-light images. The authors recognize the implicit order of the degradations (blur, noise, and low visibility), and sequentially decouple and remove each degradation in the network training. An SND module is proposed for noise removal and a CTP module is introduced for deblurring. They have conducted experiments on a dataset constructed from the LOL-Blur dataset proposed by [61].", "questions": "Please see weaknesses.", "limitations": "Yes. Limitations are included in Section 5 and negative societal impacts are included in Section A.1.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This is the first work to reconstruct degradation-free images from a set of low-light photographs with motion blur.\n2. They have introduced a new method for scene-noise decomposition from the implicit scene representation and utilized the frequency information for accurate kernel prediction NeRF. \n3. The qualitative results are appealing and the authors have shown better quantitative scores for the method.", "weaknesses": "1. The presentation of the problem through mathematical equations is not clear. In Eq. 5, $C_{noisy}$ is the color value of a bright light blurry noisy (given in Eq. 4) image. But in Eq. 5, the first term $C_{S-NeRF}$ output is the sharp image and the second term is the noise term. The blurring operation is missing in this equation. Please correct me if I misinterpret anything. Also, N-NeRF does not use any volume rendering calculation to find the noise pixel value. Then, why this module is called a NeRF? It simply finds a pixel value for a given input pose. How the second term in Eq. 5 ($n\\frac{N}{2}$) is derived? In Eq. 6, what are $i$ and $j$? In Fig. 2, the align ray is given subscripts $j$ and $k$. Please be consistent with equations and figure. \n\n2. In the paper, it is given that the camera trajectory prediction (CTP) module is novel (Line 16, 47). But the same idea of [19] (Line 180) is used for camera trajectory prediction with the additional masking of high-frequency pixel regions in the image. The main function of deblur-NeRF in the CTP module is not novel. \n\n3. Edge portions in the image will have high frequencies. When only rays containing low frequencies are used for gradient computation, typically edges (or high-frequency regions) will not come into the picture. But for deblur-NeRF, excluding the edge areas (high-frequency) is not a good idea since the blurring is mostly visible in the edges other than homogeneous (low-frequency) areas. How your frequency filter radius is selected? Did you consider any such factors for its selection?\n\n4. In Line 224, it is given that for the first 60K iterations, $\\beta$ is 0. This means SND does not come into the picture. In that case, the blur kernel estimated (by deblur-NeRF) will be wrong right (according to lines 185-186)? Is it correct to start from these trained weights after 60K iterations? What happens if $\\beta$ is not 0 in the initial iterations?\n\n5. SND relies on the image-matching method [40]. How generalizable is the method to your dataset? Have you taken the pre-trained weights? \n\n6. The contribution of the dataset (given in Line 64) is not that significant since the poses of the available dataset from [61] are computed for the dataset generation using COLMAP. The method is tested on this single dataset (containing synthetic and real videos). Since the work is meant for handheld lowlight photography, a video of a low-light scene with handheld cameras (which typically will have a camera shake) could have been captured and the proposed method could have been tested on the dataset. \n\n7. You have missed several references for Deblur NeRF papers. \"ExBluRF: Efficient Radiance Fields for Extreme Motion Blurred Images\", ICCV 2023; \"PDRF: progressively deblurring radiance field for fast scene reconstruction from blurry images\", AAAI 2023; \"Inverting the Imaging Process by Learning an Implicit Camera Model\", CVPR 2023. The related works (Sec. 2) section is not proper. There should be subsections for deblur NeRF and lowlight NeRF and briefly explain each work in both, other than explaining NeRF. Also, only low-light image enhancement methods are included. Why are deblurring methods not included? Or, it is better to have a subsection with explanations of lowlight deblurring works [61, 56]. Instead of just giving reference numbers, the essence of each very relevant work should be explained in a sentence in 'Related works'.\n\n8. Some typos are there. In Line 153 and Fig. 6 (a) caption. In Line 35, [29] is not for low-light scenes. Line 41, the abbreviation of ISP is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1 Part1: Error in Eq.4 & 5 and the missing derivation.\n**R:** We thank the reviewer for the careful reading and apologize for the mistakes in Eq.5, which **omitted the deblurring process**.\n The **missing derivation** is also added to Eq. 5. The revised Eq. 5 should be:\n$ C_{noisy}(r) = CTP(C_{S-NeRF}(r)) + C_{N-NeRF}(r) = CTP(\\sum_{i=1}^{N} w_i c_i) + n_{\\frac{N}{2}},\n\\quad  where \\quad n_{\\frac{N}{2}} = MLP_{N-NeRF}(P_{mid}, d), $\n\nwhere the $P_{mid}$ and $d$ is the intermediate point coordinate and the view direction. The $\\text{CTP}(\\cdot)$ function is the CTP module, which is illustrated in the Subsec. 3.3.\n\n### W1 Part2: Inproper name for N-NeRF.\n**R:** We refer to it N-NeRF as it is a **multilayer MLP network** similar to the S-NeRF, and it is **ray-conditioned** as the inputs are the view direction $d$ and the coordinates of the mid-point (fixed for each ray) during the whole training process. We will revise its name to **\"Noise Estimator\"** in the revision to avoid any ambiguities.\n\n### W2: Insufficient novelty of CTP module.\n**R:** Note that DP-NeRF [19] does not consider the frequency domain information. DP-NeRF and other works directly uses **the rays of all image regions** to predict the camera trajectory, which is severely interfere by the low light noise (see Fig. 6(b) for details).\n\nThe novelty of our CTP module lies in that it learns to identify **low-frequency-dominated regions** that are **more robust to noise** for kernel prediction. With these regions, LuSh-NeRF can significantly reduces the influence of noise on camera trajectory modeling. Some ablation experiments is in the table below.\n|Scene|\"Dorm\"|||\"Poster\"|||\n|-|-|-|-|-|-|-|\n||PSNR|SSIM|LPIPS|PSNR|SSIM|LPIPS|\n|NeRF+DP-NeRF Blur Kernel|18.99|0.5299|0.3630|18.05|0.6179|0.2598|\n|NeRF+CTP|**19.06**|**0.5354**|**0.3491**|**18.12**|**0.6331**|**0.2265**|\n\nFrom the table, it can be derived that the CTP module that utilizes the help of **low frequency information** can be **better employed** in low light scene. \n\n### W3: Excluding the edge areas.\n**R:** The reviewer is correct that blurring is more noticeable in high-frequency areas. However, in our task, noise may also significantly affect the images, making the high-frequency information **unreliable** for blur kernel predictions.\n\nOur CTP module is proposed to minimize the negative influence of these **low-quality** regions for blur modeling. As shown in **Fig.2 in the rebuttal PDF**, the **main edges** in the image can be effectively **preserved**, while the severely disturbed regions (e.g., the grass and the sky) are discarded in the optimization process.\n\nThe tables below and in R3Q2 show the quantitative results of using different frequency filter radius $r$ and intensity threshold $T$, based on which we set the radius to 30. Will discuss more in the revision.\n|Scene|\"Dorm\"||| \"Poster\" |||\n|-|-|-|-|-|-|-|\n|| PSNR|SSIM|LPIPS|PSNR|SSIM|LPIPS| \n|No Mask|18.99|0.5299|0.3630|18.05|0.6179|0.2598|\n|r=10,T=48|18.95|0.5052|0.3660|18.14|0.6306|0.2275|\n|r=30,T=32|19.02|0.5310|0.3515|**18.23**|0.6318|0.2373|\n|**r=30,T=48**|19.06|**0.5354**|**0.3491**|18.12|**0.6331**|**0.2265**|\n|r=50,T=48|19.10|0.5051|0.3634|17.78|0.6114|0.2384|\n\n### W4: Correct to start training N-NeRF after 60K iters?\n**R:** When $\\beta=0$, N-NeRF in SND module is still **co-optimized** with S-NeRF for noise estimation. When $\\beta$ is turned on after 60K iterations, $L_{consistency}$ will **reinforce the consistency** across different views, which **further** facilitates the noise estimation of N-NeRF. \n\nIf $L_{consistency}$ is used at the beginning of the training phase, the Image Matching method cannot accurately align the images due to the **low quality** of the images rendered by the S-NeRF in the early training stage, which may **deteriorate** the performance and extend the training time. \n\nWe report the results of turning on $\\beta$ at different iterations as references:\n|Scene|\"Dorm\"|||\"Poster\"|||\n|-|-|-|-|-|-|-|\n||PSNR|SSIM|LPIPS|PSNR|SSIM|LPIPS|\n|No $L_{consistency}$|18.80|0.5007|0.3554|18.07|**0.6343**|0.2344|\n|10K Iter|18.94|0.5161|0.3584|18.02|0.6251|0.2353|\n|30K Iter|19.06|0.5195|0.3502|**18.15**|0.6291|0.2294|\n|60K Iter (Ours)|**19.06**|**0.5354**|**0.3491**|18.12|0.6331|**0.2265**|\n\nAs shown in the table, adding $L_{consistency}$ at 60k training iteration yields better results.\n\n### W5: The generalizability of the image-matching method.\n**R:** We empirically find the pre-trained GIM[40] has great generalization to cross-domain data. Some of the matches obtained by GIM on S-NeRF rendered results are shown **in Fig.3 in the rebuttal PDF** to demonstrate its generalizability. \n\n### W6: Insufficient Dataset Contribution.\n**R:** Actually, we did **not just** take images from the LOL-Blur dataset and then ran an off-the-shelf SFM method to obtain the camera parameters. To build an effective dataset, we did the following works:\n\n (1) **Scene selection**: we go through the whole LOL-Blur dataset to select scenes **featuring different environments** (covering indoor/outdoor situations) with **different camera motions and lighting.**\n\n (2) **Image selection**: we manually select 20-25 (out of around 60) images per scene with relatively **high-quality** for the estimation of camera poses, and ensure their luminance is close to imitate the real shooting situation.\n\n (3) **Camera Pose Estimation**: Note that the estimation of camera pose by COLMAP for low-light blurry images are often **unreliable**. We first repeat the estimation of COLMAP **30 times for each scene** to select the optimal pose result, and then **manually tune** it with our baseline NeRF model to improve the accuracy.\n\nOur experimental results show that the resulting dataset helps us learn the **robust LuSh-NeRF** for handling low-light blurry scenes. We will clarify these in the revision.\n\n### W7&8: Missing references and typos.\n**R:** As suggested, we will cite and discuss these papers, and correct the typos in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their comments and suggestions. We are glad to see that reviewers comment our idea/work as novel (4gW7), sound (54cf), appreciated (4eod), and appealing (m3Lt). We address the raised concerns below and will revise our paper according to all comments. Please let us know if further clarification / discussion is needed.

Meta-review: Three reviewers recommended a weak accept for this paper, while only one reviewer rated it as borderline reject. The primary concern of the latter reviewer was the clarity of the paper, though they acknowledged its originality, aligning with the views of the other reviewers. I encourage the authors to incorporate the reviewers' feedback in the final version.
Decision: Accept (poster)