Deblurring Neural Radiance Fields with Event-driven Bundle Adjustment

Authors: yunshan  qi
jia  li
lin  zhu
yifan  zhao
nan  bao
  deblurring
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-26 15:04:01

Abstract: Neural Radiance Fields (NeRF) achieves impressive 3D representation learning and novel view synthesis results with high-quality multi-view images as input. However, motion blur in images often occurs in low-light and high-speed motion scenes, which significantly degrades the reconstruction quality of NeRF. Previous deblurring NeRF methods struggle to estimate pose and lighting changes during the exposure time, making them unable to accurately model the motion blur. The bio-inspired event camera measuring intensity changes with high temporal resolution makes up this information deficiency. In this paper, we propose Event-driven Bundle Adjustment for Deblurring Neural Radiance Fields (EBAD-NeRF) to jointly optimize the learnable poses and NeRF parameters by leveraging the hybrid event-RGB data. An intensity-change-metric event loss and a photo-metric blur loss are introduced to strengthen the explicit modeling of camera motion blur. Experiments on both synthetic and real-captured data demonstrate that EBAD-NeRF can obtain accurate camera trajectory during the exposure time and learn a sharper 3D representations compared to prior works.

Main paper: 
 introduction: 
 Neural Radiance Fields (NeRF) [27] achieves 3D implicit representation learning and photo-realistic novel view synthesis results with high-quality 2D images and precise camera poses as input. In low-light scenes, a camera often requires a longer exposure time to obtain an image with sufficient brightness [48] , and a handheld camera may cause motion blur in the captured image. Moreover, a high-speed moving camera can also cause motion blur, even in bright scenes with short exposure time. The blurred images will cause NeRF to learn a blurry 3D implicit representation, resulting in degraded quality of the synthesized novel view images. Thus, it is a practical problem to reconstruct a sharp NeRF from blurry images when facing low-light and high-speed scenes [46] .For static scenes, motion blur is caused by the camera pose change during the exposure time. The core problem of reconstructing a sharp NeRF is accurately restoring the motion trajectory to better model the motion blur formation. Correspondingly, we need to establish a connection between scene radiance fields and camera pose change. Recent deblurring NeRF works try to resolve this by using a learnable blur kernel [20, 21, 25] or interpolating the camera poses with learnable poses at the start and end of the exposure [40] . However, they solely implicitly model the motion and are supervised by RGB images, which is insufficient to establish a strong connection between camera pose change and NeRF parameters. This results in unstable network training and a decrease in performance for complex and severe motion blur. As in Figure 1 , Deblur-NeRF [25] and Bad-NeRF [40] all learn an inaccurate motion trajectory and an inferior NeRF with blurry rendering results.The bio-inspired event camera can measure brightness change asynchronously with high temporal resolution [9] , which makes up for the information loss in blurry images. Recently, E 2 NeRF Figure 1 : Motivation of our proposed method. In a low-light scene or with a high-speed moving camera, motion blur usually occurs in the captured images. To reconstruct sharp NeRF from blurry images, Deblur-NeRF [25] uses a deformable sparse kernel to model the blur process. BAD-NeRF [40] linearly interpolates the camera poses and jointly learns the start and end poses of camera motion. However, these methods are unable to model complex motion blur and lack supervision information during exposure time with only a photo-metric loss. With the proposed event-driven bundle adjustment, our EBAD-NeRF can leverage the event to recover the accurate camera trajectory and learn a sharper NeRF, resulting in sharper novel view image rendering than the previous methods.[ 30] uses the event data and blurry images to enhance the model of motion blur in terms of image formation and achieves better results than the image-based deblurring works. However, it uses pre-deblurred images and COLMAP [32] to estimate camera poses, which is not learnable during training, and the pose change information in the event data is not fully modeled and utilized.In this paper, we model the motion blur in terms of both motion trajectory and camera imaging. Event data is introduced to establish an explicit connection between the actual camera motion trajectory and the target sharp NeRF by conducting an intensitychange-metric event loss. Specifically, for each view, we first use multiple learnable camera poses to model the camera motion trajectory and transfer them into the SE(3) space [4, 40] . During the training, we jointly optimize these poses and the parameters of the NeRF network. We introduce an intensity-change-metric event loss to supervise pose changes during the exposure time. A photometric blur loss is also used to supervise the modeling of blurry image formation. The image formation is also additionally strengthened by the event loss. We extend 5 blender synthetic scenes in Deblur-NeRF with event data and use a DAVIS-346 event camera [38] to capture spatial-temporal aligned Event-RGB (ERGB) real data with ground truth. Experiments on both synthetic and real data demonstrate that our method can learn superior NeRF and more accurate camera motion trajectories than previous image-based or ERGB-based deblurring NeRF methods. The rendering results are also better than the state-of-the-art image-based or ERGB-based image deblurring methods. To summarize, we present the following contributions:1) A novel event-driven bundle adjustment deblurring neural radiance fields (EBAD-NeRF) framework is proposed to explicitly model image blur and jointly optimize the estimated camera motion trajectory and NeRF parameters.2) An intensity-change-metric event loss and a photo-metric blur loss are presented to supervise the modeled motion blur in terms of both camera pose change and image formation conjugatively.3) Experiments on extended synthetic data and real-captured data validate that our method achieves accurate motion trajectory estimation and high-quality 3D implicit reconstruction of the scene with severe blurry images and corresponding event data. 
related work 2.1 neural radiance fields: 
 The neural radiation fields model achieves impressive novel view synthesis results and inspires a lot of subsequent research [10] . Some works [2, 3, 39, 45] improve the quality of learned geometry and synthesized novel views. Other works try to improve the training and rendering speed [8, 11, 15] . Besides, BARF [22] , L2G-NeRF, and Nope-NeRF [5] explore reconstructing NeRF without camera poses estimated by COLMAP [32] and input images.Blurry Images are often captured in low-light scenes or with a high-speed moving camera on drones or robots. Reconstructing sharp NeRF from blurry images also becomes a challenging problem. As shown in Table 1 , Deblur-NeRF [25] proposes a deformable sparse kernel module to effectively model the blurring process. However, the blurring kernel is optimized based on 2D-pixel location independently. DP-NeRF [21] proposes a rigid blurring kernel and generates a 3D deformation fields, which is constructed as the 3D rigid motion of the camera for each view. Sharp NeRF [20] proposed a learnable grid-based kernel to obtain sharp output from neural radiance fields. Inspired by BARF, BAD-NeRF [40] conducts linear interpolation for the camera poses during the exposure time and jointly optimizes the poses with bundle adjustment.Traditional deblurring algorithms focus on finding the suitable blur kernel of a blurry image to recover a sharp image. Hand-crafted features or sparse priors are used to tackle this problem [7, 19, 44] . Deep learning deblurring works directly learn an end-to-end mapping from blurry images to sharp images [36, 41, 51] . MPRNet [47] and SRN [37] achieve impressive single-image deblurring results. However, the information missing from blurry images during exposure time decreases the robustness of the image-based methods and The event camera is based on a bio-inspired vision sensor that measures brightness changing asynchronously [9] . The high temporal resolution and high dynamic visual data capturing paradigm makes it ideal for flow estimation [1, 12, 14, 28, 34] , feature detection and tracking [43, 49, 50] and images deblurring [23, 29, 33, 42] . Pan et al. [29] propose a simple event-based double integral model (EDI) based on event generation to establish the connection between blurry images and events. Other works [18, 33, 35] use learningbased networks to recover sharp images with events. Recently, some event-based NeRF works have emerged. Ev-NeRF [17] models the measurement of the event sensor to learn grayscale neural radiance fields derived from the event stream. EventNeRF [31] reconstructs color NeRF with a color event camera. e-NeRF [24] improves Ev-NeRF and EventNeRF for non-uniform camera motion. SpikeNeRF [53] derives a NeRF-based volumetric scene representation from spike camera data. DE-NeRF [26] uses RGB images and events to learn a deformable NeRF. E 2 NeRF is the first to reconstruct a sharp NeRF with blurry images and corresponding event data and achieves state-of-the-art performance. However, it only explicitly models the blurring processing at the imaging aspect. For camera motion trajectory during the exposure time, E 2 NeRF uses EDI [29] to pre-deblurred images and COLMAP to estimate the camera poses that can not be optimized during training. Ev-DeblurNeRF [6] extends from DP-NeRF [21] with a novel EDI loss and an eCRF network to simulate the generation of events more realistically and achieves a remarkable deblurring effect. It also uses extra continuous events between spare image views to enhance the NeRF learning, which differs from the setting of E 2 NeRF and ours. 
method: 
 We introduce EBAD-NeRF to simultaneously learn the poses during the camera motion blur process and sharp neural radiance fields with blurred images and the event stream within the corresponding exposure time. Events serve two key purposes within the framework: (1) optimizing camera trajectories within the exposure time, and (2) contributing to the physical formation of motion blur in the image by providing information on light intensity changes. Moreover, we employ a photo-metric loss to supervise blur formation in the RGB domain. Figure 2 is the overview of our approach.In a static 3D scene, camera motion blur is caused by the changes of camera pose P during the exposure time. A digital image I is obtained by measuring the integration of light intensity 𝐼 with respect to time 𝑡 on the image sensor:EQUATIONwhere 𝑡 𝑠𝑡𝑎𝑟𝑡 and 𝑡 𝑒𝑛𝑑 are the start and end time of the exposure period. Since light intensity 𝐼 is only corresponding to the camera pose in the static scene, we can express P as a function of time P = 𝑝 (𝑡) and a blurry image B can be expressed as:EQUATIONIn NeRF [27] , given a camera pose P = 𝑃 (𝑡), for each pixel of the imaging plane, we can obtain a ray r that emits from the optical center of the camera and passes through the pixel x = (𝑥, 𝑦). With stratified sampling, we can divide the part of this ray starting with 𝑙 𝑛𝑒𝑎𝑟 and ending with 𝑙 𝑓 𝑎𝑟 into N equal parts and randomly sample one point in each part. For each of the 𝑁 sampled points, we input its 3D coordination o in the world coordinate system and the 2D view direction d represented by the ray r into NeRF MLP 𝐹 𝜃 with network parameters 𝜃 :EQUATIONThe outputs are color c and density 𝜎 of the point, and 𝛾 (•) encodes the input to a higher 𝐾 + 1 dimension:EQUATIONThen, we can obtain 𝑁 colors {c 𝑖 } 𝑁 𝑖=1 and densities {𝜎 𝑖 } 𝑁 𝑖=1 of the sampled points. By conducting volume rendering:𝐶 (r, x) = 𝑁 ∑︁ 𝑖=1 𝑇 𝑖 (1 -exp(-𝜎 𝑖 𝛿 𝑖 ))c 𝑖 ,where 𝑇 (𝑖) = exp(-EQUATIONwe can get the final color value 𝐶 (r, x) = 𝐶 (P, x) of the pixel x passed by the ray r corresponding to the pose P. 𝛿 𝑖 = 𝑙 𝑖+1 -𝑙 𝑖 is the distance between adjacent sampled points, and 𝑇 𝑖 is the transparency between 𝑙 𝑛𝑒𝑎𝑟 and the sampled point.If we assume that the learned NeRF MLP is sharp, we can model the image motion blur by discretizing Eq. ( 2) with 𝑝 poses {P 𝑖 } 𝑝 𝑖=1 = {𝑃 (𝑡 𝑖 )} 𝑝 𝑖=1 sampled evenly over the exposure time 𝑡 𝑠𝑡𝑎𝑟𝑡 to 𝑡 𝑒𝑛𝑑 :EQUATIONwhere X represents the pixels of the image sensor. Since we use virtual sharp image color 𝐶 (𝑃 (𝑡 𝑖 ), x) to replace the light intensity 𝐼 (𝑃 (𝑡 𝑖 )) of Eq. ( 2), we need to multiply each item by a weight of time. As the poses are temporally evenly sampled, we can directly use an average as 1 𝑝 to represent the weight as in Eq. ( 6). At this point, we establish a connection between image motion blur, camera pose, and neural radiation fields. In long exposure or high-speed situations, the camera is likely to have complex motion (non-uniform motion) during exposure time. To better model the motion blur, we transform the sampled 𝑝 poses{P 𝑖 } 𝑝 𝑖=1 into 𝑝 learnable pose {T 𝑖 } 𝑝 𝑖=1 ∈ SE(3) at time {𝑡 𝑖 } 𝑝 𝑖=1. Then, we can use events as a bridge between camera pose changes and neural radiance fields to bundle these poses into actual camera motion trajectory during the NeRF optimization.As described in Sec.3.1, we can express the virtual sharp image color during the exposure as 𝐶 (T 𝑖 , x) = 𝐶 (P 𝑖 , x) and convert it into grayscale as 𝐺 (T 𝑖 , x) by averaging the RGB channels. Then, according to previous event simulation work [16] , we transform 𝐺 (T 𝑖 , x) into the log function domain as Ł(T 𝑖 , x) and simulate the generation of events caused by camera motion as:EQUATIONThen with the real events E (x, 𝑡 𝑖 , 𝑡 𝑖+1 ) captured by the event camera as event-driven bundle adjustment, we can optimize adjunctive poses 𝑇 𝑖 and 𝑇 𝑖+1 with the intensity-change-metric event loss by minimizing the differences between the numbers of predicted and actual events on all 𝑚 pixels x ∈ X among all input views:L 𝑒𝑣𝑒𝑛𝑡 = 1 𝑚 ∑︁ x∈ X 1 𝑝 -1 𝑝 -1 ∑︁ 𝑖=1 ∥( Ê (x, 𝑡 𝑖 , 𝑡 𝑖+1 )) -(E (x, 𝑡 𝑖 , 𝑡 𝑖+1 ))∥ 2 2 .(8) The computation of Ê (x, 𝑡 1 , 𝑡 2 ) is differentiable with respect to {T 𝑖 } 𝑝 𝑖=1 and latent sharp NeRF parameters 𝜃 , which is optimized by the Jacobians of event loss:EQUATION)EQUATIONNotice that in Eq. ( 8) we only calculate the event loss between adjacent poses because if the sampled poses are too far apart, the event loss will fluctuate greatly, reducing the training stability.We conduct the photo-metric blur loss between the predicted and the input blurry images as in NeRF for all 𝑚 pixels x ∈ X among all input views:EQUATIONThe blur loss not only has a constraint on camera poses {T 𝑖 } 𝑝 𝑖=1 to avoid the spreading of poses but also plays a main role in supervising NeRF parameters 𝜃 to learn texture details of the scene that event data cannot provide:EQUATION)EQUATION)The final loss is defined as:EQUATION)where 𝜆 is a weight parameter of event loss. We used the design of the fine and coarse network in NeRF and conduct blur loss L 𝑓 𝑏𝑙𝑢𝑟 and L 𝑐 𝑏𝑙𝑢𝑟 on both of the networks. Event loss L 𝑓 𝑒𝑣𝑒𝑛𝑡 is only calculated for the fine network because it can render enough texture details for precise event estimation. Then, the network can learn an accurate camera motion and latent sharp 3D implicit representation with gradient propagation.We train the EBAD-NeRF on a single NVIDIA RTX 3090 GPU. The training time is close to BAD-NeRF. The sample number of the rays is 𝑁 = 64 for both fine and coarse networks. The number of sampled poses of each view 𝑝 is set to 5, and the weight of event loss 𝜆 is set to 0.005. Sec.4.4 evaluates the influence of the parameters. Additionally, we use Θ = 0.3 as the threshold of the event generation, which is a typical value in event-based vision [9] . A coarse initial pose for each view is given at the start of training. 
synthetic data:: 
 We use the five Blender scenarios of Deblur-NeRF (Cozyroom, Factory, Pool, Tanabata, and Wine) to generate our synthetic data. We increase the camera shake amplitude to generate a severe blur and add the camera motion speed change during the exposure to simulate a non-uniform camera motion blur for the Factory, Pool, Tanabata, and Wine scenes. Besides, we input the virtual sharp frames during the camera shake process rendered by Blender into the V2E [16] to generate the corresponding event stream as in the synthetic datasets of E 2 NeRF.To test the effectiveness of our method on real data, we capture two sets of real data (Bar and Classroom) with DAVIS-346 [38] event camera, which can capture spatial-temporal aligned event data and RGB data at the same time. Both sets of data are captured in low-light scenes, so RGB data requires a longer exposure time (100ms) to get a bright enough image. We use a handheld camera to capture blurry images and events for training and a tripod to capture sharp images for testing. Each scene contains 16 views of blurry images and corresponding events for training and 4 novel view sharp images for testing.4.2.1 Image Deblurring Methods. For comparison, we selected two classic learning-based image deblurring methods, MPR [47] and SRN [37] . In order to make a more fair comparison in terms of data modalities, we use D2Net [33] , which also uses events and blurry image data to achieve image deblurring. We input the images deblurred by these three methods into the original NeRF for the novel view generation task and named them as MPR+NeRF, SRN+NeRF, and D2Net+NeRF.We selected image-based deblurring NeRF methods Deblur-NeRF [25] and BAD-NeRF [40] and ERGB deblurring NeRF method E 2 NeRF [30] . All NeRF-based methods are trained with 5000 rays of batch size and 64 sampling points for coarse and fine networks. Other parameters are the default values of the methods. We perform 100,000 iterations on synthetic data. On real data, due to the reduction in image resolution and the number of input views, 50,000 iterations are enough for all NeRF-based methods to converge to the final results.We use PSNR, SSIM, and LPIPS [52] to evaluate the reconstruction results and use absolute trajectory error (ATE) to evaluate the fitting accuracy of the camera motion trajectory [13] . We analyze the proposed event-driven bundle adjustment at two levels: 3D reconstruction quality and motion trajectory fitting accuracy. In Table 2 and Figures 3, 4 , EBAD-NeRF-noe training the proposed EBAD-NeRF without event data as supervision and EBAD-NeRF-linear represents the results of adding event data constraints to BAD-NeRF with linear interpolation directly. Results in Table 2 are the average of the deblurring view and novel view.The quantitative results in Table 2 demonstrate that without event enhancement, the bundle adjustment constrained by linear interpolation (BAD-NeRF) is even worse than the unconstrained bundle adjustment (EBAD-NeRF-noe) when facing complex camera motion. With event enhancement, the same conclusion can be obtained by comparing EBAD-NeRF-linear and EBAD-NeRF-full.Comparing BAD-NeRF and EBAD-NeRF-linear, even with the negative impact of linear pose interpolation, EBAD-NeRF-linear still has improvements in trajectory fitting and reconstruction results with event enhancement as shown in Table 2 . Comparing EBAD-NeRF-noe and EBAD-NeRF, introducing events significantly improves the accuracy of trajectory estimation and reconstruction results.To sum up, during the NeRF training, event data can not only provide image-level optimization information but also pose-level practical constraints. Our proposed EBAD-NeRF effectively utilizes the ERGB-dual-modal data to achieve this goal. The qualitative comparison in Figure and We evaluate the rendering results of the Cozyroom scene to analyze the influence of 𝑝 and 𝜆 of our method in Figure 5 .The influence of 𝑝: Intuitively, the more virtual sharp frames, the more accurate the simulation of camera motion blur will be, and the corresponding network learning results will be better. The results in Figure 5 are consistent with this. When the number of virtual frames gradually increases from 1, the network's performance also significantly improves. However, after exceeding 5, the number of virtual frames has almost no impact on the reconstruction results, and more virtual sharp frames will require more calculations. Therefore, 𝑝 = 5 is set in our experiments.The influence of 𝜆: As shown in Figure 5 , when we take 𝜆 as 0.005, EBAD-NeRF achieves the best results on the three metrics. 0.0419 ± 0.0159 BAD-NeRF (Linear Interpolation [40] ) 0.0462 ± 0.0142 BAD-NeRF (Cubic Interpolation [40] ) 0.0542 ± 0.0207 Event-driven Bundle Adjustment (Ours) 0.0383 ± 0.0143 4.5 Quantitative ResultsWe use the Evo tool [13] to calculate the absolute trajectory error, considering both rotation and translation. The results in Table 3 are the average of five blender scenes, which shows the accuracy of camera motion trajectory estimation by different methods. COLMAP-Blur means directly estimating the camera poses with blurry images and COLMAP for each view. COLMAP-EDI represents the pose estimation method used in E 2 NeRF, which uses the EDI [29] algorithm to first deblur the image with events and then input the sharp image sequence into COLMAP to estimate camera poses during the blur process. BAD-NeRF learns the start and end poses of camera motion. It uses linear interpolation to simulate the intermediate poses and extends it with a cubic interpolation. The results of these two methods are also shown in the table. Our proposed event-driven bundle adjustment method introduces event into joint learning of camera pose changes and sharp NeRF, achieving the best results in restoring the camera pose, which also promotes EBAD-NeRF to further improve the reconstruction quantification results as in Sec. 4.5.2 and Sec. 4.5.3.We evaluate the reconstruction results of five blender scenarios on both deblurring views (Table 4 ) and novel views (Table 5 ). SRN and SRN+NeRF achieve the best results among single-image-deblurringbased methods and even surpass the Deblur-NeRF. With the help of event data, D2Net and D2Net+NeRF are slightly better than MPR and MPR+NeRF. BAD-NeRF with cubic interpolation is further improved over BAD-NeRF with linear interpolation and SRN+NeRF, though it is still worse than event-enhanced methods E 2 NeRF and our proposed EBAD-NeRF on both novel view and deblurring view.Note that there is no apparent difference between the E 2 NeRF and EBAD-NeRF on the Cozyroom scene where blur is only caused by linear camera motion, and E 2 NeRF is even better than EBAD-NeRF on LPIPS. However, in the other four scenes with more complex camera shaking as described in Sec. 4.1.1, the results of EBAD-NeRF in both deblurring views and novel views are significantly better than E 2 NeRF, which indicates that with fixed pre-estimated poses limits the effect of events in 3D implicit learning in E 2 NeRF, especially when facing complex camera motion. Accordingly, EBAD-NeRF uses event-driven bundle adjustment to jointly optimize the motion poses of the camera, further releasing the potential of event data in reconstructing a sharp NeRF with blurry images. 4 and 5 , which verifies the reliability of our method in practical applications.4.6.1 Qualitative Results of Reconstruction on Blender Scenes. We evaluate the reconstruction results of the deblurring view on the Pool scene (Figure 6 ) and the novel view on the Wine and Factory scenes (first two rows in Figure 7 ). Since image-deblurring methods cannot synthesize novel view images, we do not show their results in Figure 7 ). The results of EBAD-NeRF are sharper than all other methods on both deblurring and novel views. For the green plants and wooden boxes in the Pool scene, some artifacts appear in the results of E 2 NeRF, and our method effectively recovers the texture details. In the Wine and Factory scenes, though E 2 NeRF realizes noticeable results for the characters and letters, our method is closer to ground truth in color, clarity, and line thickness.4.6.2 Qualitative Results of Reconstruction on Real Scenes. As in the third row of Figure 7 , our method better recovers the white letters on the blackboard compared to E 2 NeRF. BAD-NeRF is limited in reconstructing smooth areas (white areas on the podium). The results of other image-deblurring-NeRF methods and Deblur-NeRF are all limited by the severe blurry input, which is consist with the quantitative results in Table 6 .4.6.3 Qualitative Results of Pose Estimation on Blender Scenes. In Figure 8 , we compare the estimated camera trajectory of different methods from the three dimensions of roll, pitch, and yaw. The purple curve representing EBAD-NeRF is the closest to the ground truth dashed line, which is consistent with the results in Table 3 . Experiments on synthetic and real data both show that our method can learn better poses of the camera motion blur process with event-driven bundle adjustment compared to the previous deblurring NeRF methods. Upon this basis, the event data is also superimposed on the image blur process supervision, eventually achieving better 3D reconstruction and novel view synthesis effects. 
conclusion: 
 In conclusion, this paper presents a novel approach, leveraging event-driven bundle adjustment, to address the challenge of modeling camera motion blur within neural radiance fields (NeRF). We draw inspiration from the emerging field of event-based vision, which offers high temporal resolution events, ideal for capturing dynamic information in motion blur. By integrating intensity-changemetric event loss and photo-metric blur loss, our framework enables the simultaneous optimization of blur modeling alongside NeRF reconstruction. Experiments on synthetic and real-captured datasets demonstrate the efficacy of our approach in accurately estimating camera poses and producing sharp NeRF reconstruction results. 
 
 Appendix:  


Decision: This paper was reviewed by three experts in the field. The recommendations are Borderline Reject, Borderline Reject, and Weak Reject. The main concerns raised by the reviewers include 1) Comparison to EvDeblurNeRF, 2) novelty, and 3) inapplicable to BAD-NeRF Dataset. 

However, after serious consideration, ACs feel that these concerns are less critical. 1) EvDeblurNeRF is published after ACM MM submission deadline, and thus should be considered as concurrent work. 2) Although the idea of this work comes from BAD-NeRF and E2NeRF, integrating the event signal to BAD-NeRF is non-trivial. 3) BAD-NeRF Dataset does not include event data, and thus it is acceptable that the proposed solution does not work well on it.

Also, as all reviewers agree that this is the first work that uses event data for bundle adjustment, ACs feel this submission does bring enough contribution to the field. Therefore, the decision is to recommend the paper for acceptance to ACM Multimedia 2024.

Still, we recommended the authors to carefully read all reviewers’ final feedback and revise the paper accordingly. We congratulate the authors on the acceptance of their paper!

Reviews: review: paper_summary: paper_summary: This paper introduces Event-driven Bundle Adjustment for Deblurring Neural Radiance Fields (EBAD-NeRF), which jointly optimizes learnable poses and NeRF parameters using hybrid event-RGB data. By incorporating an intensity-change-metric event loss and a photometric blur loss, EBAD-NeRF accurately models camera motion blur, resulting in improved camera poses and sharper 3D representations compared to previous methods.
strength_weakness: strength_weakness: 1. Extensive results are conducted on private datasets.
2. Combing bundle adjustment with event information is a reasonable choice.
limitations: limitations: 1. This work does not include a comparison with the most relevant research (Ev-DeblurNeRF, CVPR2024) in this area.
2. The method's novelty seems insufficient when compared to BAD-NeRF and E2NeRF. In my opinion, this work simply incorporates an event-guided loss (Eq.8) into BAD-NeRF, and this loss closely resembles the event loss employed in E2NeRF (Eq.9 in the E2NeRF paper).
3. Comparisons are only made using the proposed dataset. The effectiveness of this method could be further validated by comparing it to related works (such as E2NeRF, Ev-DeblurNeRF) on publicly available datasets like 1) https://github.com/iCVTEAM/E2NeRF, and 2) https://github.com/uzh-rpg/EvDeblurNeRF.
4. In Figure 1, BAD-NeRF produces poor results with linear interpolation. However, to my knowledge, BAD-NeRF is capable of performing cubic interpolation, which is available in its open-source code (https://github.com/WU-CVGL/BAD-NeRF). I am curious if the results would improve considering the expanded exposure time of the synthetic data used for this task.

As a result, I think this paper should not be accepted in its current state. I recommend authors to include the following results in the rebuttal:
1. Demonstrating the proposed method's superiority by comparing it with Ev-DeblurNeRF (CVPR2024) in synthetic and real datasets.
2. Offering key insights into the method's novelty (beyond simply combining bundle adjustment from BAD-NeRF with event loss).
3. Including additional comparison results on public datasets to further validate its effectiveness.
paper_summary: paper_summary: The authors consider a problem of bundle-adjusting NeRF with blur image input, where both camera poses and NeRF parameters should be jointly optimized. Additional event-RGB data is introduced to solve this problem and thus an intensity-change-metric event loss is proposed to model the camera motion blur during training. The authors carry out extensive experiments and compare with other baseline methods on various benchmark datasets, which validates the effectiveness of the proposed method.
strength_weakness: strength_weakness: Good writing. The manuscript is well-written overall. The motivation behind the authors' proposal is presented. All the steps in the proposed method are explained in a well-organized and easily understandable manner.
Extensive experiments. The authors performed comparisons with SOTA methods, which I think is sufficient to validate the effectiveness of the authors' method. The ablation study helps readers to better evaluate the contributions of this paper.
Impressive results. The authors' method outperforms the previous methods by notable margins both in synthetic and real-world datasets.
limitations: limitations: In my opinion, this is a nice paper to solve the joint optimization problem involving pose estimation and NeRF representation with blur image input. However, my concerns are as follows:
Lack of novel view synthesis results of event-camera. As event data is introduced to supervise this bundle-adjusting representation, the authors should provide quantitative results of rendered event images and show the comparison with the ground truth event data. 
Lack of novelty. As far as I know, the authors just combine BAD-NeRF and E2NeRF to solve the joint optimization issue and further improve the performance of the baseline method, BAD-NeRF. Therefore, I give a borderline accpet. Please correct me if I am wrong and I will change my rating score.
paper_summary: paper_summary: This paper proposes Event-driven Bundle Adjustment for Deblurring Neural Radiance Fields (EBAD-NeRF) to strengthen the explicit modeling of camera motion blur.
strength_weakness: strength_weakness: A novel event-driven bundle adjustment deblurring neural radiance fields (EBAD-NeRF) framework is proposed to explicitly model image blur and jointly optimize the estimated camera motion trajectory and NeRF parameters
limitations: limitations: 1.In 3.2,Event-driven Bundle Adjustment is not clearly stated.Formulas 7 and 8 only explain how events are triggered, but do not explain the representation of events in methods.There is no clear explanation of the advantages brought by introducing events.
2.In Table 2 and Table 3, Quantitative Results on the Tanabata Scene, The same scene got different results in PSNR SSIM etc.
3.In 4.1.2, The real data set is not introduced in detail, making the real data results in Table 5 and Figure 7 not confusing.
4.Suppose you compare your performance on the synthetic data set of the BAD-NeRF can better reflect the superiority of the improvement of your proposed method without events.